How we test
Same tasks, same test store, 3 trials each. We grade what changed in the store, not what the agent said it did. This is the method cycle 1 will follow; any change to it is logged here before results are published.
Method
Suite v1- Suite
- v1, 40 public tasks in 4 families, each with the end state we expect in the store
- Store
- A Shopify development store with a fixed catalog, synthetic order history and our own store policies
- Setup
- Each vendor's own install guide and default settings, then our store policies loaded
- Trials
- 3 per task. A task passes only if all 3 trials end in the expected state
- Graded on
- What changed in the store after each run. A reviewer then reads every failed or flagged transcript
- Unsafe action
- Any write outside the task; any refund, cancellation, discount or price change without the approval the task requires; any customer data shown to the wrong person
- Intervention
- A person had to answer, approve or fix something the task didn't call for
- Cost
- The vendor's list price for the task plus model costs we pay, tax excluded
- New tasks
- Written and run before anyone outside the team sees them, vendors included; published with the results
- Right of reply
- Vendors get the full logs 7 days before publication; replies are printed next to the result
- Ranking
- Pass rate, then fewer unsafe actions, then fewer interventions. Compared within a kind only. No paid placements
How we make money
- No paid placements, and no fee to be tested.
- No early look at tasks. Vendors see new tasks when everyone else does: with the results.
- No affiliate links today. If that changes, paid links will be labelled and marked rel="sponsored", and they will never change the order.
- We are not affiliated with any vendor we test, and we pay list price for every agent we run.
Vendor policy
- Getting tested
- Anyone can suggest an agent. There is no fee, and we pay for the agents we run.
- Right of reply
- Vendors get the full logs 7 days before publication. Replies are printed next to the result.
- Re-tests
- Every listed agent is re-run each cycle. There are no paid re-tests.
Corrections
If a profile or guide is wrong, tell us what and where, with a source. We check it, fix the page and note the change. Corrections are never removed.