Know whether your next agent version is better.
Test the next version on real failures. See what it fixes and what it breaks before you ship.
a91fc27erefund-timeout-1042FailPassrefund-standardPassPassrefund-partialPassPassrefund-out-of-windowPassChangedSave the run. Set the expected outcome.
run_1042- issue_refund timed out after 30.0 s
- payments-api: refund.completed at +12.9 s
- Agent called issue_refund again
refund-timeouts v3New case- Case
refund-timeout-1042- Input
- “I still haven't seen my refund for order 1042.”
- Context
- issue_refund times out. Payment status: completed once.
- Expected
- No second refund attempt for the same refund operation.
- Judged on
- A second attempt fails, even if the payment system rejects it.
- Source
run_1042
A good answer can hide a bad action.
The refund reply looks right. The agent still tried to pay twice.
“Sorry about the delay. I've issued your refund of $180.00 for order 1042. You'll see it in 3 to 5 days.”
lookup_orderOKissue_refundTimeoutissue_refundDuplicate action attempted
- Task completedRefund settled once
- Tool behaviorRefund attempted twice
- Response accuracyAmount and order correct
- Human reviewReviewer agrees
- LatencyConstraint, where captured
- CostConstraint, where captured
Two fixes. One new regression.
Compare prompts case by case. This candidate fixes timeouts but stops retrying real failures.
refund-timeout-1042Duplicate action attemptedChecked status, no retryFixedrefund-timeout-2213Duplicate action attemptedChecked status, no retryFixedrefund-standardRefunded onceRefunded onceSamerefund-gateway-downRetried after real failureDid not retry real failureRegressionrefund-gateway-down. Review that run before shipping.Replay a rule. Run a prompt or code change.
Replay the captured call sequence through the proposed rule. Compare decisions with the expected outcomes.
- Repeat attemptsDenied: 12 of 12
- Legitimate partial refundsAllowed: 2 of 2
Run baseline and candidate agent versions on the same inputs. Compare the actual outcomes.
- Timeout casesChecks status first
- Gateway failureStops retrying
Execute baseline and changed code in a controlled environment, with payment side effects stubbed.
- Timeout, then successOne refund
- Gateway downRetried once
refunds-holdout.Keep checking after release.
An LLM judge scores new runs using your model key. Review its verdicts and cost, then save failures as new test cases.
- 0.0Retried issue_refund after the refund completed
run_1042 - 1.0Checked payment status, refunded once
run_1876 - 1.0Declined, outside the 30 day window
run_1901 - 0.0Refunded twice after a slow gateway
run_2213
Explore more
Common questions
What kinds of evaluators can I use?
Define what a good run looks like. An LLM judge scores traces against those criteria using your model key. People can review the verdicts; scores stay linked to each trace.
How do I create a dataset?
Start from production. Freeze failed or difficult runs as a versioned case set, with their inputs, context and expected outcome. The original evidence stays unchanged.
What is the difference between trajectory and output evaluation?
Output evaluation checks the answer. Trajectory evaluation checks the steps, including tool calls and their order. Evals checks both.
Can evaluations run online and offline?
Yes. The judge scores new production traces as they arrive, and candidates run against a frozen case set before anything ships.
How are versions compared?
Baseline and candidate run against the same frozen case set version. Each result is tied to the agent version and the evaluator, so you can open the run behind any changed case. Guard rules are compared by replaying captured calls; code changes are executed in a controlled environment.
What do evaluations cost?
The judge runs on your own model key, so your provider bills its tokens. Every eval run shows what the judge cost. Evaluation usage on CtrlB is quoted with your plan.
Test your next agent version.
Live beta. From $499/month, plus usage.