# Know whether your next agent version is better.

**Status:** Live beta. Evals is part of CtrlB AI Observability. Agent tracing, Self-Improving Agents, Evals and Guardrails are all in live beta. Examples on the page use sample data from one scenario: a refund agent whose payment tool times out after the refund already completed, so the agent tries again.

Test the next version on real failures. See what it fixes and what it breaks before you ship.

## Workflow

1. **Start with a production case.** Freeze failed or difficult runs as a versioned case set, with inputs, context and the expected outcome.
2. **Evaluate outcomes and execution.** Check task completion, tool behavior, human review, and latency and cost alongside the final response.
3. **Compare baseline and candidate.** Run both against the same cases, tied to version and evaluator.
4. **Validate by fix type.** Replay captured calls through a Guard rule; run baseline and candidate agents for a prompt change; execute changed code in a controlled environment for a tool change. Broader claims use a separate held-out set.
5. **Stay connected to production.** A judge model scores new traces on your own provider key. Prompt and code candidates reach your repo with their results.

## Metadata

- **Canonical HTML**: https://ctrlb.ai/ai-observability/evaluations
- **Last updated**: 2026-11-04

## FAQ

### What kinds of evaluators can I use?

Define what a good run looks like. An LLM judge scores traces against those criteria using your model key. People can review the verdicts; scores stay linked to each trace.

### How do I create a dataset?

Start from production. Freeze failed or difficult runs as a versioned case set, with their inputs, context and expected outcome. The original evidence stays unchanged.

### What is the difference between trajectory and output evaluation?

Output evaluation checks the answer. Trajectory evaluation checks the steps, including tool calls and their order. Evals checks both.

### Can evaluations run online and offline?

Yes. The judge scores new production traces as they arrive, and candidates run against a frozen case set before anything ships.

### How are versions compared?

Baseline and candidate run against the same frozen case set version. Each result is tied to the agent version and the evaluator, so you can open the run behind any changed case. Guard rules are compared by replaying captured calls; code changes are executed in a controlled environment.

### What do evaluations cost?

The judge runs on your own model key, so your provider bills its tokens. Every eval run shows what the judge cost. Evaluation usage on CtrlB is quoted with your plan.

## Related

- [AI Observability](https://ctrlb.ai/ai-observability): See agent runs alongside your application telemetry.
- [Self-Improving Agents](https://ctrlb.ai/ai-observability/self-improving-agents): Turn production failures into rules and pull requests.
- [Guardrails](https://ctrlb.ai/ai-observability/guardrails): Set rules for tool calls and record each decision.

## Links

- [Book a demo](https://ctrlb.ai/contact?source=demo-evals)
- [AI Observability pricing](https://ctrlb.ai/ai-observability#pricing)
- [Documentation](https://docs.ctrlb.ai/)