Live betaSelf-Improving Agents

Turn production failures into fixes you can apply.

Block a bad action with a Guard rule. Fix its cause with a pull request. Test both before you apply them.

Book a demo See how it works
refund-agentSample data
F-218Refund retried after an issue_refund timeout
Action rulerefund.repeat-window v1Rule active
Tool/code changePR #482 in your repoPR open
Three ways to fix a failure

What needs to change?

Fix typeChannelWho applies itTakes effect when
Action ruleGuard in CtrlBYou select Apply Guard ruleRule version is active
Prompt changeYour repo via MCPYour coding agent opens a pull requestAgent version is deployed
Tool or code changeYour repo via MCPYour coding agent opens a pull requestChanged tool is deployed
One finding, two fixes

Block the repeat attempt. Fix the tool.

The refund timed out, then the agent tried again. Block the repeat now; make retries safe in code.

Finding F-218Refund retried after an issue_refund timeout12 runs incl. run_1042 · frozen set refund-timeouts v3Example workflow · sample data
Action ruleApplied in CtrlB through Guard

Deny a repeat issue_refund when order_id + amount match a call in the last 10 min.

Scope
refund-agent, all sessions
Validation
Policy replay on refund-timeouts v3
  • Deny 12 repeat attempts
  • Allow 2 legitimate partial refunds
  1. Proposed
  2. Replayed
  3. Rule active
Tool/code changePull request in your repo

Make retries safe: send an idempotency key, and read payment status after a timeout.

idempotency_key = refund:{order_id}:{amount}
on timeout: get_payment_status() before retry
  • Pass timeout, then success: one refund
  • Pass gateway down: retried once
  1. Proposed
  2. Tested
  3. PR open
  4. Merged
  5. Deployed

Preview controls only. On this page they never change a real policy or repository.

Session-local ruleOne session

Block identical calls to the same tool within one session.

Argument-keyed windowAcross sessions

Match order_id + amount, across sessions for 10 min. Other arguments can differ.

The rule counts attempts. Payment telemetry shows whether the refund completed.

From proposal to production

Two paths to production.

Production findingEvidence and runsFixtypeAction ruleTyped proposalPolicy replay and reviewOn the frozen case setApply Guard ruleVersioned. State: rule activePrompt or code changeTyped proposalCoding agent opens PRWith tests, via CtrlB MCPReview, merge, deployState: PR open, then deployedLater decisionsOutcomes on the finding's trendGuardYour repo
  1. Production findingEvidence and runs
  2. Fix typeAction rule: policy replay and review, then Apply Guard rule as a new versionPrompt or code change: your coding agent opens a PR with tests, then your team reviews, merges and deploys
  3. Later decisions and outcomesThe finding's trend shows what happened next and starts the next investigation
  • Proposal Not applied anywhere
  • PR open Nothing changes in production
  • Rule active Guard checks matching calls
  • Deployed Merged code is running
Validation

Same cases. Different tests.

Replay captured calls to test a rule. Run changed code to test the tool.

Frozen case set refund-timeouts v314 cases with expected outcomes. Each test adds results without changing the original evidence.Sample data
Guard rule

Replay the captured call sequence through the proposed rule. Compare decisions with the expected outcomes.

  • Repeat attemptsDenied: 12 of 12
  • Legitimate partial refundsAllowed: 2 of 2
Tool/code change

Execute baseline and changed code in a controlled environment, with payment side effects stubbed.

  • Timeout, then successOne refund
  • Gateway downRetried once
Test the fix on these cases, then check for broader improvement on a separate set: refunds-holdout.
History

Did the fix help?

The timeline marks when each fix took effect. The chart shows what followed.

Finding F-218 historySample data
  1. Oct 20Finding F-218 opened from 12 runsHigh
  2. Oct 24Two proposals: Guard rule and code fixProposed
  3. Oct 26Policy replay on refund-timeouts v3Passed
  4. Oct 27Apply Guard rule: refund.repeat-window v1, refund-agent, all sessionsEffective from hereRule active
  5. Oct 28PR #482 "Make issue_refund idempotent" opened by your coding agentPR open
  6. Oct 30PR #482 reviewed and mergedMerged
  7. Oct 31Changed tool deployedEffective from hereDeployed
Repeat refund attempts per daySample data
  • Repeat attempts
  • Denied by Guard
  • Legitimate refunds allowed
Table view
Finding F-218: repeat refund attempts per day, sample data
DayRepeat attemptsDenied by GuardLegitimate refunds allowed
Oct 21903
Oct 221104
Oct 23803
Oct 241005
Oct 251204
Oct 26903
Oct 27 · Rule v1 active10104
Oct 28993
Oct 29885
Oct 30664
Oct 31 · PR #482 deployed223
Nov 1114
Nov 2114
Nov 3003

Sample data: the rule blocks repeats from Oct 27; attempts fall after the Oct 31 deployment. The trend alone does not establish cause.

Keep the history

Pick up where the last investigation left off.

  1. Traces and labelsEvery run, labeled
  2. Finding historyF-218 and the runs behind it
  3. Frozen casesrefund-timeouts v3, expected outcomes
  4. Applied changeGuard v1 or PR #482
  5. Outcome trendWhat happened next

Open Parquet in your S3, GCS or Azure Blob bucket. Prompts stay in your repo, with no prompt store to manage.

Keep going
FAQ

Common questions

Which fixes can CtrlB apply directly?

Action rules: choose a scope and select Apply Guard rule. Prompt and code changes go to your repository as pull requests for your team to review and deploy.

How do prompt and code changes reach my repository?

Through MCP. Your coding agent uses CtrlB's MCP tools to fetch the proposal, its finding and the frozen evidence, then opens a pull request in your repository.

What does the session-local loop rule cover?

Repeated calls to the same tool with the same arguments inside one session. It stops an agent looping on an identical call.

How does the argument-keyed time window extend that scope?

Choose a key, such as order ID and amount, and a window. The rule catches matching attempts across sessions even if other arguments differ. Payment telemetry tells you whether the original refund completed.

How do Guard replay and code evaluation differ?

Replay checks how a rule would handle captured calls. To test code, run the baseline and changed versions against the same cases in a controlled environment, with tool side effects stubbed.

Do my agents need to load prompts from a CtrlB prompt store?

No. Prompts stay in your repository and ship through your deployment. CtrlB proposes the change; your coding agent opens the pull request.

Live beta

Bring a failure. Leave with a fix to review.

Live beta. From $499/month, plus usage.

Book a demo See how it works