Highlights
- Blitz runs more than 25 agents in production, including in-house voice agents on over 110,000 minutes a month and a support agent on more than 25,000 conversations a day.
- Datadog was too complex to fit the way Blitz's ecosystem is built, so the team ran its own observability system, with logs and metrics but no traces. During one incident that system was down as well. Nothing was recorded for the window that mattered, and the final RCA took days.
- On CtrlB, logs and traces are correlated in one place and CtrlB MCP runs beside Blitz's AWS and GitHub MCP servers, so an agent handles the first pass of an investigation.
- Root cause now takes about 5 to 7 minutes, and Blitz works to a 99.99% availability target.
See it on your own data
Behind Blitz's Agent-Led Incident Response with CtrlB
Blitz moves other brands' orders. D2C and e-commerce companies plug into it to promise delivery in 10 minutes, 60 minutes, 2 hours, same day or next day, and Blitz runs the dark stores, the fulfilment centres and the rider network behind that promise. When the window is measured in minutes, an outage is visible immediately, to the brand and to the person waiting for the order.
The system behind it is highly distributed. Many services, a lot of logs, and a request path that crosses most of them before anything reaches a rider. More than 25 agents run on top of that, most of them facing customers directly, which gives observability two jobs: explain what broke in the infrastructure, and show how the agents are behaving with the people they serve.
This is where CtrlB came in.
The challenge: The observability system was the first thing to go down
Why Datadog did not fit
Datadog was not a fit for the way Blitz's ecosystem is built, and getting it in would have been too complex to be worth the work. Rather than reshape the infrastructure around a tool, the team built and ran its own observability system. It carried logs and metrics. It did not carry traces, which is the part that tells you where in a distributed request path the problem actually sits.
One incident on the previous stack
- 1Downtime starts.An incident takes production systems down while customers are placing orders.
- 2The team opens its own observability system.It is already down.
- 3Nothing was recorded.No data exists for the incident window, so the event the team needed to study left no trace of itself.
- 4Service returns, the answer does not.Blitz is live again in reasonable time, but the final RCA takes days.
We went to our own observability system to find out what has happened. But we found out that our observability system was already down. There was no data of that incident, which cost us heavily.
Gaurav Piyush · Co-founder, BlitzWithout traces, every incident starts with the same question, which service is the bottleneck, and answers it by elimination. The challenge was to get logs and traces into one searchable place, and put the answer within reach during the incident rather than days after it.
The solution: Logs and traces in one lake, with CtrlB MCP in the incident path
CtrlB holds logs, metrics and traces on one lake. The gap on Blitz's previous stack was traces, so that is where the change shows up first. Blitz ships its telemetry into CtrlB and runs CtrlB's MCP server alongside its AWS and GitHub MCP servers, and when any alerting system fires, an agent starts the investigation before a human opens a dashboard.
Blitz incident path
Traces and logs, correlated
Before CtrlB, Blitz had no visibility of traces at all. Logs, metrics and traces now sit on the same lake, so the failing service and the log line that explains it come out of the same investigation.
CtrlB MCP in the incident path
On a downtime trigger, the agent runs the RCA through CtrlB MCP, scans the code for context through GitHub, and raises a pull request to fix it, while informing the team.
Plain language instead of query syntax
Engineers describe what they are looking for and get back the instance, the API and the function that is breaking, and whether the cause is a database or a service.
Anomaly detection on agent behaviour
Alerts fire on behaviour outside the shape of a use case, such as one customer holding far more conversations than it allows. That is how misuse and jailbreak attempts surface early.
See it on your own data
The results: Root cause in 5 to 7 minutes, and a 99.99% uptime target
The impact shows up in how fast Blitz reaches a root cause, and in what the team does with the time it gets back.
Time to root cause: from days to 5 to 7 minutes
Time to final root cause
An alert now triggers the agent, the agent runs the RCA on CtrlB, scans the code for context, raises a pull request with a fix and informs the team. Gaurav puts the whole cycle at about 5 to 7 minutes, against days to establish the final RCA on the previous stack.
Finding the bottleneck in a distributed system
Blitz's engineers no longer open an investigation by narrowing down which service to suspect. They ask, and the answer comes back in a fraction of a minute.
We just have to give a command in natural language, and we can finally find what is wrong, what is the instance, what is the API, and what is the function which is breaking, and why it is breaking.
Gaurav Piyush · Co-founder, Blitz25+ agents in production, watched for the behaviour nobody planned for
Four problems come with agents at this scale: keeping replies consistent, users trying to jailbreak them, hallucination, and an autonomous agent with tools doing something that was not required. A fixed set of evals per use case, with an LLM as judge, covers the failures Blitz has already seen. Anomaly detection and alerts on CtrlB cover the ones it has not, by flagging conversation patterns that sit outside what the use case permits.
From firefighting to a 99.99% uptime target
Blitz now works to 99.99% availability, and treats latency and availability as goals rather than as things it reacts to. Next on the list are CtrlB's self-improving agents, which Blitz is being onboarded onto, and which Gaurav wants for the failure that has never happened before. The part he says other eval platforms do not have is application telemetry, which CtrlB holds beside agent telemetry on the same lake.
We were more involved in firefighting and a reactive approach. Now finally we are at a stage where we are proactively thinking about our infrastructure, platform and agents.
Gaurav Piyush · Co-founder, BlitzWhen the promise is a 10 minute delivery, every minute of an incident belongs to somebody's order. With CtrlB, Blitz's logs and traces sit in one searchable lake, an agent runs the first pass of the RCA and proposes the fix before anyone opens a dashboard, and the root cause arrives in minutes instead of days.
