Designing an agent loop that survives production

Almost every agent that dies in production dies the same way: it was written as a single function that runs for eleven minutes, and something on minute nine went wrong. The model was fine. The loop was the problem.

A loop that survives production is not smarter than one that does not. It is just built from steps that can each fail independently, and it writes down where it got to.

Make every step resumable

The unit of work is not the run, it is the step. After each one, write the state somewhere durable — what you did, what came back, what you plan to do next. If the process dies, the next worker reads that and continues.

This sounds like bookkeeping until the first time an eleven-minute run fails at minute nine and resumes at minute nine instead of minute zero. Then it sounds like the only sane way to build.

The unit of work is the step, not the run.The one rule worth internalising

Separate retryable from fatal

Three different things get called "failure" and they want three different responses:

• Transient — a timeout, a 429, a flaky network. Retry with backoff. Most failures are this.
• Invalid — the model passed an argument your tool cannot accept. Do not retry blindly; hand the validation error back to the model as an observation and let it correct itself.
• Fatal — the account is closed, the record does not exist. Stop. Park the run with the error attached and tell a human.

Collapsing these into one catch is why agents loop forever burning tokens on an error that was never going to resolve.

Budget the loop, not just the call

Give every run a ceiling on steps, wall-clock time, and money — and enforce all three. An agent with no step budget will happily try the same broken approach forty times. The ceiling is not a cost control, it is a correctness control: hitting it is a signal that the task was underspecified.

Let it park, not die

When a run exhausts its budget or hits a fatal error, the wrong move is to throw. The right move is to park: keep the state, attach the reason, and surface it for a human. Nearly every parked run is either resumable after a small fix or a genuine bug report about your prompt.

Write the trace as you go

Record each step's prompt, arguments, result, tokens, and duration at the moment it happens, not at the end. Runs that crash are exactly the ones you most need the trace for, and a trace assembled after the fact never exists for the run that mattered.

Do these four things and the loop stops being the interesting part of your system — which is the goal. The interesting part should be the tools.