Latency budgets in agent loops

An agent loop runs against a deadline that was set before it started: the user's patience, an upstream timeout, a scheduler's slot. The loop does not know the deadline is there, and nothing in it is priced. We instrumented a production-shaped loop across three runs of the protocol and asked two questions — where does the time actually go, and when the budget runs short, what should a loop drop first?

This is the write-up of the concluded project. The protocol, every revision of it, and the trial data behind each number live in the repository; the log on the home page recorded each step as it happened.

What we asked

Revision 1 asked the naive question: how much of a loop's wall-clock is model time? The pilot answered it too fast — 61% on the first configuration we tried — and in doing so showed the question was wrong. Model time is the one cost everyone already watches. The interesting budget was the other 39%, which no one had itemised.

Revision 2, the one that held, asked instead: itemise the non-model time by mechanism, and test whether the cheapest-looking line items are load-bearing. The gate was pre-registered per hypothesis; a mechanism counted as droppable only if removing it cost less than 2 points of task success across the evaluation suite.

What we found

H1 — tool result serialization is a rounding error: supported. Under 3% of wall-clock in every trial, across payload sizes to 1 MB. Not worth an optimisation pass.

H2 — retry backoff is where the tail lives: supported. The p95 run spent 41% of its budget waiting between retries, against 9% at the median. The backoff schedule, not the model, is what a slow run is made of.

H3 — trimming context re-reads buys the budget back: refuted. This was the result we expected to headline the report, and it failed its gate cleanly. Aggressive context trimming saved 12% of wall-clock and cost 4.1 points of task success against a 2.0 point gate, across six trials. The re-reads were doing work. The saving is real; it is not free; we say so with the same prominence as the wins.

What to drop first

The ordering that survives the data, for a loop that finds itself past its budget: cap retry backoff before anything else, then shed optional tool calls, and leave context alone — it is the last thing to trim, not the first. The protocol carries the exact numbers behind each step.

What not to conclude

One loop shape, one model family, one evaluation suite. The 2-point gate was ours, and a product with a different tolerance would draw the droppable line elsewhere. Nothing here measures loops that parallelise tool calls, and the backoff result in particular should be re-run anywhere the provider's rate limits differ from ours.

If we had another run

Backoff was the largest untested lever left: H2 located the cost but no trial tested a schedule change against a gate. That trial is specified in the final revision of the protocol, unrun, for whoever picks it up — the repository is the handoff.

@misc{nullsilver2026latencybudgets,
  author = {Nullsilver},
  title  = {Latency budgets in agent loops},
  year   = {2026},
  month  = may,
  url    = {https://nullsilver.com/research/latency-budgets}
}