Request Budgets for Synchronous Agent Calls

Request Budgets for Synchronous Agent Calls

An agent that plans, calls tools, and retries is a batch job. Putting that loop on a user request is how a 200 ms page becomes a 4 second spinner.

We see this when a support or checkout flow "just asks the agent." The first demo is impressive. The p95 is not. The request budget has to be written down before the agent is wired to the route.

The request path is not an agent runtime

A synchronous HTTP request has a deadline. For an authenticated product action we use 800 ms at p95, 1.5 s at p99, including network. An agent that needs two tool calls and a model round trip will miss that on a good day.

If the answer is allowed to be stale by minutes, precompute it. If the answer must be fresh and cheap, do not use an agent. Use a rule, a retrieval hit, or a cached classification.

The agent belongs on the request path only when a human is already waiting on a conversation, and even then the first token has to start inside the budget.

Precompute versus fail over

Two options when the agent cannot finish in time.

Precompute: run the agent on a queue when the underlying data changes. The request reads the last approved result. Freshness is the age of that result, which you publish.

Fail over: the request hits a deterministic path — a rules engine, a keyword router, a human queue — and the agent writes a better answer later.

We precompute for support macros and product FAQs. We fail over for anything that charges a card or changes entitlement. A late agent suggestion is fine. A late charge is not.

On one integration the team kept the agent on the checkout "explain this fee" button. Median was 1.1 s. p95 was 3.8 s because a tool timed out and retried with the full transcript. Moving the explanation to a precomputed note, refreshed on price-list change, put the button under 120 ms. The agent still writes the note. It just does not sit in the click.

Tool calls need their own clocks

If you must call the agent in-request, give each tool a timeout that fits the remaining budget. A 2 second tool inside an 800 ms request is a guaranteed miss.

We budget 200 ms for retrieval, 350 ms for the model, and nothing for a second tool. A second tool is a queue job. The response can say the extra lookup is in progress.

Retries are not free. One retry of a 350 ms model call blows the budget by itself. We disable automatic retries on the request path. Failures go to the fallback.

A failure mode with "just one more tool"

An agent planner decided a missing SKU required a second catalog lookup. That lookup was correct and late. The user saw a spinner, refreshed, and created a second in-flight agent run.

The mitigation was a hard tool-count of one on synchronous calls, plus an idempotency key on the conversation turn. A refresh joins the first run or hits the fallback. It does not start a sibling loop.

What we measure

Track time-to-first-token and time-to-fallback, not only "agent success rate." A 99 percent success rate at 3 seconds is a failed request path.

Also count how often the fallback is used. If it is above 20 percent, the agent should not be on that route.

Final takeaway

Write the request budget first. If the agent cannot finish inside it with one tool and no retries, take it off the click and put it on a queue.