Token Budgets Belong on the Workflow, Not the Model

Token Budgets Belong on the Workflow, Not the Model

We started putting workflow token budgets in place after a client swapped models mid-project and their monthly LLM spend jumped 3x in two weeks. The old caps sat on individual calls. They did not survive the change.

Workflow token budgets track total tokens across every step in a single run. That single change kept costs inside the original forecast even after the model swap.

Model call caps hide the real spend

Teams often set token limits on each LLM call. The limits feel precise until a retry loop or a multi-step agent burns through the budget without any single call hitting the cap.

In one case we reviewed, a workflow made 12 calls. Each stayed under 4,000 tokens. The total reached 47,000. The per-call guard never triggered.

Workflow token budgets survive model changes

We now attach the budget to the workflow identifier instead of the model endpoint. The budget travels with the job even if the underlying model or provider changes.

A typical setting we use is 50,000 tokens per workflow run with an 80 percent warning threshold. Once the warning fires, the workflow can still finish but new branches are blocked.

Hard stops versus soft throttling

Two approaches appear most often in our work.

Hard stops cut the workflow the moment the budget is reached. They protect the spend ceiling but can leave partial results that downstream systems must handle.

Soft throttling reduces model temperature or switches to a smaller model for remaining steps. They keep the workflow moving but require extra routing logic and can still exceed the budget if the cheaper model is slower to converge.

We choose hard stops for financial reporting workflows and soft throttling for internal search agents where partial output still has value.

Retry storms and budget guards

One failure mode we see repeatedly is retry storms after a rate limit or transient error. Each retry adds tokens without new progress.

The mitigation we apply is a per-workflow retry token bucket set at 15 percent of the total budget. Once that bucket is empty, further retries are rejected and the workflow fails cleanly with a logged reason. This keeps the storm from consuming the main budget.

Observability that shows actual consumption

We expose three numbers on every completed workflow: total tokens used, share spent on retries, and share spent after the first model call. These numbers go into the same dashboard as cost and latency.

Over successive weeks the data shows which workflow steps consistently exceed their share. That pattern drives the next round of prompt or routing changes rather than blanket increases to the overall budget.

Final takeaway

Attach the token budget to the workflow identifier and enforce it before any model call is made.