Token Spend Chargeback: Put LLM Cost on the Teams That Create It

Token Spend Chargeback: Put LLM Cost on the Teams That Create It

Most engineering orgs now have a shared LLM bill and no owner for the line items. The model key lives in a platform account. Product teams call it from agents, support tools, and internal copilots. Finance sees one vendor invoice.

This is the same problem we already solved for cloud compute, except the unit is tokens instead of vCPU hours, and the waste shows up as retries and bloated context rather than idle instances.

Shared keys hide the only number that matters

A single API key is convenient. It is also why token spend chargeback never starts. Without a caller identity, you cannot tell a support summarizer from a nightly evaluation job that re-embeds the whole corpus.

We issue one key or project per product surface, then require a workflow attribute on every call: support-summarize, agent-plan, eval-regression. The billing export has to join on that attribute, not on the model name. Model names change. Workflows do not.

If a call arrives without workflow, it is rejected. That sounds strict. It is cheaper than a month of unattributed traffic.

Charge the workflow, not the model

Teams often want to allocate by model (gpt-x vs a local model) because the unit price is obvious. That allocation breaks the first time you swap models or add a cheap fallback.

The durable unit is the workflow. A support summarizer that burns 800k tokens a day is a product cost whether it hits a hosted model or an internal one. An eval suite that runs twice a week is a platform cost.

We keep two ledgers:

• product workflows: charged to the owning team at the blended token rate for that month
• platform workflows: evals, safety filters, and shared retrieval, held on the AI platform budget

Blended rate is intentional. If every team is billed the exact vendor price of whichever model they hit, they optimize for the cheapest model instead of the right one. The platform can still steer traffic; the chargeback should not fight that.

Caps beat after-the-fact reports

A monthly PDF of token spend does not change behavior. A daily cap does.

We set a soft cap at 80 percent of the workflow's monthly token budget and a hard cap at 100 percent for non-customer-facing jobs. Customer-facing workflows degrade instead of failing: they shorten context, skip optional tools, or fall back to a smaller model. Batch evals stop.

One client had an agent planner that retried failed tool calls with the full conversation attached. Token volume doubled in nine days. The hard cap on agent-plan stopped the loop on day two of the next month, before the invoice did.

A failure mode with cached prompts

Prompt caches look like savings until chargeback uses uncached token counts. We have seen reports that over-allocated a retrieval workflow by about 35 percent because the export counted every cache-write as new input.

The mitigation is to bill on billable_tokens from the vendor, not on prompt length. If the vendor does not expose that field, estimate cache hits from identical prefix hashes and hold the difference on the platform budget rather than guessing.

Ownership that survives a model swap

Assign one owner per workflow: the person who can change the prompt, the retrieval set, and the retry policy. Platform owns keys, routing, and the cap. Product owns the workflow budget.

When a model is swapped, the owner does not change. The budget does not reset. Only the blended rate moves. That is the point of charging the workflow.

Final takeaway

Stop allocating the LLM invoice by model or by shared key. Tag every call with a workflow, cap that workflow, and charge the team that can actually change the prompt.