When Latency Misses, Fix Retrieval Before You Quantize

When Latency Misses, Fix Retrieval Before You Quantize

When an agent or RAG path blows its latency budget, the first proposal is usually a smaller or quantized model. That is the visible cost. It is often the wrong layer.

We have timed production traces where the model was 280 ms and the retrieval join was 900 ms. Quantizing the model would have saved 80 ms and left the user waiting on the same database.

Measure the path before you change the weights

A useful trace has three clocks: retrieval, model, post-process. If you only log end-to-end time, every delay looks like "the LLM."

We require those three spans on every production agent call before anyone opens a quantization ticket. The ticket has to name the span that missed. If retrieval is 60 percent of the budget, the model is not the work.

On one support agent the p95 was 2.4 s. Retrieval of policy docs plus a CRM join was 1.7 s. The model was 450 ms. They had already planned a GPU upgrade. We added a 24-hour cache on the policy corpus and a single-key CRM lookup. p95 fell to 780 ms. The model did not change.

Quantization has a quality bill you will pay later

Smaller weights help when the model span is actually the miss, and when you have an eval set that can detect the quality drop. Without that eval, you will ship a faster, worse answer and call it a win until support tickets rise.

The tradeoff is real. An INT8 model that saves 200 ms and drops tool-call validity from 94 percent to 88 percent is not a latency fix. It is a silent product change.

We only quantize after retrieval is inside its budget and the eval set is green on the current model. Then we run the same eval on the quantized build. If tool-call validity drops more than 2 points, we keep the larger model and keep cutting retrieval.

Retrieval is usually a join problem

The slow span is rarely "vector search." It is the join after the search: three systems to build a prompt, a missing index, a cold embedding table, a filter applied in Python after 200 chunks come back.

Fixes that have mattered more than weight format:

• filter before you embed, not after
• cache the top corpus for the ten most common intents
• stop joining a system that is allowed to be 5 minutes stale; read a replica or a snapshot
• cap retrieved chunks at the number the prompt can use. Extra chunks cost milliseconds and tokens

A smaller model does none of that.

A failure mode with "optimize everything"

A team quantized the model, added a reranker, and turned on speculative decoding in the same week. Latency improved 12 percent. They could not say which change did it, and answer quality dropped on a subset they were not evaluating.

The mitigation is one change per release on the serving path, with the three-span trace and the eval set as the merge gate. If you cannot attribute the win, you cannot keep it.

Ownership

Retrieval owners fix indexes, caches, and joins. Model owners change weights and decoding. If those roles are the same person, the model still gets blamed first because it is more interesting.

We put the p95 budget on the retrieval owner until that span is under 40 percent of the total. Only then does the model owner get a quantization task.

Final takeaway

Split the trace. If retrieval is the long span, cache and join first. Quantize only after that span is inside budget and the eval still passes.