Migrate Off a Shared LLM Vendor Key Before You Change Models

Migrate Off a Shared LLM Vendor Key Before You Change Models

The risky LLM migration is not GPT to Claude. It is one shared vendor key in a platform account to keys and projects that map to real workflows.

We have seen teams spend a quarter on a model bake-off while every product surface still billed through a single secret in CI. They changed the model. They still could not say who spent the money, or turn one surface off without taking down the rest.

A shared key is a coupling you will feel during the incident

One key means one rate limit, one compromise, one invoice. A runaway eval job throttles checkout. A leaked GitHub variable exposes every agent. Finance gets a vendor PDF with no workflow column.

Before you migrate models, migrate identity. Each product surface gets its own project or key. Each call carries a workflow attribute. The old shared key is scheduled for deletion, not "kept for emergencies."

If a surface cannot get its own key this sprint, it stays on the shared key and is named on a public exception list. Exceptions older than 30 days are incidents.

Split traffic without splitting the product

Two migration styles.

Big bang: issue new keys, cut DNS or env vars, revoke the old key on Friday. Fast, and every missed caller fails together.

Strangler: new callers must use a new key. Old callers are wrapped by a proxy that injects the workflow tag and logs the caller. Each week the noisiest untagged caller is moved. The old key's rate limit is lowered until only the exception list remains.

We use the strangler unless the shared key has already leaked. After a leak, you do the big bang and accept the weekend.

On one account the proxy showed that 40 percent of tokens were a nightly embedding job with no owner. That job moved to its own key and a batch budget. The "we need a cheaper model" conversation ended.

Do not migrate the prompt store in the same change

Teams try to change the vendor, the key layout, and the prompt repository together. When something breaks, they cannot tell which lever did it.

Key migration is a routing and identity change. Model migration is a quality change. Prompt migration is a content change. One per release.

If you need the new vendor to get per-project billing, you can add the vendor under a new key without moving existing traffic. Dual-running for two weeks is cheaper than a combined cutover.

A failure mode with leftover SDK defaults

A service was moved to a new key in env. The SDK also read an old key from a well-known config file on the image. Half the traffic stayed on the shared key. The dashboard looked migrated. The invoice did not.

The mitigation is to fail boot if more than one vendor credential is present, and to log the last four characters of the key in use on every process start. A weekly scan greps images and secret stores for the revoked key's prefix.

What done looks like

The shared key returns 401. Every remaining caller has a named workflow and an owner. Token spend reports split by workflow. A single surface can be rate-limited without touching the others.

Until those four are true, you have not migrated. You have added a model.

Final takeaway

Split the key and tag the workflow first. Change models after you can turn one surface off without taking down the rest.