LLM fallback strategy: how to route around provider errors without surprises

When to fall back to another model, which errors should and should not trigger it, how streaming changes things, and how to test a fallback chain.

Guide · Updated · by the Switchyard team

In short

Fall back to another model on provider outages, rate limits (429) and timeouts, but not on 4xx errors caused by the request itself, such as a 400 for an invalid request. With streaming, fallback is only possible until the first content reaches your client, so test each chain with forced failures.

Every model provider has bad minutes: overloaded endpoints, rate limits, slow responses. A fallback chain lets your product keep answering by trying another target when the first one fails. Done carelessly, it also hides bugs and swaps in a model your prompts were never tested with. This guide covers the decisions that matter, using Switchyard's aliases as the example.

Which failures should trigger a fallback

Not every error means "try someone else":

ResponseFall back?Why
5xx from the providerYesThe provider failed; another may succeed
429 rate limitedYesYour quota or their capacity is exhausted for now
Timeout or connection errorYesNo answer is coming
400 or another 4xxNoThe request itself is wrong; another provider would usually reject it too

Switchyard follows exactly this rule: it moves to the next target after a 5xx, a 429, a timeout or an unreachable provider, and returns 4xx errors to you straight away. Falling back on a 400 would only multiply a bug across providers and hide it from your logs.

Choosing targets

An alias in Switchyard has up to five targets, tried in order. Good chains share three properties:

  1. Similar capability. If your prompt relies on long context or strict JSON output, every target must handle it.
  2. Different failure domains. Two models from the same provider often fail together. Put a different provider second.
  3. Tested prompts. Run your evaluation set against every target in the chain, not just the first.

Two or three targets are usually enough. A long chain mostly adds latency on bad days.

Timeouts

A fallback is only as fast as the timeout that triggers it. If the timeout per attempt is too long, users wait for the first provider to give up; too short, and you abandon answers that were about to arrive. Set it from your own latency data: look at the latency column in the logs for successful requests and pick a value comfortably above what most requests take. The timeout per attempt is set under Limits & logging.

Streaming changes the rules

With streaming, the first tokens may already be on the user's screen when a provider fails. Switching to another model mid-answer would produce a stitched-together reply, so Switchyard only falls back until the first content reaches your client. After that, a provider failure or a stall longer than your workspace timeout ends the stream with an error event in the usual error shape (and no [DONE]). Your client should handle that event, for example by showing "The answer was interrupted, try again". The routing and fallback help article and the quickstart describe the stream format.

Make fallbacks visible

A fallback that nobody notices is a problem waiting to grow. Use what the gateway returns:

  • The response header x-gateway-attempts shows how many targets were tried.
  • x-gateway-provider and x-gateway-upstream-model show who actually answered.
  • The logs record attempts per request, so you can filter for requests that needed more than one.

If fallbacks become frequent, treat it as an incident with your first provider, or reorder the chain.

Rate limits on your side too

Fallback protects you from provider limits; per-key limits protect providers (and your bill) from you. Give each environment its own gateway key with a requests-per-minute cap. A runaway retry loop in staging then hits a 429 from the gateway instead of exhausting your production quota. See rate limits and content logging for how the per-minute window works.

Testing the chain

  1. Create a test alias whose first target uses a deliberately invalid model name. Confirm you get a 4xx straight back: no fallback, as designed.
  2. Disable the first target's provider and send a request; disabled providers are skipped, so the second target should answer.
  3. Send a streaming request and cancel it mid-way; the log shows status 499 and the provider request is stopped.
  4. Check the headers and the log entry for each case.

A note on cost

Fallbacks can move traffic to a pricier model without anyone deciding to. Check usage per model after an incident. With your own keys, each provider bills you directly for what it served.

Try Switchyard free

One OpenAI-compatible endpoint in front of your own model provider keys: routing, fallback, rate limits, usage and logs. The Free plan needs no card; you can export or delete your data any time.