LLM gateway glossary: aliases, fallback, tokens, streaming and rate limits

Plain-English definitions of LLM gateway terms: OpenAI-compatible APIs, model aliases, fallback, BYOK, tokens, SSE streaming, rate limits and logs.

Glossary · Updated · by the Switchyard team

Definitions of the terms you meet when you put a gateway in front of language model providers, with notes on how each works in Switchyard. For code, see the API quickstart.

The basics

LLM gateway. A service between your application and model providers. Your app talks to one endpoint; the gateway handles provider keys, routing, limits and logs.

OpenAI-compatible API. An API that accepts and returns the same shapes as OpenAI's chat completions endpoint, so the OpenAI SDKs work by changing the base URL. Switchyard exposes POST /v1/chat/completions and GET /v1/models.

Provider connection. One provider account added to the gateway: Anthropic, OpenAI, or any service with an OpenAI-compatible chat completions API (for example a self-hosted vLLM server). Free allows 2, Pro 20.

BYOK (bring your own key). You use your own provider accounts and keys. The provider bills you directly; Switchyard adds no markup on tokens. Keys are encrypted at rest and never displayed again.

Gateway key. The key your application uses to call the gateway, separate from provider keys. Create one per app or environment and revoke it independently.

Routing

Model alias. A name such as fast that your code uses instead of a vendor model id. Behind it is an ordered list of provider and model targets.

Target. One provider plus model inside an alias. Switchyard allows up to five per alias.

Fallback. Trying the next target when one fails. Switchyard falls back after a 5xx, a 429, a timeout or a connection error, and returns 4xx errors directly. Our fallback strategy guide explains why.

Direct model call. Calling provider-name/model instead of an alias, when a request must use one exact model.

Request translation. Converting between API formats. Requests to Anthropic are translated: system messages become the system field, same-role turns are merged, stop becomes stop_sequences, and max_tokens defaults to 1024 when you don't send it.

Streaming

Streaming. Receiving the answer in pieces as it is generated, instead of all at once.

SSE (server-sent events). The HTTP format used for streaming: a text/event-stream response made of data: lines. OpenAI-style streams end with data: [DONE].

Chunk. One streamed piece in the chat.completion.chunk format: a role delta, content deltas, then a final chunk with finish_reason.

Mid-stream failure. A provider failing after content has been sent. Switchyard falls back only before the first content; after it, the stream ends with an error event.

Usage and limits

Token. The unit models count text in, roughly a few characters of English. Prompt tokens are what you send; completion tokens are what the model writes.

Usage reporting. Token counts the provider returns. If a stream has none, Switchyard estimates about 4 characters per token and marks the log entry.

Rate limit (RPM). Requests per minute allowed per gateway key: 60 by default, or a per-key override. Over it, the gateway returns 429 with retry-after.

Plan limit. Monthly requests and tokens per workspace (Free: 1,000 requests and 1M tokens; Pro: 200,000 and 200M). At the limit the gateway returns 402 with plan_request_limit or plan_token_limit.

Status 499. Logged when the client closed the connection; the provider request is stopped too.

Logs and data

Request log. One entry per request: time, key, requested model, provider, upstream model, status, attempts, latency and tokens. Kept 7 days on Free and 30 on Pro, exportable as CSV.

Content logging. Storing prompt and response text. Off by default in Switchyard; admins can turn it on for new requests.

Response headers. x-gateway-provider, x-gateway-upstream-model, x-gateway-attempts and x-gateway-request-id (which matches the log entry). See Switchyard for agencies for using them in client reports.

Try Switchyard free

One OpenAI-compatible endpoint in front of your own model provider keys: routing, fallback, rate limits, usage and logs. The Free plan needs no card; you can export or delete your data any time.