Routing and Reliability

Every provider is exactly one outage away from being your fallback.

Deploy, Gateway-Guarding AI Agent

Fun Fact: Trivia: OpenRouter started as a Discord bot

OpenRouter (founded 2023) began as a side project to let a single Discord community switch between GPT-4 and Claude without juggling API keys. By 2025 it routes over a billion tokens per day across 100+ models and bills it on a single invoice. The "unified billing across vendors" feature that LiteLLM Proxy now implements as a first-class concept started as a workaround for a server admin who got tired of expense reports.

63.2.1 LiteLLM Proxy: Unified Model Access

LiteLLM is an open-source library that provides a unified interface to 100+ LLM providers using the OpenAI API format. The LiteLLM Proxy Server extends this into a standalone gateway service that your applications call instead of calling LLM providers directly. It supports load balancing, fallbacks, spend tracking, rate limiting, and virtual API keys for multi-team environments.

# litellm_config.yaml - LiteLLM Proxy configuration
model_list:
 # Primary model with multiple deployments for load balancing
 - model_name: "gpt-4o"
 litellm_params:
 model: "openai/gpt-4o"
 api_key: "os.environ/OPENAI_API_KEY"
 rpm: 500 # Requests per minute limit
 tpm: 200000 # Tokens per minute limit

 - model_name: "gpt-4o"
 litellm_params:
 model: "azure/gpt-4o-eastus"
 api_key: "os.environ/AZURE_API_KEY"
 api_base: "https://my-eastus.openai.azure.com"
 rpm: 300

 # Fallback model (cheaper, faster)
 - model_name: "gpt-4o-mini"
 litellm_params:
 model: "openai/gpt-4o-mini"
 api_key: "os.environ/OPENAI_API_KEY"

 # Alternative provider (Anthropic via OpenAI-compatible API)
 - model_name: "claude-sonnet"
 litellm_params:
 model: "anthropic/claude-sonnet-4-20250514"
 api_key: "os.environ/ANTHROPIC_API_KEY"

# Router settings
router_settings:
 routing_strategy: "latency-based-routing"
 num_retries: 3
 retry_after: 5
 allowed_fails: 2 # Remove deployment after 2 consecutive failures
 cooldown_time: 60 # Re-check failed deployment after 60s

# Budget and rate limiting
general_settings:
 master_key: "sk-gateway-master-key"
 database_url: "postgresql://user:pass@db:5432/litellm"
 max_budget: 1000.0 # Monthly budget in USD
 budget_duration: "30d"

With the proxy running, your application code uses the standard OpenAI client library pointed at the gateway. No LiteLLM-specific code is needed in your application, which keeps the gateway concern cleanly separated from business logic.

import openai

# Point the standard OpenAI client at the LiteLLM Proxy
client = openai.OpenAI(
 base_url="http://litellm-proxy:4000/v1",
 api_key="sk-team-analytics-key", # Virtual key for cost tracking
)

# This request is transparently routed, load-balanced, and tracked
response = client.chat.completions.create(
 model="gpt-4o", # Logical model name, resolved by the proxy
 messages=[{"role": "user", "content": "Summarize this quarter's revenue."}],
 temperature=0.3,
)

# The proxy handles:
# 1. Routing to the fastest available gpt-4o deployment
# 2. Automatic retry if the first deployment fails
# 3. Token counting and cost attribution to the virtual key
# 4. Rate limiting per the team's allocated budget
print(response.choices[0].message.content)

63.2.2 Fallback Chains and Provider Failover

Provider outages are inevitable. OpenAI, Anthropic, Google, and Azure all experience periodic degradation or downtime. A production LLM application must handle these outages gracefully, ideally without the user noticing any disruption. AI gateways implement this through fallback chains: ordered lists of alternative models or providers to try when the primary option fails.

Fallback strategies range from simple to sophisticated. The simplest approach is a static ordered list: try OpenAI first, then Azure, then Anthropic. More advanced strategies consider the nature of the failure (rate limit vs. server error), the request characteristics (latency sensitivity, cost sensitivity, quality requirements), and real-time provider health metrics. Some gateways support "hedging," where the request is sent to multiple providers simultaneously and the fastest response wins.

from litellm import Router

# Configure a router with fallback chains
router = Router(
    model_list=[
    {
    "model_name": "primary-chat",
    "litellm_params": {
    "model": "openai/gpt-4o",
    "api_key": OPENAI_KEY,
    },
    },
    {
    "model_name": "primary-chat",
    "litellm_params": {
    "model": "anthropic/claude-sonnet-4-20250514",
    "api_key": ANTHROPIC_KEY,
    },
    },
    {
    "model_name": "primary-chat",
    "litellm_params": {
    "model": "google/gemini-2.5-pro",
    "api_key": GOOGLE_KEY,
    },
    },
    ],
    # Fallback configuration
    fallbacks=[
    {"primary-chat": ["fallback-chat"]},
    ],
    # Context window overflow handling
    context_window_fallbacks=[
    {"primary-chat": ["long-context-chat"]},
    ],
    routing_strategy="latency-based-routing",
    num_retries=2,
    retry_after=1,
    )

# The router automatically handles failover
async def resilient_completion(messages, **kwargs):
    """Make an LLM call with automatic multi-provider failover."""
    try:
        response = await router.acompletion(
            model="primary-chat",
            messages=messages,
            **kwargs,
            )
        return response
    except Exception as e:
        # All providers failed; log and raise
        logger.error(f"All providers exhausted: {e}")
        raise ServiceUnavailableError("LLM service temporarily unavailable")

Multi-Region Failover with Provider Region-Affinity

Provider outages are rarely global. When OpenAI's us-east-1 capacity is saturated at noon Pacific time, the eu-central-1 endpoint typically has headroom; the same is true in reverse during European business hours. Sophisticated gateway routing exploits this by giving each deployment a region tag and routing user traffic with regional affinity: a request from a user with detected latency under 40 ms to us-east goes to openai/gpt-4o-useast, a request from a Frankfurt user lands on openai/gpt-4o-eucentral, and the cross-region fallback only kicks in when the affinity-preferred region returns 5xx or exceeds an SLO-based latency budget.

In LiteLLM this is expressed by declaring multiple deployments under the same logical model name, each tagged with a region in the deployment metadata, and using either tag-based routing or the latency-routing strategy combined with sticky per-user session affinity. The cost-of-data-egress argument matters: routing a user in Frankfurt to a us-east endpoint is correct for reliability but doubles the round-trip latency and may violate data-residency commitments. Production gateways separate the preferred region list from the permitted region list, so that a Frankfurt user can fail over to Dublin but never to Virginia when contractual data-residency is in force. The same machinery covers self-hosted vLLM clusters split across availability zones.

63.2.3 Rate Limiting and Request Management

LLM providers impose rate limits measured in requests per minute (RPM) and tokens per minute (TPM). Exceeding these limits results in 429 errors that degrade user experience. An AI gateway centralizes rate limit management, ensuring that aggregate traffic from all application instances stays within provider limits. This is especially important in multi-team environments where several applications share the same provider account.

Gateway-level rate limiting operates at two levels. Upstream limiting ensures that the total traffic to each provider stays within that provider's rate limits. The gateway tracks request and token rates per deployment and queues or rejects requests that would exceed limits. Downstream limiting enforces per-team or per-user quotas, ensuring fair resource allocation across internal consumers. Virtual API keys enable this: each team gets a key with specific RPM, TPM, and budget limits.

import asyncio
from dataclasses import dataclass, field
from collections import defaultdict
import time

@dataclass
class RateLimitConfig:
    requests_per_minute: int = 60
    tokens_per_minute: int = 100000
    max_budget_usd: float = 500.0
    budget_period_days: int = 30

class GatewayRateLimiter:
    """Token-aware rate limiter for AI gateway traffic."""

    def __init__(self):
        self.team_configs: dict[str, RateLimitConfig] = {}
        self.request_windows: dict[str, list[float]] = defaultdict(list)
        self.token_windows: dict[str, list[tuple[float, int]]] = defaultdict(list)
        self._lock = asyncio.Lock()

    async def check_rate_limit(
        self, team_id: str, estimated_tokens: int
        ) -> bool:
        """Check if a request is within rate limits. Returns True if allowed."""
        config = self.team_configs.get(team_id, RateLimitConfig())
        now = time.monotonic()
        window_start = now - 60 # 1-minute window

        async with self._lock:
            # Clean old entries
            self.request_windows[team_id] = [
                t for t in self.request_windows[team_id] if t > window_start
                ]
            self.token_windows[team_id] = [
                (t, n) for t, n in self.token_windows[team_id]
                if t > window_start
                ]

            # Check RPM
            if len(self.request_windows[team_id]) >= config.requests_per_minute:
                return False

                # Check TPM
                current_tokens = sum(
                    n for _, n in self.token_windows[team_id]
                    )
                if current_tokens + estimated_tokens > config.tokens_per_minute:
                    return False

                    # Record this request
                    self.request_windows[team_id].append(now)
                    self.token_windows[team_id].append((now, estimated_tokens))
                    return True

# litellm_config.yaml:
# general_settings:
# master_key: sk-admin-key
# model_list:
# - model_name: gpt-4o
# litellm_params:
# model: openai/gpt-4o
# rpm: 100
# tpm: 500000
#
# Start: litellm --config litellm_config.yaml
# Then create per-team keys via the admin API:
import litellm
key = litellm.create_key(
 models=["gpt-4o"], max_budget=500.0, budget_duration="30d", tpm_limit=100000,
)
# All rate limiting and budget enforcement is handled by the proxy

63.2.4 OpenRouter: The Commercial Aggregator

OpenRouter is a commercial multi-model API that predates and, in several places, inspired the patterns LiteLLM Proxy later adopted. The pitch is operationally simple: OpenRouter holds the API keys for every major provider, exposes one OpenAI-compatible endpoint, presents one unified billing surface (per-million-tokens, vendor-by-vendor pricing rolled into a single invoice), and lets the caller choose a model by string identifier such as anthropic/claude-sonnet-4 or openai/gpt-4o. From the application's point of view it is the gateway and the billing arrangement at the same time.

The trade-off is who holds the contract. OpenRouter charges a small markup (typically a few percent) above the underlying provider price in exchange for unified billing, per-model price discovery, and a routing layer that already speaks every provider's quirks. For startups that do not want to negotiate enterprise contracts with five vendors, that is a clean win. For larger users, the markup eventually exceeds the cost of running LiteLLM Proxy in front of direct contracts with each vendor, which is why the production pattern often migrates from "OpenRouter as the gateway" to "LiteLLM Proxy as the gateway, with direct provider contracts" sometime between 50 and 500 thousand dollars of monthly LLM spend. Many of the routing-strategy ideas that ship in LiteLLM (latency-based ordering, per-model fallbacks, deployment health scoring) were observable in OpenRouter's user-facing behavior before they were documented as best practice.

63.2.5 Portkey: Gateway with Observability and Guardrails

Portkey is a commercial gateway alternative whose differentiator is what sits next to the proxy: a built-in observability stack, a guardrails layer, and a prompt-management UI. The routing primitives overlap heavily with LiteLLM (virtual keys, fallback chains, budget enforcement, semantic caching), so the choice between them rarely turns on routing capability. It turns on whether the team wants to run its own metrics pipeline and prompt store or pay Portkey to host them.

Portkey's guardrails are the more interesting wedge. The gateway can be configured to refuse outbound requests containing PII (pre-flight), or to refuse inbound responses containing flagged content (post-flight), without the application implementing any of that logic. For teams that have not yet built their own content moderation stack, this collapses several months of work into a configuration change. The risk, the standard one for any vendor-hosted policy enforcement, is that the policies become invisible: the team that adopts hosted guardrails should still maintain a small set of integration tests that exercise the boundary, so that the day the vendor's defaults change is not the same day a customer notices the change.

63.2.6 Cloudflare AI Gateway: Edge-Deployed Routing

Cloudflare AI Gateway exploits a different leverage point: it runs at Cloudflare's edge PoPs, which means a cached response can be returned from a location within roughly 50 ms of the end user almost anywhere on the planet. For applications whose traffic is dominated by repeat queries (status pages, FAQ bots, public documentation assistants), the gateway pays for itself not through routing intelligence but through cache geography. The cache lives at the edge; the origin call to OpenAI or Anthropic only happens on a miss.

The architectural trade-off is the reverse of Portkey's: Cloudflare AI Gateway has comparatively thin routing logic (one provider per route, simple fallbacks) but unmatched edge caching and analytics. The natural composition is to put Cloudflare AI Gateway in front of your origin LiteLLM Proxy: Cloudflare handles cache and bot mitigation at the edge, LiteLLM handles routing and budget enforcement at the origin. For a deeper treatment of the cache layer itself, see Section 63.3.

pip install 'litellm[proxy]' && export OPENAI_API_KEY=sk-... && export ANTHROPIC_API_KEY=sk-ant-...

Exercises