Caching and Cost Management

A bill no one is watching is a bill that grows without permission.

Deploy, Gateway-Guarding AI Agent

Fun Fact

Mental Model: A semantic cache is a poor man's distillation

Semantic caching with a 0.95 similarity threshold turns your most-frequent live prompts into a free, zero-training distillation of the production model into a cheap lookup. Every cache hit is a question that the big model has already answered, at marginal cost zero. The catch: cached answers go stale with the corpus, and "the model would have answered differently today" is a hard failure to detect without periodic invalidation, which is why the canonical pattern is "TTL of one week, force-invalidate on model upgrade".

63.3.1 Semantic Caching for LLM Responses

Traditional API caching uses exact key matching, but LLM requests rarely repeat exactly. Two users might ask "What is the capital of France?" and "What's France's capital city?" and expect the same answer. Semantic caching uses embedding similarity to identify semantically equivalent requests and serve cached responses. This can reduce costs significantly for applications with repetitive query patterns, such as customer support or FAQ bots.

The trade-off is cache freshness versus cost savings. Semantic caching works well for factual queries with stable answers, but poorly for queries that depend on real-time data, user context, or conversation history. Most gateway implementations allow you to configure which requests are cache-eligible based on the model, temperature (only cache deterministic calls with temperature=0), and custom metadata.

Provider-side prompt caching is a separate, complementary layer worth flagging explicitly: Anthropic's cache_control blocks (Aug 2024), OpenAI's automatic prompt caching on long prefixes (Oct 2024), Google's Gemini explicit context caching, and AWS Bedrock prompt caching all give 50-90% input-token discounts on the shared prefix when the same long context is reused across requests. Unlike semantic caching, provider caching is exact-match on the prefix and is the right first lever for any RAG or agent system that resends a large system prompt or document context many times; semantic caching applies on top for paraphrased user-side queries.

import time
import numpy as np
from typing import Optional
class SemanticCache:
    """Embedding-based semantic cache for LLM responses."""
    def __init__(self, embedding_model, similarity_threshold=0.95):
        self.embedding_model = embedding_model
        self.threshold = similarity_threshold
        self.cache: list[dict] = [] # In production, use a vector DB
    async def get(self, messages: list[dict]) -> Optional[str]:
        """Look up a semantically similar cached response."""
        query = self._extract_query(messages)
        query_embedding = await self.embedding_model.embed(query)
        best_score = 0.0
        best_response = None
        for entry in self.cache:
            score = self._cosine_similarity(
                query_embedding, entry["embedding"]
                )
            if score > best_score:
                best_score = score
                best_response = entry["response"]
                if best_score >= self.threshold:
                    return best_response
                return None
    async def put(self, messages: list[dict], response: str):
        """Cache a response with its embedding."""
        query = self._extract_query(messages)
        embedding = await self.embedding_model.embed(query)
        self.cache.append({
            "query": query,
            "embedding": embedding,
            "response": response,
            "created_at": time.time(),
            })
    def _extract_query(self, messages: list[dict]) -> str:
        """Extract the cacheable query from messages."""
        return messages[-1]["content"]
    def _cosine_similarity(self, a, b) -> float:
        return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))

from qdrant_client import QdrantClient, models

client = QdrantClient(":memory:") # or url="http://qdrant:6333"
client.create_collection("llm_cache",
    vectors_config=models.VectorParams(size=1536, distance=models.Distance.COSINE))

hits = client.search("llm_cache", query_vector=query_embedding, limit=1,
                     score_threshold=0.95)
cached = hits[0].payload["response"] if hits else None

63.3.2 Cost Tracking and Budget Enforcement

One of the most valuable functions of an AI gateway is centralized cost tracking. Every request flowing through the gateway carries token count and model information, allowing precise cost calculation. The gateway can enforce budgets at multiple levels: per team, per project, per user, and globally. When a budget threshold is approached, the gateway can send alerts, switch to cheaper models, or reject non-critical requests.

Cost tracking at the gateway level complements the OTel-based cost attribution in Section 44.4. While OTel provides per-request cost visibility for debugging and analysis, the gateway provides real-time enforcement: it can block a request before it is sent to a provider, preventing budget overruns rather than merely reporting them after the fact. The combination of gateway-level enforcement and OTel-level visibility provides complete cost control.

from datetime import datetime, timedelta

class BudgetEnforcer:
    """Enforce spending limits per team with tiered responses."""

    def __init__(self, db):
        self.db = db

async def check_and_record(
    self, team_id: str, model: str, estimated_cost: float
    ) -> dict:
    """Check budget, record spend, return routing decision."""
    team = await self.db.get_team(team_id)
    period_start = datetime.utcnow() - timedelta(days=team.budget_period_days)
    current_spend = await self.db.get_spend(team_id, since=period_start)

    remaining = team.max_budget - current_spend
    utilization = current_spend / team.max_budget

    # Tiered response based on budget utilization
    if utilization >= 1.0:
        return {
            "action": "reject",
            "reason": f"Budget exhausted (${current_spend:.2f}/${team.max_budget:.2f})",
            }
    elif utilization >= 0.9:
        return {
            "action": "downgrade",
            "target_model": team.budget_fallback_model,
            "reason": f"Budget >90%, routing to {team.budget_fallback_model}",
            }
    elif utilization >= 0.75:
        return {
            "action": "allow",
            "warning": f"Budget at {utilization:.0%}. ${remaining:.2f} remaining.",
            }
    else:
        return {"action": "allow"}

async def record_spend(self, team_id: str, model: str, cost: float):
    """Record actual spend after a successful response."""
    await self.db.record_transaction(
        team_id=team_id,
        model=model,
        cost=cost,
        timestamp=datetime.utcnow(),
        )

Prompt-Budget Patterns and Back-Pressure

The tiered budget enforcer above operates on dollars per period. A complementary pattern operates on tokens per user per period: a prompt budget. The motivation is asymmetric. A team budget protects the company from a runaway service; a prompt budget protects the service from a runaway user. The fintech anecdote from Section 63.1 (one user consuming 30% of spend as a homework tutor) is the textbook case where a per-user token budget would have caught the anomaly at hour one rather than month three.

The mechanism is a token bucket per user identity, refilled at a configured rate (for example, 500 thousand tokens per day for free-tier users, 5 million for paid users) and decremented at request time by the estimated input plus generated tokens. When the bucket runs dry, the gateway has three options. Hard reject returns a 429 with a retry-after header; the simplest correct behaviour for transactional APIs. Soft reject downgrades the user to a cheaper model (the same downgrade lever from the team-budget enforcer, repurposed against the user dimension). Back-pressure intentionally introduces latency: the gateway accepts the request but holds it for a few seconds before forwarding, which throttles loop-style abuse without breaking the user experience for genuine human-pace usage.

Back-pressure is the underused option. Loops that issue 50 requests per second collapse under a 3-second injected delay (their throughput drops to 0.3 RPS), while users typing prompts by hand never notice 3 seconds added to the second-or-longer LLM call they were already going to wait for. The gateway is the right place to implement this because it sits at the choke point between application and provider and already knows the user identity from the virtual key. Application-layer back-pressure cannot reliably enforce limits because every replica enforces its own, and the user simply round-robins across replicas.

Model-Version Pinning Across Vendors

Every major provider deprecates models on a rolling schedule. OpenAI announces a sunset for gpt-4-turbo-2024-04-09, Anthropic retires a Claude snapshot, Google rotates a Gemini point release. When that happens, every application calling the deprecated string by name starts failing at the deprecation date, regardless of how long the team has known about the change. Without a gateway, the migration is an N-application coordination problem; the prompts, retrieval indexes, and evaluation harnesses calibrated against the old snapshot all have to migrate at once.

The gateway pattern is to pin the logical model name to a specific vendor snapshot in the gateway config, decoupled from the version the application requests. Application code calls model="quality-tier-a"; the gateway resolves it to openai/gpt-4o-2024-08-06 today and to anthropic/claude-sonnet-4-20250514 next month when the OpenAI snapshot is deprecated. The application never knew the underlying snapshot existed. The gateway log records every resolution, so the team can audit which application traffic was on which snapshot when.

The harder variant is when the change is not a like-for-like swap: the new snapshot has different output formatting, a more restrictive content policy, or a smaller context window. The gateway's contract with the application is identical, but the gateway operator now owns a small compatibility shim that, for example, reformats the structured-output JSON to match what the old snapshot produced. This shim is the price of decoupling, and it is exactly the kind of code that belongs in a gateway and would be terrible in every application separately. Pair this with a thin set of integration tests that exercise the contract; the day the shim itself starts diverging from reality is the day you want a test to tell you, not a customer.

Exercises