Observability, Monitoring, and Drift Detection

"My latency spiked, my cost doubled, and my outputs got shorter. Three traces, three explanations, three teams to convince. Observability is the receipt you hand to each of them."

An OpenTelemetry-Emitting AI Agent

44.3.1 The Three Pillars, LLM Edition

Classical observability has three pillars: metrics (cheap numerical aggregates), traces (per-request causal chains), and logs (high-cardinality structured events). LLM systems use all three, but the content and cardinality of each shift in ways that break naive Prometheus dashboards built for stateless REST services.

Metrics gain new dimensions: input tokens, output tokens, cache-hit ratio, and dollar-cost per request. Latency distributions stop being unimodal; for streaming endpoints you track time-to-first-token (TTFT) and inter-token latency separately, with p50, p95, and p99 percentiles that often differ by an order of magnitude (long-tail prompts can take 50x the median). Traces have to capture the prompt, the rendered template, the retrieved documents, every tool call, and the final response; a single user-facing turn in an agent can fan out to 30+ spans. Logs have to redact PII at the SDK level because the prompt body is now part of the payload, and prompt strings exceed the line limits of most log shippers.

44.3.2 OpenLLMetry and the GenAI Semantic Conventions

Until 2024, every observability vendor invented its own attribute names: was the token count llm.usage.total_tokens, gen_ai.usage.tokens, or tokens_used? The OpenTelemetry community resolved this with the GenAI semantic conventions, which moved from experimental to stable across 2024-2025. The conventions define a span schema covering provider, model, request parameters, prompt and completion content, token counts, finish reason, and tool calls.

OpenLLMetry, maintained by Traceloop, is the reference SDK implementation. It is OpenTelemetry under the hood, ships with auto-instrumentors for OpenAI, Anthropic, Bedrock, Vertex AI, Cohere, LangChain, LlamaIndex, Haystack, and most vector stores, and emits the standard GenAI attributes by default. Because it is plain OpenTelemetry, you point it at any OTLP-compatible backend (Phoenix, Honeycomb, Datadog, Grafana Tempo, Jaeger), not a proprietary one. Auto-instrumentation looks like this:

from traceloop.sdk import Traceloop
from openai import OpenAI

# One-line instrumentation: every OpenAI call now emits OTel spans
# with prompt, completion, token counts, and latency.
Traceloop.init(
    app_name="support-bot",
    api_endpoint="https://api.traceloop.com",  # or your own OTel collector
    disable_batch=False,
)

client = OpenAI()
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Summarize the refund policy."}],
)
# Span captured: gen_ai.system=openai, gen_ai.request.model=gpt-4o,
# gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, latency_ms,
# gen_ai.response.finish_reasons, plus the rendered prompt and response.

Production teams typically run a local OTel Collector as a sidecar, dual-writing spans to a long-term store (S3 or a managed service) and a hot store (Phoenix or Tempo) for interactive debugging. This pattern survives vendor swaps because the SDK and the span format are open.

44.3.3 The Tool Landscape

Five tools cover the bulk of the production market in 2025-2026. They differ on three axes: self-hostability, depth of LLM-specific analysis, and whether they replace your existing observability stack or sit alongside it.

Arize Phoenix (open source, Apache 2.0) is an OTel-native, locally runnable trace viewer with built-in LLM eval views (groundedness, relevance, toxicity scores per span). It is the right default for teams that want to own their data and have an existing OTel collector. Phoenix ships with Jupyter integration so you can pull traces into a notebook and slice them with pandas.

LangSmith from LangChain is the deepest tool if you are already building on LangChain or LangGraph. It captures the framework-internal state (graph node transitions, tool routing decisions, memory updates) that generic OTel spans miss. SaaS-first; the self-hosted tier is enterprise-licensed. As of 2025 LangSmith works with non-LangChain code via its tracing SDK but the value drops sharply outside the LangChain stack.

Helicone takes a different architectural stance: it is an LLM gateway proxy. You change your base URL from api.openai.com to oai.helicone.ai/v1 and pass a Helicone API key; every request flows through their proxy, which logs the full prompt and response, deduplicates with a request cache, and exposes per-user cost dashboards. The proxy model trades a small added latency for zero application-side code changes, which is decisive when you are instrumenting a legacy codebase or third-party SaaS that you cannot modify. Helicone open-sourced its core in 2024 and self-hosting is a documented path.

Pydantic Logfire is the newer entrant (GA in 2024) from the Pydantic team. It is a managed OTel backend with first-class Python ergonomics: logfire.instrument_openai() and logfire.instrument_anthropic() produce dashboards without any other configuration. Logfire is the easiest on-ramp for FastAPI-heavy stacks and integrates cleanly with Pydantic AI agents.

Langfuse is a self-hosted-first competitor to LangSmith with an MIT-licensed core. As of 2025 it offers prompt management (versioned templates with A/B routing), eval-in-prod sampling, and per-session timelines. Langfuse is the standard answer when an enterprise will not accept SaaS-only data residency.

44.3.4 Practical Vendor Cases

Three named 2024-2025 cases ground the choices. Shopify, in a 2024 AI Engineer Summit talk, migrated Shopify Magic from a custom logging stack to OTel + Phoenix because the SDK-neutral standard let them swap model providers without touching observability code. Notion's 2024 engineering blog documented using Helicone's caching proxy to cut Anthropic spend by ~30% on retrieval-augmented summaries. Cursor credited LangSmith with debugging autocomplete tool-routing drift after a mid-2024 model upgrade, citing LangGraph node-transition visibility. For server-side metrics (GPU utilization, KV cache pressure, batch size) see vLLM & Inference Servers (vLLM and TGI Prometheus exporters); join them to the application traces in this section by request ID.

44.3.5 The Three Drift Modes

Classical ML monitoring assumes the model is fixed and the data shifts around it. LLM systems break that on both sides: the model can change (when a vendor updates GPT-4o behind the same endpoint) and the data can change (when users adapt their prompting style). Three drift modes apply:

Prompt drift: the distribution of user queries shifts. A support bot trained on "reset my password" starts seeing "your December invoice is wrong"; a code assistant for Python starts seeing TypeScript. Detect it via embedding distributions, topic clusters, or per-token-budget shifts over rolling windows.

Response drift: the model's output behavior changes for fixed inputs. This is the new failure mode in 2024-2026, almost entirely caused by vendor-side model updates. When OpenAI replaces gpt-4o-2024-08-06 with a newer snapshot under the floating gpt-4o pointer, your prompts may produce 20% more tokens, different formatting, or new refusal patterns. Detect it with a fixed golden-set replay: a curated 50-500 prompt set called hourly and diffed against the previous run.

Quality drift: faithfulness, groundedness, helpfulness, or safety scores degrade even when prompts and model are nominally fixed. Hardest to detect because evaluating outputs costs money and time. Standard approach: eval-in-prod sampling on 1-5% of traffic with an LLM-as-judge or cheap classifier, alerting on the trailing average. Ragas (matured through 2024) is the standard for RAG metrics; DeepEval covers general LLM eval-in-prod.

44.3.6 Hash-Pinned versus Floating Model Versions

Vendors offer two naming patterns, each pushing response-drift risk to a different party. Hash-pinned snapshots (gpt-4o-2024-11-20, claude-sonnet-4-5, gemini-2.5-pro-001) are immutable: the same prompt produces a deterministic distribution until the vendor sunsets the snapshot. Floating pointers (gpt-4o, claude-3-5-sonnet-latest) resolve to whatever the vendor considers current at request time. Floating gives free upgrades; pinned gives stable behavior. Anthropic's deprecation policy commits to 6 months notice; OpenAI's documents an analogous timeline. Production teams almost always pin and qualify new snapshots against the eval suite before bumping the config. The Air Canada case sits on this seam: the airline did not control the underlying model and could not roll back the chatbot's hallucinated policy.

44.3.7 The Open-Source Drift Stack

Four tools cover most of the OSS drift-detection market. Evidently AI (Apache 2.0) added LLM-specific tests in 2024 for text-quality drift, toxicity scoring, and prompt-template adherence. WhyLabs LangKit turns text into numeric signals (readability, sentiment, jailbreak-attempt probability) that feed any existing detector. NannyML focuses on quality estimation without ground truth, the realistic case in production. For RAG, Ragas computes faithfulness, answer-relevance, and context-recall scores and is now first-class in LangSmith, Phoenix, and Langfuse. For agents, DeepEval and OpenAI Evals wire LLM-as-judge prompts into the monitoring loop. The 2025 pattern: emit traces (Section 44.3.1), sample 1 to 5%, run a judge, store the score as a span attribute, alert on the trailing 24-hour mean by user-segment.

44.3.8 Operational Patterns

The drift signals are useless without an operational pattern that turns them into safe rollouts. Three patterns recur across mature LLM production teams:

Golden-set replay (nightly): maintain a curated set of 50 to 500 prompts that cover your traffic distribution. Each night a job re-runs them against the current production configuration and diffs the outputs against the last accepted run. Failures fire an alert; the on-call engineer compares the diffs and either approves or rolls back. This is the cheapest layer of defense and catches response drift from vendor model updates within 24 hours.

Shadow traffic: when you want to qualify a new model or a new prompt, mirror live requests to a parallel inference path that does not return user-visible output. Compare aggregate metrics (response length, refusal rate, eval-in-prod quality score) between the shadow and live paths; promote when shadow is statistically not worse. vLLM & Inference Servers covers the serving-side details; here we care about the diffing logic, which Ragas and DeepEval support out of the box.

Eval-in-prod (continuous): sample a small fraction of live traffic, run an LLM-as-judge or a classifier, store the score, alert on trends. The cost trade-off is straightforward: a 1% sample with a cheap judge costs roughly 1% of total inference, which buys you continuous quality signal. Below is a minimal pattern using OpenLLMetry trace attributes plus a Ragas judge:

import random
from ragas.metrics import Faithfulness
from openai import OpenAI

client = OpenAI()
judge = Faithfulness()
SAMPLE_RATE = 0.02  # 2% eval-in-prod sampling

def chat_with_eval(messages, retrieved_docs):
    response = client.chat.completions.create(
        model="gpt-4o-2024-11-20",  # hash-pinned, NOT "gpt-4o"
        messages=messages,
    )
    answer = response.choices[0].message.content
    if random.random() < SAMPLE_RATE:
        score = judge.score(
            question=messages[-1]["content"],
            answer=answer,
            contexts=retrieved_docs,
        )
        # Attach as OTel span attribute for trend dashboards
        from opentelemetry import trace
        trace.get_current_span().set_attribute("eval.faithfulness", score)
    return answer

44.3.9 Tool Comparison

44.3.10 Real-World Case: Anthropic's Model-Version Policy

Anthropic publishes its model deprecation policy: 6 months notice before any pinned snapshot retires, with behavior-diff documentation. This lets you plan the qualification cycle, run the new snapshot against your eval suite, diff it against the pinned snapshot, and re-qualify monitoring thresholds before cutover. OpenAI and Google publish analogous (less formal) policies. For self-hosted open-weights models (Llama-3, Mistral, Qwen via Hugging Face Hub) the version is whatever you downloaded; drift comes only from upgrades you initiate, but the qualification cost moves to you.

pip install langfuse openai faiss-cpu sentence-transformers