Libraries & Frameworks

The eval and production Python libraries split into eval harnesses, serving libraries, and observability SDKs.

45.2.1 Eval harnesses

• lm-evaluation-harness (EleutherAI, 2020) is the most-cited open eval library, supporting hundreds of academic benchmarks (MMLU, HellaSwag, ARC, GSM8K, TruthfulQA, BIG-Bench, etc.) under a uniform interface. Its objective is to provide one harness that reports comparable numbers across models, which matters because eval-implementation variance can swing benchmark numbers by several points. The core concept is YAML-defined tasks with prompts, scoring methods, and few-shot templates; the same model gets compared identically across tasks. Pick lm-eval-harness for academic-paper-style numbers; for agentic or tool-use evals, Inspect AI is better suited.
• Inspect AI (UK AISI, 2024) is the AISI's Python-native eval framework with first-class support for agent, tool-use, and sandboxed-code evals. Its objective is to support the more complex evaluations that lm-eval-harness was not designed for (multi-turn, tool-using, sandboxed), which matters for safety and agentic eval workflows. The core concept is Task + Solver + Scorer with composable Solvers for chains of thought, tool use, and self-critique. Pick Inspect AI for agentic and safety evals; it has rapidly become the academic standard for 2025+.
• openai/evals (OpenAI, 2023) is OpenAI's open-source eval framework with a community-contributed registry. Its objective is to provide a standardized way to define and share evals for OpenAI-compatible models, which matters when you want to publish reproducible evals. Pick when you want the OpenAI-ecosystem ecosystem; for academic benchmarks, lm-eval-harness has wider adoption.
• simple-evals (OpenAI, 2024) is OpenAI's stripped-down library used to produce the model-card numbers in OpenAI's own announcements. Its objective is minimal-code, transparent eval implementations that anyone can audit, which matters when you want to reproduce OpenAI's reported numbers precisely. Pick simple-evals when reproducing a specific OpenAI-cited benchmark; for breadth, lm-eval-harness covers more tasks.
• RAGAS (Exploding Gradients, 2023) is the RAG-specific eval framework with LLM-as-judge metrics for faithfulness, answer relevance, context precision, and context recall. Its objective is to evaluate RAG output quality on your own data, which matters because RAG eval has no public benchmark that mirrors your corpus. Pick RAGAS as the production RAG-eval default.
• deepeval (Confident AI, 2023) is the pytest-style LLM eval framework, letting you write evals as test functions. Its objective is to make LLM evals first-class CI tests rather than separate notebook workflows, which matters when you want CI gating on eval pass-rates. Pick deepeval when integrating eval into pytest-shaped CI pipelines.

RAGAS deserves a runnable example here, not just a catalog line, because it is the one eval library on this list aimed squarely at retrieval-augmented systems. It computes reference-free RAG metrics (faithfulness: is every claim in the answer grounded in the retrieved context; answer relevancy: does the answer address the question; context precision: are the retrieved passages relevant; context recall: was all the needed evidence retrieved) by using an LLM-as-judge, so you can score a RAG pipeline on your own corpus without a labeled benchmark. Reach for it when you are tuning chunking, retrieval, or prompts and want each change to move a number rather than a vibe. Code Fragment 45.2.1 below scores a one-row evaluation set so the metric objects and their float outputs are concrete; the keys (user_input, retrieved_contexts, response, reference) and metric classes match the fuller treatment in Section 32.4.

# RAGAS scores a RAG pipeline with reference-free, LLM-judged metrics.
# Build a tiny EvaluationDataset of (question, contexts, answer, reference)
# rows, then evaluate() runs all four metrics in one pass (pip install ragas).
from ragas import EvaluationDataset, evaluate
from ragas.metrics import (
    Faithfulness, ResponseRelevancy,
    ContextPrecision, LLMContextRecall,
)

dataset = EvaluationDataset.from_list([{
    "user_input": "How many vacation days do full-time staff get?",
    "retrieved_contexts": ["Full-time employees receive 20 days of paid vacation."],
    "response": "Full-time employees get 20 paid vacation days per year.",
    "reference": "20 paid vacation days annually for full-time employees.",
}])

report = evaluate(
    dataset,
    metrics=[Faithfulness(), ResponseRelevancy(),
             ContextPrecision(), LLMContextRecall()],
)
print(report)  # aggregated score per metric, 0.0 to 1.0

45.2.2 Serving libraries

Install with uv (Astral, 10-100x faster than pip and the default modern installer). The three production-default inference techniques are speculative decoding, prefix caching, and attention sinks; all three ship enabled by default in vLLM v1 and SGLang in 2025. LMCache (2024) is the KV cache offloading layer increasingly added on top.

• vLLM v1 (UC Berkeley, 2023; v1 architecture 2025-Q1) is the open-source LLM serving library built around PagedAttention and continuous batching. The v1 architecture (2025-Q1) is a major rewrite that turns on prefix caching by default and exposes a first-class speculative-decoding API; if your vLLM mental model is from 2023-24, re-read the v1 docs. Pick vLLM as the default production server in 2026.
• TGI (Hugging Face, 2023) is Hugging Face's production server with Apache 2.0 core. Its share collapsed in 2024-25 in favor of vLLM v1 and SGLang for general production use; TGI primarily lives on as the backend of HF Inference Endpoints in 2026. Consider vLLM or SGLang first for new deployments.
• SGLang (LMSYS, 2024) is vLLM's competitor focused on structured generation and shared-prefix workloads via RadixAttention. Pick when you have heavily-shared prompt prefixes (agent loops, eval harnesses) or need fast JSON-constrained output.
• Triton Inference Server (NVIDIA, 2018) is the framework-agnostic inference server (PyTorch, TF, ONNX, TensorRT). Pick when your fleet has heterogeneous models including non-LLMs; for LLM-only, vLLM is simpler.
• LoRAX (Predibase, 2023) is the multi-LoRA serving server: one base model, many fine-tuned adapters, served simultaneously. Its objective is to amortize a single base-model GPU across hundreds of customer-specific fine-tunes, which matters for multi-tenant SaaS where each customer has their own LoRA. Pick LoRAX when your fine-tunes are adapter-shaped and per-tenant; for single-model serving, vLLM is simpler.

45.2.3 Observability SDKs

• LangSmith SDK (LangChain Inc., 2023) is the Python and JS SDK for LangChain's LangSmith tracing service. Its objective is one-import auto-instrumentation for LangChain apps plus framework-agnostic decorators for everything else. Pick when using LangChain; for non-LangChain code, the manual decorators still work but Langfuse may feel more natural.
• Langfuse SDK (Langfuse, 2023) is the SDK for the open-source Langfuse tracing platform. Its objective is to provide LangSmith-shaped tracing with framework-agnostic decorators, which matters when you want vendor-neutral observability or self-hosting.
• Phoenix (Arize OpenSource) (Arize AI, 2023) is the open-source local-first tracing library with OpenTelemetry integration. Its objective is zero-cost zero-account observability for local dev and small-team production, which matters when you want to look at LLM traces without paying for SaaS. Pick Phoenix for local development; for team production, Langfuse or LangSmith scale better.
• Helicone (Helicone, 2023) is the proxy-based observability SDK; replace your OpenAI base URL with Helicone's and your traffic is logged automatically. Its objective is zero-code-change observability, ideal for adding monitoring to existing code without refactoring.
• OpenTelemetry + OpenLLMetry (CNCF / Traceloop, 2024) is the vendor-neutral instrumentation standard: OpenTelemetry for general tracing, OpenLLMetry for LLM-specific semantic conventions. Its objective is to make LLM tracing portable across observability backends (Datadog, Honeycomb, Grafana, etc.), which matters when you already run an OTel-based monitoring stack. Pick OTel + OpenLLMetry when standards-compliance and portability matter; for LLM-first features, specialist tools (Langfuse, LangSmith) lead.

import os
from phoenix.otel import register

# One-line instrumentation: any LangChain / LlamaIndex / OpenAI Agents SDK
# call now emits OTel spans into Phoenix
os.environ["PHOENIX_PROJECT_NAME"] = "support-agent-prod"
tracer_provider = register(endpoint="https://phoenix.example.com/v1/traces")

# Now run your agent normally; every tool call, every LLM step,
# every retry is a span in the trace tree.
from langgraph.prebuilt import create_react_agent
agent = create_react_agent(model="openai:gpt-4o-mini", tools=[search, code_exec])
result = agent.invoke({"messages": [("user", "Find recent SOTA on multi-agent coordination")]})

from langgraph.errors import GraphRecursionError

# Bound the loop with two layers: graph recursion and a token budget callback
class TokenBudget:
    def __init__(self, max_tokens=50_000):
        self.spent = 0
        self.max_tokens = max_tokens
    def on_llm_end(self, response, **_):
        self.spent += response.usage_metadata["total_tokens"]
        if self.spent > self.max_tokens:
            raise RuntimeError(f"Token budget exhausted: {self.spent}")

budget = TokenBudget(max_tokens=50_000)

try:
    result = agent.invoke(
        {"messages": [("user", question)]},
        config={"recursion_limit": 20, "callbacks": [budget]},
    )
except (GraphRecursionError, RuntimeError) as e:
    # Caps fired; return a graceful error to the user
    log.error("agent_terminated", reason=str(e), spent_tokens=budget.spent)
    return {"error": "request_too_complex", "spent": budget.spent}

from guardrails import Guard
from pydantic import BaseModel, Field

class SupportResponse(BaseModel):
    intent: str = Field(description="The customer's intent")
    suggested_action: str = Field(description="What the support rep should do")
    contains_pii: bool = Field(description="True if response leaks customer PII")

guard = Guard.from_pydantic(SupportResponse)
result = guard(
    llm_api=openai_call,
    prompt="Customer asked: {{question}}",
    num_reasks=2,  # retry if validation fails
)
# If the LLM emits malformed JSON or leaks PII, Guardrails AI re-prompts
# with the validation error until the output is valid or retries are exhausted.