Eval-as-Product: Braintrust, Latitude, Laminar

"Observability tells you what already broke. Eval-as-product tells you what will break tomorrow if you ship the new prompt today."

Eval, Replay-Native AI Agent

44.7.1 From Eval-as-CI to Eval-as-Product

The eval-as-CI pattern from Section 42.1 is necessary but incomplete. It runs the same fixed golden set on every PR, blocks merges on metric thresholds, and reports a pass/fail. That works well for regression prevention: it catches cases where a new prompt or model degrades a well-understood behavior. It does not handle the three jobs that emerged as LLM apps got more complex:

• Scoring live traffic. A golden set is static; production traffic shifts daily. A reasonable score on yesterday's eval set does not guarantee a reasonable score on today's incoming queries. You need a way to score live traffic, at least on a sample, and surface trends as they emerge.
• Replaying historical traffic. When you want to try a new prompt, the question "would this prompt have produced better results on the last week of traffic?" is more decision-relevant than "does this prompt pass the golden set?" Replay against historical traces, with scoring side-by-side, is the core eval-as-product operation.
• Comparing many variants. A spreadsheet of 20 prompts and 4 models is 80 cells. CI gates only the merge candidate; they don't let a PM or domain expert sweep a candidate space. A UI that shows the 80 cells with per-cell scores, side-by-side diffs, and confidence intervals is a fundamentally different workflow than CI.

Eval-as-product is the category of tools that do those three jobs in one UI. The first generation (Promptfoo, OpenAI Evals) emphasized the static-eval job and had limited replay and live-traffic features. The second generation (Braintrust, Latitude, Laminar, LangSmith, Helicone, plus offerings from the major model providers) treats live-scoring, replay, and variant-comparison as first-class operations alongside static eval.

44.7.2 The Eval-First Workflow: Experiment, Score, Compare, Iterate

The workflow that eval-as-product platforms optimize for has four steps, and the shape of the workflow is what distinguishes good platforms from bad ones.

Step 1: Experiment. Define a hypothesis (a new prompt, a new model, a new tool, a new retrieval strategy), and a dataset (production traces, a curated golden set, a synthetic eval set). The platform runs the hypothesis against the dataset and produces a result table: one row per dataset item, one column per metric.

Step 2: Score. Each row's output is scored along multiple axes: rule-based metrics (exact-match, JSON validity), LLM-as-judge scores (helpfulness, faithfulness), and custom metrics (latency, cost, token count). The good platforms make adding a custom metric a 5-minute job: write a function that takes (input, output, expected) and returns a score, register it as a scorer.

Step 3: Compare. The platform diffs your experiment against a baseline (the production prompt, the last shipped version). Per-row diffs highlight which items improved, which regressed, and by how much. Aggregate diffs show overall direction. This is the step where most teams catch the cases that CI gates miss: a 0.5% aggregate improvement that masks a 20% regression on one specific user segment.

Step 4: Iterate. Based on the comparison, refine the prompt, retry, score, compare. The fastest platforms make this loop tight (sub-minute for small datasets). The slow ones make it slow enough that teams stop using them after a week.

The crucial insight is that this is the same loop that ML researchers run for fine-tuning experiments, but with a UI tailored for prompt-engineering iteration cycles instead of training-run cycles. The platforms that have succeeded in 2025-2026 are the ones that nailed the UI ergonomics, the ones that treated this as an ML-platform problem in disguise.

44.7.3 Braintrust: The Polished Default

Braintrust (braintrust.dev) is the most polished of the three platforms covered here, and it has captured a large share of the venture-backed AI-product market. Its design choices reflect that audience: rich playground for prompt iteration, deep integration with OpenAI and Anthropic SDKs, automatic regression detection against the previous experiment, and a Python/TypeScript SDK that maps cleanly onto existing eval patterns.

The core abstractions are datasets (collections of input/expected-output pairs), experiments (a run of your task against a dataset), and scorers (functions that judge each row of an experiment). The Braintrust scorer library includes pre-built scorers for many common metrics: Factuality (LLM-as-judge for factual accuracy), Faithfulness (RAG-style faithfulness scoring), AnswerRelevancy, JSONDiff (structural diff for JSON outputs), Levenshtein, NumericDiff, Sql (SQL-equivalence checking), and others. You can add custom scorers by writing a Python or TypeScript function.

Strengths:

• Rich scoring DSL. The scorer library is broad and well-curated. The same scorer interface works for batch eval, live scoring, and replay.
• OpenAI/Anthropic native. First-class integration with both major provider SDKs. Drop-in proxy mode that captures all calls without code changes is a major time-saver for teams already on those SDKs.
• Automatic regression detection. Compare any new experiment against the production baseline with one click; the UI surfaces per-row regressions and aggregates.
• Playground. Interactive prompt editing with side-by-side comparison across models, useful for the PM-and-domain-expert audience.

Weak spots:

• Enterprise pricing. The free tier is generous for small projects, but enterprise pricing scales with traffic volume and gets expensive quickly. Teams running millions of traced requests per day are typically looking at five-figure annual contracts.
• Closed-source. Self-hosting is not an option; if you need on-premises evaluation (regulated industries, air-gapped environments), Braintrust is off the table.
• Vendor lock-in risk. The scorer DSL is proprietary; migrating away requires rewriting the eval definitions.

44.7.4 Latitude: The Open-Source Alternative

Latitude (latitude.so) takes the opposite design stance: open-source under an MIT-style license, self-hostable, prompt-versioning as the central abstraction. The hosted SaaS version exists, but Latitude's pitch is the on-premises option. For regulated industries (finance, healthcare, defense) and for teams that already run Postgres-and-Docker infrastructure, the self-hostable property dominates other considerations.

Latitude's primary abstraction is a prompt, with versioning, parameters, and metadata as first-class properties. Datasets and experiments build on top of prompts. The LLM-as-judge functionality is built in: you write a judge prompt in the same DSL as a regular prompt, and Latitude scores experiments using it.

Strengths:

• Open-source and self-hostable. Run it on your own Postgres and Docker. The license is permissive enough for commercial use.
• Prompt-versioning-first. Treats prompts as the unit of work, with diff, history, and rollback. For teams whose iteration is mostly prompt-engineering (rather than full-stack agent development), this is a natural fit.
• Built-in LLM-as-judge. Judge prompts are first-class objects, not custom scorer code.
• Cheap. Self-hosted incurs only your infra cost; the hosted tier is priced below Braintrust.

Weak spots:

• Less polished UI. The trade-off for being open-source: the UX is functional but lags Braintrust in places.
• Smaller scorer library. You will write more custom judges; the pre-built scorer catalogue is thinner.
• Younger community. Fewer Stack Overflow answers, fewer integration examples for niche provider SDKs.

44.7.5 Laminar: Observability Plus Eval, Span-Based

Laminar (lmnr.ai) sits in a third position: it merges observability and eval into a single span-based system, treating every LLM call (and every tool call, every retrieval, every span in your trace) as both an observability event and a potential eval target. The unit of work is the span, and you can attach scorers to spans the same way you attach metrics to traces.

Laminar's design choices favor agent-style applications where the trace is the primary object of analysis: long traces, many tool calls, multi-turn conversations, hierarchical span structure. For teams whose evaluation question is "did this 30-step agent trajectory succeed?" rather than "did this single LLM call produce the right answer?", Laminar's span-centric model maps more cleanly.

Strengths:

• Cheap and fast. Pricing is among the lowest in the category. Ingestion latency is low; queries on large trace sets are fast.
• Span-based filtering and scoring. Filter traces by any span property; attach scorers at span granularity rather than trace granularity. Useful for agent debugging.
• Combined observability and eval. One platform for both jobs, with a unified trace model. Fewer tools to integrate, fewer SDKs to learn.

Weak spots:

• Newer and less mature. Smaller team, fewer integrations, fewer pre-built scorers than Braintrust. Documentation gaps in places.
• Self-hosting requires more legwork. The open-core model exists, but the deployment story is rougher than Latitude's.
• Less polished playground. The interactive prompt-editing UX is functional but not the equal of Braintrust's.

Comparison table

The decision matrix below summarizes the three platforms (and includes LangSmith and OpenAI Evals as anchors) across the dimensions that typically drive selection.

44.7.6 Wiring Braintrust Into a Production Agent Loop

The simplest useful Braintrust integration: instrument your production agent to log every run to Braintrust, then run a nightly job that scores the previous day's runs and surfaces the results in the Braintrust UI.

# Input: prod traces logged to Braintrust by braintrust.wrap_openai()
# Output: nightly scored experiment in the Braintrust UI
import braintrust
from braintrust import init_dataset, Eval
from autoevals import Factuality, AnswerRelevancy, JSONDiff

# 1) at app startup, wrap OpenAI so every prod call is auto-logged
from openai import OpenAI
client = braintrust.wrap_openai(OpenAI())      # traces -> project "prod-agent"

# 2) nightly: pull yesterday's traces as a dataset, score against scorers
def nightly_score():
    dataset = init_dataset(project="prod-agent", name="yesterday",
                           filter="created > now() - interval '1 day'")
    Eval(
        name="prod-agent-nightly",
        data=lambda: dataset,
        task=lambda row: row["output"],   # replay-mode: score existing outputs
        scores=[Factuality(), AnswerRelevancy(), JSONDiff()],
    )

if __name__ == "__main__":
    nightly_score()                                # cron: 0 3 * * *

The same pattern works in TypeScript, in Latitude (with a different SDK shape), and in Laminar (with a span-filter instead of a trace-filter). The interesting part is not the SDK, it is the workflow: scoring happens after the fact, on real traffic, with the latest version of your scorer library. If you change a scorer's prompt, you can rescore the last month of traffic with one command and see whether the new scorer agrees with the old one on the same data.

44.7.7 When Not to Use Eval-as-Product

Eval-as-product is not a universal good. The platforms have real costs (subscription fees, integration effort, vendor lock-in), and there are workflows where those costs do not pay back.

Fully automated, CI-gated eval. If your eval is 100% rule-based or unit-test-style (exact-match against expected outputs, JSON-schema validation, latency thresholds), and there is no human review loop, then a simple promptfoo or pytest harness running in CI is sufficient. The eval-as-product features (replay UX, side-by-side comparison, live scoring) add little value to a workflow that does not use them.

Tiny traffic, slow iteration. A team shipping a low-volume internal tool that updates its prompts once a quarter does not need a replay loop. The fixed cost of adopting a platform exceeds the savings.

Regulated environments without self-hosting. If you cannot send production data to a third-party SaaS, Braintrust and Laminar's hosted tiers are off the table. Latitude's self-host option or a custom-built solution becomes the relevant choice.

When your eval workflow is fundamentally upstream of LLMs. If your team's evaluation is mostly about classical ML (a tabular classifier, an embedding retriever, a search-quality A/B test), MLflow or Weights and Biases are a better fit. Eval-as-product platforms are optimized for LLM-shaped evaluation; using them for non-LLM workloads is forcing a square peg into a round hole.

44.7.8 The Platform vs. Build Decision

The recurring question: should we adopt a platform or build the equivalent in-house? For most teams in 2026, the answer is platform. The integration effort is hours to days, not weeks; the feature surface (replay, side-by-side, scorer library, live scoring, multi-judge orchestration) is hard to replicate quickly; and the platforms genuinely move faster than internal eval teams can.

The cases where building in-house pays back are narrow but real:

• Air-gapped or PII-saturated environments. Where no SaaS option is acceptable and Latitude's OSS does not fit your stack.
• Highly domain-specific scoring. If your eval is dominated by a single bespoke metric (code-execution success, theorem-proving validity, drug-interaction safety) and the platform's pre-built scorers add little value, a thin in-house harness around your bespoke scorer may be enough.
• Eval as part of a larger ML platform. If your organization already runs MLflow or W&B at scale, integrating LLM eval into that stack may produce a more coherent system than adding a separate platform.

For everyone else, the platform-or-build calculus comes out heavily in favor of adopting one of the three platforms covered here, ideally after running a 2-week trial with at least two of them on your real data.

Exercises