LLM Evaluation Fundamentals

"If your only evaluation metric is vibes, your only production guarantee is also vibes."

Eval, Vibe-Averse AI Agent

42.1.1 Intrinsic Language Modeling Metrics

# Input: judge LLM, question, candidate answers a and b
# Output: winner ('a wins', 'b wins', 'tie / unreliable') after running the judge on both orderings
def judge_with_swap(judge, question, a, b):
    v1 = judge(question, a, b)      # a first
    v2 = judge(question, b, a)      # b first
    if v1 == "first" and v2 == "second":  return "a wins"
    if v1 == "second" and v2 == "first":  return "b wins"
    return "tie / unreliable"   # judge disagreed with itself

Intrinsic metrics measure how well a model captures the statistical properties of language itself. These metrics are computed directly from the model's probability distributions over tokens, without requiring any downstream task. They are most useful for comparing base (pretrained) models and for monitoring training progress.

Perplexity

Perplexity measures how "surprised" a model is by a sequence of text. Formally, it is the exponentiated average negative log-likelihood per token. A model that assigns high probability to the correct next token at every position will have low perplexity. Lower perplexity means the model is a better predictor of natural language.

For a sequence of N tokens, perplexity is defined as:

Perplexity depends heavily on the tokenizer. A model that uses byte-pair encoding (Chapter 1) with a 50K vocabulary will have a different perplexity from one using a 100K vocabulary, even if both models are equally capable. This makes direct perplexity comparisons across model families unreliable.

Bits-per-Byte (BPB)

Bits-per-byte normalizes the language modeling loss by the number of UTF-8 bytes rather than tokens, making it comparable across tokenizers and vocabularies. BPB measures how many bits the model needs, on average, to encode each byte of the original text. In plain terms, a lower BPB means the model compresses text more efficiently, which reflects better language understanding. This is the preferred metric when comparing models with different tokenization schemes.

# implement compute_perplexity_and_bpb
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
def compute_perplexity_and_bpb(model_name: str, text: str):
    """Compute perplexity and bits-per-byte for a text sample."""
    tokenizer = AutoTokenizer.from_pretrained(model_name)
    model = AutoModelForCausalLM.from_pretrained(
        model_name, torch_dtype=torch.float16, device_map="auto"
        )
    encodings = tokenizer(text, return_tensors="pt")
    input_ids = encodings.input_ids.to(model.device)
    with torch.no_grad():
        outputs = model(input_ids, labels=input_ids)
        neg_log_likelihood = outputs.loss # average NLL per token
        num_tokens = input_ids.size(1)
        perplexity = torch.exp(neg_log_likelihood).item()
        # Bits-per-byte: convert nats to bits, normalize by bytes
        total_nll_bits = neg_log_likelihood.item() * num_tokens / torch.log(torch.tensor(2.0))
        num_bytes = len(text.encode("utf-8"))
        bpb = total_nll_bits / num_bytes
        return {
            "perplexity": round(perplexity, 2),
            "bits_per_byte": round(bpb.item(), 4),
            "num_tokens": num_tokens,
            "num_bytes": num_bytes
            }
        # Example usage
        text = "The transformer architecture revolutionized natural language processing."
        result = compute_perplexity_and_bpb("gpt2", text)
        print(result)

Figure 42.1.1a: Taxonomy of LLM evaluation metrics, arranged by approach and cost-validity tradeoff.

42.1.2 Reference-Based Text Metrics

Reference-based metrics compare model output against one or more "gold standard" reference texts. They originated in machine translation and summarization, where human-written references provide a ground truth. These metrics are fast, deterministic, and cheap to compute, but they share a fundamental limitation: they assume the reference text captures the space of acceptable answers, which is rarely true for open-ended generation.

BLEU, ROUGE, and BERTScore

Among these metrics, BLEU is the most established. Understanding its formula clarifies both its strengths (penalizing short or repetitive outputs) and its limitations (ignoring meaning entirely):

A complementary recall-oriented metric captures how much of the reference content appears in the candidate:

Moving beyond surface-level matching, semantic similarity metrics use neural embeddings to capture meaning:

# Implementation example
from rouge_score import rouge_scorer
from bert_score import score as bert_score
import evaluate
# Reference and candidate texts
reference = "The cat sat on the mat and watched the birds outside."
candidate = "A cat was sitting on a mat, observing birds through the window."
# ROUGE scores
scorer = rouge_scorer.RougeScorer(["rouge1", "rouge2", "rougeL"], use_stemmer=True)
rouge_results = scorer.score(reference, candidate)
for key, value in rouge_results.items():
    print(f"{key}: precision={value.precision:.3f}, recall={value.recall:.3f}, f1={value.fmeasure:.3f}")
    # BLEU score
    bleu = evaluate.load("bleu")
    bleu_result = bleu.compute(
        predictions=[candidate],
        references=[[reference]]
        )
    print(f"BLEU: {bleu_result['bleu']:.4f}")
    # BERTScore
    P, R, F1 = bert_score([candidate], [reference], lang="en", verbose=False)
    print(f"BERTScore: precision={P[0]:.4f}, recall={R[0]:.4f}, f1={F1[0]:.4f}")

42.1.3 LLM-as-Judge

The LLM-as-Judge paradigm uses a strong language model (typically GPT-4, Claude, or a specialized judge model) to evaluate the outputs of other models. This approach has rapidly become the most popular evaluation method for instruction-tuned models because it can assess open-ended qualities like helpfulness, safety, and reasoning quality that reference-based metrics cannot capture.

Judge Prompt Design

The quality of LLM-as-Judge evaluation depends entirely on the judge prompt. A well-designed prompt specifies clear evaluation criteria, provides a rubric with concrete examples at each score level, and asks the judge to produce a structured output (typically a score plus reasoning). The reasoning-before-score pattern (where the judge explains its rationale before assigning a numeric score) has been shown to produce more reliable evaluations.

# implement llm_judge_evaluate
from openai import OpenAI
import json
client = OpenAI()
def llm_judge_evaluate(question: str, answer: str, criteria: str) -> dict:
    """Use an LLM as a judge to evaluate answer quality."""
    judge_prompt = f"""You are an expert evaluator. Assess the following answer to a question.
    QUESTION: {question}
    ANSWER: {answer}
    EVALUATION CRITERIA: {criteria}
    RUBRIC:
    5 = Excellent: Fully addresses the question with accurate, comprehensive, well-organized content
    4 = Good: Addresses the question well with minor gaps or imprecisions
    3 = Adequate: Partially addresses the question but has notable gaps
    2 = Poor: Mostly fails to address the question or contains significant errors
    1 = Very Poor: Completely off-topic, factually wrong, or incoherent
    Provide your evaluation as JSON with these fields:
    - "reasoning": Your step-by-step analysis (2-3 sentences)
    - "score": Integer from 1 to 5
    - "strengths": List of strengths
    - "weaknesses": List of weaknesses"""
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": judge_prompt}],
        response_format={"type": "json_object"},
        temperature=0.0
    )
    return json.loads(response.choices[0].message.content)
# Example evaluation
result = llm_judge_evaluate(
    question="What causes seasons on Earth?",
    answer="Seasons happen because Earth's axis is tilted at 23.5 degrees relative to its orbital plane. This means different hemispheres receive more direct sunlight at different times of year.",
    criteria="Factual accuracy, completeness, and clarity"
)
print(json.dumps(result, indent=2))

{
  "reasoning": "Correctly identifies axial tilt (23.5 deg) as the cause and explains the hemispheric sunlight mechanism clearly. Omits the role of orbital position and does not mention equinoxes or solstices.",
  "score": 4,
  "strengths": ["Correct identification of axial tilt", "Specific 23.5 degree figure", "Clear mechanism"],
  "weaknesses": ["No mention of orbital position", "Missing equinox / solstice details"]
}

Reliability of an LLM-as-Judge pipeline is measured against a small held-out human-rated set. Given paired judge scores $(j_1, \ldots, j_n)$ and human scores $(h_1, \ldots, h_n)$, the Spearman rank correlation

captures how well the judge orders the candidates the same way humans do, even when the two scales differ in scale or offset. A Spearman correlation above 0.7 is the typical bar for treating an LLM-as-Judge metric as a credible proxy at scale, with the calibration set re-audited every quarter or whenever the judge model is upgraded (a judge swap from GPT-4o to GPT-5 can shift the score distribution by 0.3 to 0.5 points even when the rank order is preserved).

import anthropic, json

client = anthropic.Anthropic()

def claude_judge(question: str, answer: str, criteria: str) -> dict:
    """One-shot LLM-as-Judge call against Claude, returns the same JSON shape as Code 42.1.3a."""
    rubric = (
        "5=Excellent, 4=Good, 3=Adequate, 2=Poor, 1=Very Poor.\n"
        "Return JSON with fields: reasoning, score, strengths, weaknesses."
    )
    msg = client.messages.create(
        model="claude-3-5-sonnet-latest",
        max_tokens=512,
        temperature=0.0,
        messages=[{"role": "user", "content": (
            f"QUESTION: {question}\nANSWER: {answer}\n"
            f"CRITERIA: {criteria}\nRUBRIC: {rubric}"
        )}],
    )
    return json.loads(msg.content[0].text)

from deepeval.metrics import AnswerRelevancyMetric, FaithfulnessMetric
from deepeval.test_case import LLMTestCase
test_case = LLMTestCase(
    input="What causes seasons on Earth?",
    actual_output="Seasons happen because Earth's axis is tilted...",
    retrieval_context=["Earth's axis is tilted 23.5 degrees..."]
)
relevancy = AnswerRelevancyMetric(threshold=0.7)
relevancy.measure(test_case)
print(f"Score: {relevancy.score}, Reason: {relevancy.reason}")

Common LLM-as-Judge Biases

LLM judges are powerful but carry systematic biases that evaluators must account for. Understanding these biases is essential for interpreting judge results correctly. Figure 42.1.2a shows the LLM-as-Judge evaluation pipeline.

• Position bias: Judges tend to prefer the first option in pairwise comparisons. Mitigation: randomize order and average scores across both orderings.
• Verbosity bias: Judges often favor longer, more detailed answers even when the extra content adds no value. Mitigation: include "conciseness is a virtue" in the rubric.
• Self-preference bias: Models tend to rate their own outputs higher than outputs from other models. Mitigation: use a judge model different from the models being evaluated.
• Authority bias: Confident, well-formatted answers receive higher scores even when factually wrong. Mitigation: explicitly instruct the judge to verify factual claims.

Position bias can be quantified with a position-balanced score. Let $s(A, B)$ be the probability that the judge picks A when A is shown first, and $s(B, A)$ the probability when B is shown first. The unbiased preference is

If a judge has no position bias, both terms are equal and $\hat{p}$ is just the raw win rate; if it has a strong "first answer wins" prior, the two terms diverge, and the average correctly de-biases the estimate.

# Position-balanced LLM-as-Judge for pairwise model comparison.
# Calls the judge twice per pair (A-then-B and B-then-A) and averages.
# This is the standard MT-Bench protocol; without the swap, position bias
# alone can inflate a model's apparent win rate by 5-15 percentage points.
import random
from openai import OpenAI

client = OpenAI()
RUBRIC = "You are an expert judge. Decide which answer is better. " \
         "Output exactly one of: 'A' or 'B'. Be concise. Avoid favoring " \
         "longer answers; conciseness is a virtue."

def judge_one_order(question, ans_first, ans_second, model="gpt-4o-mini"):
    prompt = (
        f"{RUBRIC}\n\nQuestion: {question}\n\n"
        f"Answer A:\n{ans_first}\n\nAnswer B:\n{ans_second}\n\nVerdict:"
    )
    r = client.chat.completions.create(
        model=model, temperature=0,
        messages=[{"role": "user", "content": prompt}],
    )
    return r.choices[0].message.content.strip().upper().startswith("A")

def pairwise_winrate(question, ans_X, ans_Y, n_trials=3):
    """Position-balanced win-rate of X over Y."""
    wins = 0
    for _ in range(n_trials):
        # Trial 1: X shown first.
        if judge_one_order(question, ans_X, ans_Y):
            wins += 1
        # Trial 2: X shown second.  Y is "A", X is "B"; we negate.
        if not judge_one_order(question, ans_Y, ans_X):
            wins += 1
    return wins / (2 * n_trials)

Code Fragment 42.1.2b: Position-balanced pairwise LLM-as-Judge. Each comparison runs twice (A first, then B first), and the answers are averaged so that the judge's position prior cancels out. This is the protocol used by MT-Bench, Arena-Hard, and most reproducible LLM leaderboards.

Figure 42.1.2c: The LLM-as-Judge evaluation pipeline, from test inputs through structured scoring output.

Intuitively, LLM-as-Judge works because strong models have internalized enough of human preferences through RLHF that they can approximate the judgment of a human annotator. The main risk is that the judge model shares the same biases as the model being evaluated (both prefer verbose, formal answers) creating a feedback loop that rewards style over substance. This is why calibrating LLM judges against human judgments, as described in the arena methodology of evaluation metrics, remains essential for high-stakes evaluation.

42.1.4 Human Evaluation

Human evaluation remains the gold standard for assessing LLM quality, especially for subjective dimensions like helpfulness, creativity, and conversational naturalness. However, human evaluation is expensive, slow, and introduces its own variability. The key challenge is designing evaluation protocols that are reliable (different annotators agree) and valid (they measure what you care about).

Evaluation Protocol Design

Effective human evaluation protocols share several characteristics. First, they define clear, specific criteria with concrete examples at each rating level. Second, they include calibration rounds where annotators evaluate the same examples and discuss disagreements before the main annotation begins. Third, they compute inter-annotator agreement metrics (like Cohen's kappa or Krippendorff's alpha) to verify that the task is well-defined enough for consistent human judgment.

Cohen's kappa corrects raw agreement for the agreement expected by chance: $\kappa = \dfrac{p_o - p_e}{1 - p_e}$, where $p_o$ is the observed proportion of items the two raters agree on and $p_e$ is the proportion expected if each rater labeled independently at their own base rates. Subtracting $p_e$ removes credit for lucky agreements (two raters who both answer "yes" 90% of the time start from a high $p_e$), and dividing by $1 - p_e$ rescales so that $\kappa = 1$ is perfect agreement and $\kappa = 0$ is chance level.

# implement compute_inter_annotator_agreement
import numpy as np
from sklearn.metrics import cohen_kappa_score
def compute_inter_annotator_agreement(annotations: dict) -> dict:
    """Compute pairwise Cohen's kappa between annotators.
    Args:
    annotations: dict mapping annotator_id to list of scores
    """
    annotator_ids = list(annotations.keys())
    kappas = {}
    for i in range(len(annotator_ids)):
        for j in range(i + 1, len(annotator_ids)):
            a1, a2 = annotator_ids[i], annotator_ids[j]
            kappa = cohen_kappa_score(
                annotations[a1], annotations[a2], weights="quadratic"
                )
            kappas[f"{a1}_vs_{a2}"] = round(kappa, 3)
            avg_kappa = np.mean(list(kappas.values()))
            return {"pairwise_kappas": kappas, "mean_kappa": round(avg_kappa, 3)}
            # Three annotators rating 10 examples on a 1-5 scale
            annotations = {
                "annotator_A": [4, 3, 5, 2, 4, 3, 5, 4, 3, 4],
                "annotator_B": [4, 3, 4, 2, 5, 3, 5, 3, 3, 4],
                "annotator_C": [5, 3, 5, 3, 4, 2, 4, 4, 3, 5],
                }
            agreement = compute_inter_annotator_agreement(annotations)
            print(f"Pairwise kappas: {agreement['pairwise_kappas']}")
            print(f"Mean kappa: {agreement['mean_kappa']}")
            # Interpretation: >0.8 = excellent, 0.6-0.8 = good, 0.4-0.6 = moderate

42.1.5 Standard Benchmarks

Benchmarks provide standardized test sets that enable comparison across models. The LLM community has developed dozens of benchmarks, each targeting different capabilities. Understanding what each benchmark measures (and what it does not measure) is essential for interpreting model comparison tables. Figure 42.1.3 maps the benchmark landscape by capability domain.

Figure 42.1.3d: The LLM benchmark landscape organized by capability domain.

Benchmark Comparison

Chatbot Arena ranks models using pairwise human comparisons rather than fixed benchmarks. The underlying Elo rating system, borrowed from chess, updates each model's score after every head-to-head comparison:

# implement run_mmlu_sample
import json
from datasets import load_dataset
def run_mmlu_sample(model_fn, num_samples: int = 20):
    """Evaluate a model on a sample of MMLU questions.
    Args:
    model_fn: callable that takes a prompt and returns A/B/C/D
    num_samples: number of questions to evaluate
    """
    dataset = load_dataset("cais/mmlu", "all", split="test")
    dataset = dataset.shuffle(seed=42).select(range(num_samples))
    correct = 0
    results_by_subject = {}
    for example in dataset:
        choices = example["choices"]
        prompt = f"""Question: {example['question']}
        A) {choices[0]}
        B) {choices[1]}
        C) {choices[2]}
        D) {choices[3]}
        Answer with just the letter (A, B, C, or D):"""
        prediction = model_fn(prompt).strip().upper()
        answer_map = {0: "A", 1: "B", 2: "C", 3: "D"}
        correct_letter = answer_map[example["answer"]]
        is_correct = prediction[0] == correct_letter if prediction else False
        subject = example["subject"]
        if subject not in results_by_subject:
            results_by_subject[subject] = {"correct": 0, "total": 0}
            results_by_subject[subject]["total"] += 1
            if is_correct:
                results_by_subject[subject]["correct"] += 1
                correct += 1
                accuracy = correct / num_samples
                return {
                    "overall_accuracy": round(accuracy, 3),
                    "num_correct": correct,
                    "num_total": num_samples,
                    "by_subject": results_by_subject
                    }

# pip install lm-eval
# Command-line (most common):
# lm_eval --model hf \
# --model_args pretrained=meta-llama/Llama-3.1-8B-Instruct \
# --tasks mmlu,hellaswag,arc_challenge \
# --num_fewshot 5 \
# --batch_size auto \
# --output_path ./eval_results/
# Programmatic usage:
import lm_eval
results = lm_eval.simple_evaluate(
    model="hf",
    model_args="pretrained=meta-llama/Llama-3.1-8B-Instruct",
    tasks=["mmlu", "hellaswag"],
    num_fewshot=5,
    batch_size="auto",
    )
for task, metrics in results["results"].items():
    acc = metrics.get("acc,none", metrics.get("acc_norm,none", "N/A"))
    print(f"{task}: {acc:.3f}")

42.1.6 Building an Evaluation Harness

In practice, you rarely evaluate with a single metric. A well-designed evaluation harness combines multiple metrics, runs them across a curated test set, and produces a structured report that tracks performance over time. The following example shows how to build a simple but extensible evaluation framework.

# Define EvalCase, EvalResult, EvalHarness; implement __init__, evaluate, summary
from dataclasses import dataclass, field
from typing import Callable
import json, time
@dataclass
class EvalCase:
    """A single evaluation test case."""
    question: str
    reference: str = ""
    metadata: dict = field(default_factory=dict)
@dataclass
class EvalResult:
    """Result of evaluating one test case."""
    case: EvalCase
    model_output: str
    scores: dict
    latency_ms: float
class EvalHarness:
    """Lightweight evaluation harness for LLM applications."""
    def __init__(self, model_fn: Callable, scorers: dict[str, Callable]):
        self.model_fn = model_fn # callable: prompt -> response
        self.scorers = scorers # dict of name -> scoring function
    def evaluate(self, cases: list[EvalCase]) -> list[EvalResult]:
        """Run all test cases through the model and score them."""
        results = []
        for case in cases:
            start = time.time()
            output = self.model_fn(case.question)
            latency = (time.time() - start) * 1000
            scores = {}
            for name, scorer in self.scorers.items():
                scores[name] = scorer(
                    question=case.question,
                    output=output,
                    reference=case.reference
                    )
                results.append(EvalResult(
                    case=case, model_output=output,
                    scores=scores, latency_ms=round(latency, 1)
                    ))
                return results
    def summary(self, results: list[EvalResult]) -> dict:
        """Aggregate results into a summary report."""
        import numpy as np
        metric_names = list(results[0].scores.keys())
        summary = {}
        for metric in metric_names:
            values = [r.scores[metric] for r in results]
            summary[metric] = {
                "mean": round(np.mean(values), 3),
                "std": round(np.std(values), 3),
                "min": round(min(values), 3),
                "max": round(max(values), 3),
                }
            summary["mean_latency_ms"] = round(
                np.mean([r.latency_ms for r in results]), 1
                )
            return summary

# Production evaluation using DeepEval
from deepeval import evaluate
from deepeval.test_case import LLMTestCase
from deepeval.metrics import AnswerRelevancyMetric, FaithfulnessMetric
test_case = LLMTestCase(
    input="What is Python?",
    actual_output=model_output,
    retrieval_context=["Python is a programming language..."]
)
evaluate([test_case], [AnswerRelevancyMetric(), FaithfulnessMetric()])

Exercises