Multi-Judge Ensembles and Production Patterns

"Alone I am partial; in a panel of three I am merely outvoted. Length-controlled, swap-augmented, and ensembled, I am finally too embarrassed to lie."

Eval, Panel-of-Judges Ensemble AI Agent

46.5.1 AlpacaEval and Length-Controlled Debiasing

AlpacaEval (Li et al., 2023) is an automated evaluation framework that uses an LLM judge to compare model outputs against a reference model (typically GPT-4 Turbo) on a curated set of 805 instructions. The standard AlpacaEval metric is the win rate: the percentage of instructions where the judge prefers the evaluated model's output over the reference. While straightforward, the original AlpacaEval metric was heavily influenced by length bias, as models that produced longer outputs consistently achieved higher win rates regardless of actual quality improvements.

AlpacaEval 2 introduced length-controlled (LC) win rates to address this problem. The LC win rate uses a logistic regression model to estimate the expected win rate at a controlled output length, effectively removing the contribution of length to the overall score. This debiasing technique revealed that several models that appeared to outperform GPT-4 on the original AlpacaEval were simply producing longer outputs. Code Fragment 46.5.2 demonstrates how length-controlled debiasing works.

# AlpacaEval length-controlled win rate debiasing
# Install: pip install alpaca-eval
import numpy as np
from sklearn.linear_model import LogisticRegression
def compute_length_controlled_winrate(
    wins: list[bool],
    model_lengths: list[int],
    reference_lengths: list[int],
    ) -> dict:
    """Compute length-controlled win rate using logistic regression.
    The idea: fit a model that predicts win/loss from length difference,
    then compute the expected win rate at zero length difference.
    """
    # Feature: log ratio of model length to reference length
    length_ratios = np.array([
        np.log(ml / rl) for ml, rl in zip(model_lengths, reference_lengths)
        ]).reshape(-1, 1)
    wins_array = np.array(wins, dtype=int)
    # Raw win rate (length-biased)
    raw_winrate = wins_array.mean()
    # Fit logistic regression: P(win) = sigmoid(a * log_length_ratio + b)
    clf = LogisticRegression(fit_intercept=True)
    clf.fit(length_ratios, wins_array)
    # Length-controlled win rate: predict at zero length difference
    lc_winrate = clf.predict_proba(np.array([[0.0]]))[0][1]
    # Length coefficient: positive means length bias exists
    length_coefficient = clf.coef_[0][0]
    return {
        "raw_winrate": round(raw_winrate, 4),
        "lc_winrate": round(lc_winrate, 4),
        "length_coefficient": round(length_coefficient, 4),
        "length_bias_detected": length_coefficient > 0.1,
        "debiasing_effect": round(raw_winrate - lc_winrate, 4),
    }
# Example: a model with high raw win rate but large length bias
# results = compute_length_controlled_winrate(
# wins=[True]*70 + [False]*30,
# model_lengths=[500]*70 + [200]*30, # Wins are mostly on longer outputs
# reference_lengths=[300]*100,
# )
# raw_winrate: 0.70, lc_winrate: 0.55, debiasing_effect: 0.15

from collections import Counter
import statistics
def ensemble_pairwise(verdicts: list[str]) -> str:
    """Majority vote over pairwise judges."""
    return Counter(verdicts).most_common(1)[0][0]
def ensemble_scalar(scores: list[float]) -> dict:
    """Mean and stdev over scalar judges, plus agreement flag."""
    m, s = statistics.mean(scores), statistics.stdev(scores)
    return {"mean": m, "stdev": s, "high_disagreement": s > 0.75}
# Run all three judges on the same item
verdicts = [
    gpt4_pairwise_judge(q, r1, r2),
    claude_pairwise_judge(q, r1, r2),
    prometheus_pairwise_judge(q, r1, r2),
]
final = ensemble_pairwise(verdicts)  # "A" or "B"
# For scalar (e.g. 1-5 rubric scoring):
scores = [geval_gpt4(x), geval_claude(x), prometheus_score(x)]
agg = ensemble_scalar(scores)
# If agg["high_disagreement"] -> escalate to human review.

# 1. Install and clone the registry; 2. define a model-graded eval YAML;
# 3. run it. Outputs JSONL plus a summary written to /tmp/evallogs/.
pip install evals
# evals/registry/evals/my_judge.yaml
# my-judge:
#   id: my-judge.v1
#   description: GPT-4o judges helpfulness on a 1-5 scale
#   metrics: [accuracy]
# my-judge.v1:
#   class: evals.elsuite.modelgraded.classify:ModelBasedClassify
#   args:
#     samples_jsonl: my_judge/samples.jsonl
#     eval_type: cot_classify
#     modelgraded_spec: helpfulness
oaieval gpt-4o-mini my-judge --record_path runs/my-judge.jsonl

You now have a complete LLM-as-Judge toolkit: bias taxonomy, G-Eval scoring, debiasing techniques, trained judge models, and multi-judge ensembles with length-controlled correction. The next part, Part X (Security and Runtime Safety), shifts from measuring model behavior to defending against adversaries who deliberately try to break it. Chapter 47 starts with adversarial security and red-teaming (prompt injection, jailbreaks, data exfiltration), followed by runtime safety, content filtering, and policy enforcement. The connection back to this chapter is direct: every defense mechanism you build in Part X needs evaluation, and the LLM-as-Judge patterns here generalize naturally to safety classifiers (does this response leak the system prompt? does this response violate the policy?). Read Chapter 47 with that lens: the judges become safety auditors.

pip install openai anthropic datasets scipy pandas

Figure 46.5.2 leaves the chapter on its central image: when one judge is not enough, you convene a panel whose members fail in different ways.