Code-Generation Evaluation

"The test for code is not 'does it look like code'. The test for code is 'does it run, does it pass, and does it survive code review next Tuesday'."

Eval, Vibe-Averse AI Agent

43.4.1 The pass@k Metric

The pass@k metric was popularized by the Codex paper (Chen et al., 2021, arXiv:2107.03374). For a given problem, you sample k candidate solutions from the model and check whether at least one of them passes the unit tests. The pass@k for a benchmark is the average of this 0/1 indicator across all problems. The intuition is operational: if a developer is willing to ask the model for k attempts (and is willing to skim each one), the relevant question is whether any one of them works.

A naive estimator of pass@k draws exactly k samples per problem and reports the fraction of problems with at least one passing sample. This estimator has high variance because k is small. The Codex paper introduces an unbiased lower-variance estimator: draw n ≥ k samples, count the number c that pass, and compute pass@k as one minus the probability that a random k-subset contains zero passing samples:

This works whenever n - c ≥ k (otherwise the formula returns 1 trivially, which is correct: if there are more passing samples than the size of any unlucky draw, every k-subset must contain a pass). Using n = 200 with k ∈ {1, 10, 100} is the convention; the variance is dramatically lower than the naive estimator at n = k.

The estimator is easy to misuse. If your n samples are not iid (you generate them with diverse prompts, then average), the formula's variance guarantees no longer hold. If your tests are not deterministic (depend on the current time, random seeds, network state), c is noisy and your pass@k is noisy with it. Always run pass@k evaluation with seeded execution and an upper time budget per test invocation.

43.4.2 HumanEval and HumanEval+

HumanEval (Chen et al., 2021) is the founding benchmark of modern code-generation evaluation. It is 164 Python function-completion problems, each consisting of a docstring with examples and a function signature; the model must produce the function body. A typical problem looks like: "Given a list of integers, return the sum of the squares of the odd numbers." Hidden unit tests verify correctness on inputs not shown in the docstring. The benchmark is small, clean, and runs in seconds; pass@1 has become a standard quick-look quality number for any new code model.

The headline metric pass@$k$ is the probability that at least one of $k$ independently sampled solutions passes all hidden tests. Estimating it naively from a small sample is biased downward, so HumanEval reports the unbiased estimator from Chen et al. (2021): generate $n \ge k$ samples, count $c$ that pass, and use

Per-benchmark pass@$k$ is then averaged across all 164 problems. Reporting $n = 100, k \in \{1, 10, 100\}$ has become standard because the same 100 samples produce all three numbers at no extra inference cost.

By 2024, HumanEval was saturated. GPT-4 reached 67% pass@1 in early 2023; by mid-2024 GPT-4o, Claude-3.5-Sonnet, and DeepSeek-Coder-V2 all cleared 85%; by 2026, frontier models cluster between 92% and 96%. The remaining gap is partly genuine difficulty (a few problems have ambiguous specifications) and partly test-suite incompleteness: HumanEval's hidden test sets are small (median around eight test cases per problem), so a function that handles the typical case but fails edge cases will still pass.

HumanEval+ (Liu et al., 2023, arXiv:2305.01210) addresses this by augmenting HumanEval's test suite roughly 80x using property-based fuzz testing. The same problems are scored against far more rigorous tests. Pass@1 scores drop substantially: a model that scored 85% on HumanEval typically scores 70-75% on HumanEval+, exposing solutions that were happy-path correct but edge-case wrong. HumanEval+ is the version any serious 2026 evaluation reports alongside the original.

43.4.3 MBPP: Mostly Basic Python Problems

MBPP (Austin et al., 2021, arXiv:2108.07732) was released alongside HumanEval as a larger, easier counterpart. The benchmark contains 974 short Python problems, each with a one-sentence natural-language description and three example assertions. Examples include "Write a function to remove all whitespace from a string" and "Write a function to find the volume of a cone given radius and height". MBPP problems are crowdsourced from a pool of programmers (the "Mostly Basic" in the name is sincere) and emphasize standard-library usage more than algorithmic puzzling.

MBPP's role in 2024-2026 evaluation is as a coverage signal that complements HumanEval. Where HumanEval probes algorithmic correctness on isolated functions, MBPP probes whether a model can produce idiomatic uses of common standard-library modules (collections, itertools, re, math). MBPP has a sanitized subset (MBPP-sanitized, 427 problems) that fixes ambiguous specifications and broken test cases. Modern reports cite MBPP-sanitized pass@1.

The HumanEval-MBPP pair has the structural limitation that both benchmarks consist of short, isolated functions. Real software engineering requires reading existing code, understanding cross-file dependencies, and editing modules without breaking other modules. Benchmarks that capture this came later.

43.4.4 SWE-Bench and SWE-Bench Verified

SWE-Bench (Jimenez et al., 2024, arXiv:2310.06770) was the benchmark that finally moved the conversation past function completion. The setup: take 2,294 real GitHub issues from twelve popular Python repositories (Django, Flask, SymPy, Matplotlib, Astropy, requests, scikit-learn, pytest, sphinx, pylint, xarray, pyramid), each with the eventual human-written fix and the test suite that confirmed the fix. The model is given the issue text and the full repo at the parent commit; it must produce a patch that, when applied, causes the previously-failing tests to pass.

This is hard in ways that HumanEval is not. The model must:

• Locate the right file out of hundreds and the right function within that file.
• Understand the existing code, its conventions, and how callers depend on the function under repair.
• Edit multiple files coherently when the fix spans modules.
• Avoid regressions: existing passing tests must continue to pass.

SWE-Bench's metric is resolved rate: the fraction of issues where the model's patch makes the failing test suite pass without breaking any previously-passing tests. Initial 2023 numbers were brutal: GPT-4 with retrieval scored 1.74%, Claude-2 scored 4.8%. The benchmark was clearly measuring something agents could not yet do.

SWE-Bench Verified (OpenAI, August 2024, openai.com/swe-bench-verified) released a manually-curated subset of 500 of the original 2,294 instances. The verification was done by professional software engineers, who flagged and removed instances with problems including: ambiguous issue descriptions, fix-required test cases that were too strict (rejected valid alternative fixes), under-specified requirements, and instances whose ground-truth test suite was broken at the parent commit. Verified is the gold-standard subset; 2025-2026 reports almost always cite both the full SWE-Bench and the Verified subset.

The resolved rate on SWE-Bench Verified is the benchmark that has anchored the 2024-2026 coding-agent leaderboard. Initial Verified numbers were around 33% for the best system (Cognition's Devin, late 2024); by mid-2025, Claude-3.5-Sonnet with a custom agent harness reached 49%; by early 2026, the leaderboard frontier reached approximately 70-72% with extensive scaffolding. The gap between the full benchmark and Verified is informative: an agent that scores 50% on Verified and 35% on full is being penalized by problematic instances, not by its own weakness.

43.4.5 LiveCodeBench and the Contamination Scaling Problem

The deeper problem behind HumanEval saturation and the SWE-Bench Verified curation effort is data contamination: as more code is scraped into pretraining corpora, benchmark problems and their reference solutions are increasingly likely to have been seen during training. A model that "passes" a contaminated benchmark is doing retrieval, not problem-solving, and the headline pass rate is measuring memory rather than capability.

Contamination has been measured directly. The HumanEval audit found roughly 5-10% of problems with verbatim or near-verbatim matches in The Pile, RedPajama, and StarCoder's training data. SWE-Bench's instances draw from public GitHub repositories whose contents and issue threads are routinely scraped; some fraction of the eventual human-written fixes are in training data too. As models train on ever-more-recent crawls, contamination is monotonically increasing.

LiveCodeBench (Jain et al., 2024, arXiv:2403.07974) is the field's most rigorous answer. It continuously ingests programming-contest problems from LeetCode, AtCoder, and Codeforces, time-stamping each with its release date. To evaluate a model, you select the subset of problems released after the model's training-data cutoff. Because contest problems are novel by construction and released at known dates, post-cutoff subsets are guaranteed contamination-free.

LiveCodeBench reports show frontier models performing markedly worse on the post-cutoff subset than on the pre-cutoff subset of the same benchmark, even after controlling for problem difficulty. The gap (roughly 10-25 percentage points for popular models in 2024-2025) is the field's clearest evidence that contamination has been inflating reported numbers.

43.4.6 A Minimal pass@k Harness

The conceptual machinery of pass@k evaluation is small enough to implement in a few dozen lines. The skeleton below is the evaluation loop used by HumanEval-style benchmarks: load problems, sample solutions, execute each in a sandboxed subprocess against the unit tests, collect pass counts, and apply the unbiased estimator. A production harness adds containerization, timeout enforcement, resource limits, and per-test attribution; the skeleton omits these to keep the structure visible.

# Minimal pass@k harness for HumanEval-style problems.
# WARNING: this executes untrusted code. Wrap the runner in a container in real use.
import subprocess, tempfile, math, json
from pathlib import Path

def unbiased_pass_at_k(n: int, c: int, k: int) -> float:
    """Codex (Chen et al., 2021) unbiased estimator of pass@k.
    n: total samples drawn, c: number that passed, k: target k.
    """
    if n - c < k:
        return 1.0
    # 1 - C(n-c, k) / C(n, k)
    return 1.0 - math.comb(n - c, k) / math.comb(n, k)

def run_single(code: str, test_code: str, timeout: float = 10.0) -> bool:
    """Execute code+tests in a subprocess; return True if all tests pass."""
    with tempfile.NamedTemporaryFile(mode="w", suffix=".py", delete=False) as f:
        f.write(code + "\n\n" + test_code + "\n")
        path = f.name
    try:
        result = subprocess.run(
            ["python", path], capture_output=True, timeout=timeout, text=True
        )
        return result.returncode == 0
    except subprocess.TimeoutExpired:
        return False
    finally:
        Path(path).unlink(missing_ok=True)

def evaluate_pass_at_k(problems, generate_fn, n: int = 20, ks=(1, 10)) -> dict:
    """problems: iterable of {prompt, test} dicts.
       generate_fn: callable(prompt) -> code string."""
    per_problem_pass_counts = []
    for prob in problems:
        c = 0
        for _ in range(n):
            candidate = generate_fn(prob["prompt"])
            if run_single(candidate, prob["test"]):
                c += 1
        per_problem_pass_counts.append((n, c))

    results = {}
    for k in ks:
        per_problem_pass_at_k = [
            unbiased_pass_at_k(n_i, c_i, k) for n_i, c_i in per_problem_pass_counts
        ]
        results[f"pass@{k}"] = sum(per_problem_pass_at_k) / len(per_problem_pass_at_k)
    return results

# Example
problems = [
    {"prompt": "def add(a, b):\n    \"\"\"Return a + b.\"\"\"\n",
     "test": "assert add(2, 3) == 5\nassert add(-1, 1) == 0"},
]
# results = evaluate_pass_at_k(problems, my_generate_fn, n=20, ks=(1, 10))
# -> {"pass@1": 0.85, "pass@10": 0.97}

43.4.7 Beyond pass@k: Edit Rate, CodeBERTScore, Build/Lint Signals

pass@k is the headline metric for code generation, but it is not the only metric a production code-assistant team should track. Several complementary metrics capture aspects of code quality that pass/fail unit tests miss.

Code-edit rate: how much a human developer modifies the generated code before merging. A patch that passes tests on the first try but requires extensive renaming, refactoring, or commenting is less valuable than one a developer can merge unchanged. GitHub Copilot's internal metrics report acceptance-rate (was the suggestion accepted?) and edit-rate (how many tokens changed between acceptance and merge). Edit rates correlate with developer satisfaction more closely than raw pass rates do.

CodeBERTScore (Zhou et al., 2023, arXiv:2302.05527) is the code analog of BERTScore: compute contextual embeddings of generated code and reference code, then measure cosine similarity at the token level. It captures semantic similarity that an exact-match metric misses (rename a variable, swap equivalent constructs, get a high score), without requiring executable tests. CodeBERTScore is most useful in domains where running tests is expensive (large repositories), or where multiple valid solutions exist and you want to credit the model for finding any of them.

Build-passes and lint-passes are blunt but practical signals. A generated patch that fails the project's CI build, fails its type checker, or trips its linter will not be merged regardless of whether it nominally passes the unit test for the issue. Production code-assistant evaluations run the full CI pipeline (mypy, pylint, ruff, project-specific tooling) against the patch and report build-pass-rate and lint-pass-rate as separate numbers. These metrics also correlate with the contamination-resistant signal that the model has actually learned the project's conventions.

BigCodeBench (Zhuo et al., 2024, arXiv:2406.15877) is worth mentioning separately. It tests Python problems that require coordinated use of multiple libraries (e.g., NumPy + Pandas + Matplotlib in one solution), a closer match to real data-science and engineering work than the single-library focus of HumanEval and MBPP. It is the benchmark that has best exposed weaknesses in models that memorized common idioms but cannot compose them.

43.4.8 The Contamination Arms Race in Practice

The contamination scaling problem has changed how code-generation evaluation is run in practice. Three norms have emerged in the 2024-2026 period.

Norm 1: Report both contaminated and contamination-resistant numbers. A 2026 model card typically lists HumanEval, HumanEval+, MBPP, SWE-Bench Verified, and the LiveCodeBench post-cutoff subset. The first three are useful for backwards comparability; the last two are the numbers you trust.

Norm 2: Disclose training-data cutoff alongside benchmark scores. A LiveCodeBench number is uninterpretable without a cutoff date. Model cards now report cutoff explicitly so that readers can compute which benchmark subset was eligible.

Norm 3: Avoid public benchmark optimization. Teams that publish code models have begun to hold out portions of their internal evaluation suites from external dissemination, on the rationale that a fully-public benchmark is a fully-contaminated benchmark within a year of release. Anthropic's RE-Bench, OpenAI's internal code-evals, and Google DeepMind's code-eval portfolio all include private subsets reported only as aggregate numbers.

pip install evalplus openai psutil tqdm

Exercises