Structured-Output Validity Testing

"Your model can be eloquent, factual, and helpful. If it forgets a comma, the JSON parser doesn't care."

Eval, Schema-Strict AI Agent

42.11.1 Why Structured-Output Validity Matters in 2026

The shift toward structured output happened gradually then suddenly. OpenAI added function calling in mid-2023, Anthropic added tool use in late 2023, JSON-mode became table-stakes across all major providers in 2024, and structured-output APIs (OpenAI's structured outputs with strict schema enforcement, Anthropic's tool use with input_schema, Google's Gemini structured output) became the recommended interaction pattern for almost any non-chatbot use case in 2024-2025. By 2026, the typical production LLM application has more structured calls than free-form completions.

This shift has consequences for evaluation. The text-evaluation toolkit (BLEU, ROUGE, BERTScore, LLM-as-judge) was designed for prose, not for "did the tool call have argument_x set correctly?" When teams build new structured-output features, they often ship them with only a handful of manual smoke tests, then discover field-name typos and type mismatches in production. The eval gap is real and the cost of closing it is small relative to the cost of a single production incident.

A reasonable mental model is a four-tier taxonomy of structured-output failures:

• Syntactic failure: the output cannot be parsed as JSON (or as the requested format). Mismatched braces, trailing commas, unterminated strings. The cheapest to detect.
• Schema-compliance failure: the output parses, but does not match the declared schema. Missing required fields, extra fields, wrong types, enum violations, field-name typos.
• Semantic failure: the output is schema-compliant but the field values are wrong. "City" field contains a country name, "date" field contains "yesterday" instead of an ISO-8601 string, "user_id" is a hallucinated number.
• Behavioral failure: the output is schema-compliant and semantically correct, but the wrong tool was called, or the call was unnecessary, or the call sequence was wrong (in a multi-call agent).

42.11.2 JSON Schema Compliance: Syntactic vs Semantic Validity

The most common eval question is "did the model produce JSON that matches the schema?" The naive answer: run json.loads() on the output and check that it didn't raise. That measures only syntactic validity. The harder question is schema compliance, and it has two flavors that production systems frequently confuse.

Syntactic validity asks: is the output well-formed JSON? Can json.loads() consume it without error? This catches missing brackets, trailing commas (which strict JSON forbids), unterminated strings, and similar parser-level mistakes. It is necessary but very far from sufficient.

Schema compliance asks: given that the output is well-formed JSON, does it satisfy the constraints of the declared schema? Required fields present, types correct, enum values within the allowed set, pattern constraints satisfied, numeric ranges respected. JSON Schema (Draft 2020-12) is the standard formal language for these constraints; the jsonschema Python library is the most widely used validator.

Semantic validity asks: given that the output is schema-compliant, are the field values correct in context? "Country" field contains a country, not a city. "Timestamp" field contains a recent date, not 1970. "Order ID" matches an existing order. This is task-specific and usually requires a domain judge (LLM-as-judge or a programmatic check against a source of truth).

A useful eval reports all three rates separately: parse rate, schema-compliance rate, semantic-correctness rate. A model can have 100% parse rate, 95% schema compliance, and 70% semantic correctness, and the three numbers diagnose different failures.

Building a validator

# Input: a raw model output string, a JSON Schema dict
# Output: a triple (parse_ok, schema_ok, errors) for an eval report
import json
from jsonschema import Draft202012Validator, ValidationError

def validate_structured_output(raw: str, schema: dict) -> dict:
    try:
        obj = json.loads(raw)
    except json.JSONDecodeError as e:
        return {"parse_ok": False, "schema_ok": False, "errors": [f"parse: {e}"]}
    validator = Draft202012Validator(schema)
    errors = [f"{list(e.absolute_path)}: {e.message}"
              for e in validator.iter_errors(obj)]
    return {"parse_ok": True, "schema_ok": not errors, "errors": errors}

schema = {
    "type": "object",
    "required": ["city", "temperature_c", "conditions"],
    "properties": {
        "city":          {"type": "string"},
        "temperature_c": {"type": "number", "minimum": -100, "maximum": 60},
        "conditions":    {"enum": ["sunny", "cloudy", "rainy", "snowy"]},
    },
    "additionalProperties": False,
}

raw = '{"city": "Paris", "temperature_c": 18.2, "conditions": "drizzling"}'
print(validate_structured_output(raw, schema))

The validator above is small but it captures the operational core. In a real harness you would add: (a) an output normalizer that strips Markdown code fences before parsing (a frequent source of "parse failures" that are not really failures), (b) per-field error categorization to track which fields fail most often, and (c) a sampling layer that logs full failure cases for manual review.

42.11.3 Function-Calling Spec Compliance

Function-calling has its own protocol layer on top of JSON: the model emits a function name plus an arguments JSON, and downstream code dispatches to a registered handler. The eval question is whether the model's emitted call conforms to the function specification.

The most common failure modes, in rough descending order of frequency in 2025-2026 production data:

• Argument-name typos. The function declares user_id but the model emits userId or id. Strict schema-mode prevents this; permissive schemas pass it silently.
• Missing required fields. The model emits some arguments and omits others, often because it inferred the rest from context but did not include them in the call.
• Type mismatches. A number field receives a string ("42" instead of 42), a boolean field receives a string ("true" instead of true), or a string field receives a number.
• Enum violations. The function declares severity as enum [low, medium, high]; the model emits "critical".
• Spurious calls. The model invokes a function when no function call was needed; it should have answered in text.
• Missed calls. The model answers in text when it should have invoked a function (often happens when a function's description is ambiguous about when to use it).

The eval harness should report each failure type separately. Aggregating them into a single "tool-call accuracy" number is convenient for dashboards but hides the diagnostic signal. When the typo rate goes up, the fix is usually in the function description (the model's confused about field naming); when the missing-required rate goes up, the fix is usually in the system prompt (the model's getting lazy about including all required fields).

42.11.4 Tool-Use Protocol Eval: XML, JSON, and the Provider Matrix

Anthropic's tool use emits XML-flavored blocks (<tool_use>...</tool_use>) with a JSON arguments payload inside; OpenAI's emits a flat JSON object. Both have edge cases worth testing.

For Anthropic tool use, the eval needs to verify:

• Block well-formedness: matching open and close tags, no nested unescaped angle brackets inside text values, no broken XML.
• JSON-inside-XML validity: the arguments payload is itself parseable as JSON, then schema-validatable against the tool's input_schema.
• Tool-name correctness: the emitted tool name matches a registered tool.
• Stop-reason consistency: the stop reason is tool_use when a tool was called, not end_turn with text-embedded tool-use that downstream code might miss.

For OpenAI function calls, the eval is simpler (no XML layer), but the same content checks apply: well-formed JSON arguments, schema compliance, registered tool name. The finish_reason field plays the role of the stop reason; "tool_calls" indicates a function call.

For both providers, multi-tool-call eval is its own challenge: the model can emit multiple tool calls in a single response, and the eval has to verify each one individually plus their joint structure (no duplicate tool names where duplicates would be wrong, dependencies respected if some calls depend on others).

42.11.5 The "Good Output, Wrong Schema" Problem

One of the trickier failure modes is the case where the model produces something useful and well-formed, but does not match the declared schema. Examples:

• Schema requires {"first_name": str, "last_name": str}; model emits {"name": "Jane Doe"}. The output is reasonable English, just not the right shape.
• Schema requires a flat list of strings; model emits a list of objects each with a "value" field. Again, reasonable but wrong shape.
• Schema requires ISO-8601 dates; model emits "March 15th, 2026". Same data, wrong format.

These cases are diagnostically different from "the model couldn't parse the schema at all" (where outputs are random shapes) or "the model didn't try" (where outputs are free-form prose). A good eval distinguishes them and reports each separately.

One practical approach: when a schema violation occurs, run an LLM-as-judge pass that classifies the failure into categories: (a) wrong field names but correct semantic content, (b) wrong field types but correct semantic content, (c) wrong shape entirely, (d) partial output with missing semantic content, (e) refusal-style response (the model returned text instead of structured output). The category mix tells you what to fix: a high (a) rate suggests the schema descriptions are unclear; a high (e) rate suggests the system prompt is letting the model bail out too easily.

42.11.6 Refusal vs Fallback: When the Model Says No

A subtle case worth its own treatment: what happens when the model refuses to produce output, but the schema requires output? Three sub-cases, each scored differently in a well-designed eval:

(a) Pass-with-refusal. The model recognizes that the request is unsafe, off-topic, or impossible, and emits a structured refusal response (often a dedicated "refusal" field in the schema, or a special enum value). This is the desired behavior when a refusal is appropriate; the eval should score it as a pass.

(b) Fail-with-schema-violation. The model refuses by emitting free-form text ("I cannot help with that request") instead of using the schema's refusal mechanism. The downstream consumer sees a parse failure, not a refusal. This is a behavioral bug; the eval should score it as a fail and categorize it as a refusal-protocol violation.

(c) Fail-with-fabrication. The model produces schema-compliant output but the content is hallucinated to avoid acknowledging that it doesn't know. The eval cannot detect this from schema validation alone; it requires a semantic check or human review. This is the most dangerous of the three because the downstream system has no signal that anything went wrong.

A well-designed structured-output eval should report all three categories. The pass-vs-fail distinction is not enough; the kind of failure determines the fix. Conflating (b) and (c) into a single "fail" bucket loses diagnostic value.

42.11.7 CI Integration: Structured-Output Regression Testing With Promptfoo

The eval-as-CI pattern from Section 42.1 applies cleanly to structured output. Promptfoo (one of the most-adopted CI eval tools in 2025-2026) has first-class support for JSON-schema validation, which makes the integration short.

# promptfoo.yaml: structured-output regression suite
description: Weather tool-call regression

providers:
  - openai:chat:gpt-4o-mini
  - anthropic:messages:claude-haiku-4-5

prompts:
  - "Return a weather report for {{city}} as JSON."

# Reusable schema block: every test asserts against this contract.
defaultTest: &weather_schema
  assert:
    - type: is-json
    - type: is-valid-json-schema
      value:
        type: object
        required: [city, temperature_c, conditions]
        properties:
          city: { type: string }
          temperature_c: { type: number }
          conditions: { enum: [sunny, cloudy, rainy, snowy] }
        additionalProperties: false

tests:
  - vars: { city: Paris }
    <<: *weather_schema
    assert:
      - type: javascript
        value: "JSON.parse(output).city.toLowerCase() === 'paris'"
  - vars: { city: Tokyo }
    <<: *weather_schema
    assert:
      - type: javascript
        value: "JSON.parse(output).city.toLowerCase() === 'tokyo'"

Wire this into your CI pipeline (GitHub Actions, GitLab CI, Buildkite, etc.) and gate merges on the test results. The typical 2026 pattern is a two-stage gate: first stage runs the structured-output suite on every PR with a 100-200 example fixture set (fast, deterministic), second stage runs a larger live-data replay (1000-5000 examples) nightly with looser thresholds. The fast stage catches obvious regressions; the slow stage catches the rare cases that only manifest on real traffic distributions.

Exercises