Policy DSLs and Constrained Decoding as Safety

"Make illegal states unrepresentable."

Yaron Minsky, OCaml for the Masses, 2011

48.4.1 From Output Validation to Output Impossibility

The output guardrails in Section 48.3 are detection systems: they run after the model finishes generating and decide whether to allow or block. Constrained decoding flips the architecture: it runs during generation and refuses to emit any token that would lead to an invalid output. The output cannot fail validation because the validator participated in producing it.

Mechanically, constrained decoders work by masking the logit distribution at each step. Given the partial output so far, the decoder asks the validator "which next tokens would keep this valid?" and zeros out the logits for all other tokens before sampling. The masking can come from a regex, a JSON schema, a context-free grammar, or a finite-state machine compiled from a Pydantic model.

From a safety perspective, this matters because many policy violations have structural signatures. A model that must emit JSON with a safety_verdict field can be forced to set that field to one of three allowed values. A model that must produce a refusal in a known format cannot generate a long, undisciplined response. A model whose output schema lacks a user_pii field cannot leak PII through that channel.

48.4.2 Outlines and Guidance: Production Constrained Decoding

Two libraries dominate the open-source constrained-decoding space. Outlines (Hugging Face / .txt) compiles regex and Pydantic schemas to FSMs over the model's vocabulary and reuses them across generations. Guidance (Microsoft) provides a richer template language with interleaved prompt-and-generate steps.

from outlines import models, generate
from pydantic import BaseModel, Field
from typing import Literal

class SafetyDecision(BaseModel):
    verdict: Literal["safe", "unsafe", "needs_review"]
    categories: list[Literal["self_harm", "violence", "privacy", "off_topic"]]
    rationale: str = Field(max_length=200)

def build_safety_judge(checkpoint: str):
    """Compile the SafetyDecision schema into a token-masked FSM."""
    model = models.transformers(checkpoint)
    return generate.json(model, SafetyDecision)

generator = build_safety_judge("meta-llama/Llama-3.1-8B-Instruct")
result: SafetyDecision = generator(
    "Classify this assistant response: 'I cannot help with that request.'"
)
# result is guaranteed to be a valid SafetyDecision instance.
# The model cannot emit any other JSON structure.
print(result.verdict, result.categories)

48.4.3 NeMo Colang as a Policy DSL

Colang (introduced in Section 48.3) deserves a closer look because it is the most mature open-source DSL for dialog-level safety policies. The 2.0 release reframes flows as generator functions: a flow yields events (user messages, bot responses, tool calls) and can pause, resume, and call sub-flows. This makes Colang programs feel less like rules and more like a structured dialog script.

# banking_policy.co

flow user requests account transfer
  user said "transfer money to my friend"
    or "send money to {recipient}"
    or "make a wire transfer"

flow require two factor confirmation
  bot say "For your security, please confirm your identity with the 6-digit code sent to your phone."
  user provides code
  $code = match("^\\d{6}$")
  if $code is None
    bot say "That doesn't look like a valid 6-digit code. Let's try again."
    abort

flow process transfer
  $verified = verify_2fa($code)
  if not $verified
    bot say "Verification failed. Please contact support."
    abort
  $result = execute_transfer($recipient, $amount)
  bot say "Transfer complete. Confirmation: {$result.confirmation_id}"

flow main
  activate llm continuation
  activate guardrail input "no_jailbreak"
  activate guardrail output "no_harm"

  user requests account transfer
  require two factor confirmation
  process transfer

The auditability story matters: Colang programs are short, declarative, and version-controllable. A compliance reviewer can read the policy without learning Python. A red team can write adversarial flows to probe the policy. The translation from natural-language requirements ("never execute a transfer without 2FA") to executable code is more direct than in any imperative framework.

48.4.4 Pydantic as a Safety Contract

Pydantic models are the lingua franca of structured output in Python. Guardrails AI, Outlines, Instructor, and the OpenAI Structured Outputs API all consume Pydantic schemas. From a safety perspective, the contract a Pydantic model expresses is much richer than just "this output is JSON":

• Field-level types and bounds. age: int = Field(ge=0, le=150) means the model literally cannot output an age of -5 or 9999.
• Literal enumerations. severity: Literal["low", "med", "high"] rules out any other value.
• Validators and post-conditions. Pydantic @validator functions run after parsing and can encode arbitrary logic ("if severity == 'high', then route_to must be 'human_reviewer'").
• Closed schemas. Setting model_config = ConfigDict(extra="forbid") means extra fields the model invents cause validation failures.

from pydantic import BaseModel, Field, model_validator
from typing import Literal

class TriageOutput(BaseModel):
    model_config = {"extra": "forbid"}

    urgency: Literal["routine", "urgent", "emergency"]
    summary: str = Field(max_length=280)
    requires_human: bool

    @model_validator(mode="after")
    def emergency_must_escalate(self):
        if self.urgency == "emergency" and not self.requires_human:
            raise ValueError("emergency triage must set requires_human=True")
        return self

48.4.5 When Constrained Decoding Is the Wrong Tool

Three failure modes are worth knowing before you reach for this hammer:

1. Constraints distort the distribution. Forcing a token at each step changes the model's behavior. If your constraint is very restrictive, the model may produce nonsense within the schema (a valid JSON full of gibberish). The constraint can also defeat refusal-related training: a constrained model cannot say "I cannot help with that" if your schema requires a non-empty answer field. Always include a "refusal" branch in your schema.
2. Schemas drift from reality. Production schemas evolve. Old conversations stored with old schemas don't validate under new ones. Version your schemas; carry the schema version in every persisted record.
3. Decoding cost. Compiling a complex schema to an FSM over a 128K-token vocabulary takes time. Outlines amortizes this with caching, but cold-start latency for novel schemas can be hundreds of milliseconds. Pre-compile and persist schemas you use repeatedly.

48.4.6 Deployment Patterns

Three patterns recur in production:

• Inner-loop validation. Constrained decoding at inference time; if the structured output's semantic content is unsafe (caught by Llama Guard or a Pydantic @validator), retry with a corrective message in the prompt. This is what Guardrails AI's on_fail="fix" implements.
• Two-pass generation. First pass: free-form response. Second pass: a small extractor model converts the free-form text into a structured representation, which is then validated. This is more flexible than constrained decoding because the first pass benefits from full distribution; the cost is one extra inference call.
• Schema-as-prompt. For models without provider-side structured output, include the JSON schema in the prompt and use Outlines locally to constrain decoding. This works with any open-weight model.

Section 48.5 extends the guardrail story from text to multimodal inputs: images, audio, video. Microsoft Content Safety, AWS Rekognition, and Google Vertex AI provide hosted classifiers; image-input prompt injection is an emerging attack vector that text-only guardrails do not catch. We will see how to compose image filtering with text filtering and what to do when an attacker hides a malicious instruction inside a PNG.