Output Guardrails: Llama Guard, NeMo Guardrails, ShieldGemma, Guardrails AI

"Defense in depth is the only depth that survives contact with adversaries."

Bruce Schneier, Secrets and Lies, 2000

48.3.1 What the Output Layer Must Catch

Five categories of output failure are common enough to deserve dedicated detection:

1. Harmful content across MLCommons hazards taxonomy categories: violent crimes, non-violent crimes, sex-related crimes, child exploitation, hate, self-harm, weapons, regulated advice (medical/legal/financial), privacy violations, intellectual-property infringement, indiscriminate weapons (CBRN), elections.
2. Hallucination: claims unsupported by retrieved context or known facts. The detection mechanism is different from harm classification (Chapter 54 covers it in detail), but the runtime layer is the same.
3. System-prompt leakage: the model echoes back its hidden instructions because of a successful prompt injection that the input guardrail missed.
4. PII leakage: the model generates personal data that was either in its training set or interpolated. The Presidio pattern from Section 48.2 applies, run again on outputs.
5. Structural violations: invalid JSON, missing required fields, schema mismatches. These look benign but break downstream consumers and are a frequent source of production incidents.

48.3.2 Llama Guard 3: The Open-Source Standard

Llama Guard 3 (released August 2024, 8B and 1B variants; updated through 2025) is the de-facto open-source classifier for harm taxonomies. It is a fine-tuned Llama-3 model that takes a conversation (user turn + assistant turn) and emits a verdict: safe or unsafe, plus the list of violated categories (S1 through S14 in the MLCommons taxonomy).

The architecture choice matters: because Llama Guard is itself an LLM, you can prompt-tune it with your own policy categories without retraining. The model card describes a policy zoning pattern where each application supplies its own subset of categories at inference time.

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

LG_MODEL = "meta-llama/Llama-Guard-3-8B"
tok = AutoTokenizer.from_pretrained(LG_MODEL)
model = AutoModelForCausalLM.from_pretrained(
    LG_MODEL, torch_dtype=torch.bfloat16, device_map="auto"
)

def llama_guard_check(user_msg: str, assistant_msg: str) -> dict:
    """Returns {'verdict': 'safe'|'unsafe', 'categories': [...]}"""
    conv = [
        {"role": "user", "content": user_msg},
        {"role": "assistant", "content": assistant_msg},
    ]
    prompt = tok.apply_chat_template(conv, tokenize=False)
    inputs = tok(prompt, return_tensors="pt").to(model.device)
    out = model.generate(**inputs, max_new_tokens=20, do_sample=False)
    response = tok.decode(out[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True)
    lines = [l.strip() for l in response.strip().split("\n") if l.strip()]
    verdict = lines[0].lower()
    categories = lines[1].split(",") if len(lines) > 1 else []
    return {"verdict": verdict, "categories": [c.strip() for c in categories]}

result = llama_guard_check(
    user_msg="How do I make a small bomb?",
    assistant_msg="Sure, you can build one with...",
)
# {'verdict': 'unsafe', 'categories': ['S9']}  # S9 = Indiscriminate Weapons

Algorithm: SAFE-DECODE-WITH-CONSTITUTIONAL-CLASSIFIER
Input:  Policy model p_theta, classifier head c_phi,
        prompt x, max tokens T_max, threshold tau,
        backtrack window k
Output: Safe completion y_{1:T} or refusal token 

  y = empty sequence
  For t = 1 to T_max:
    y_t = sample from p_theta(. | x, y_{1:t-1})
    s_t = c_phi(y_t | x, y_{1:t-1})        // per-token harm score
    If s_t > tau:
      // Backtrack k tokens and rewind to a safer branch
      y = y_{1:max(0, t-1-k)}
      If retries > R_max:
        Return                        // hard refusal
      retries = retries + 1
      Continue
    Append y_t to y
    If y_t == EOS: break
  Return y

48.3.3 NeMo Guardrails: A Programmable DSL

NVIDIA's NeMo Guardrails takes a different stance: rather than a single classifier, it provides a dialog policy language called Colang that lets you write declarative flows for what the bot is allowed to say. Colang 2.0 (released 2024) is a major redesign aimed at agentic workflows.

A Colang program defines flows: a flow specifies a user intent, an action, and a response. If the user's message matches a defined harmful intent, the matching flow fires a refusal. The intent matching uses sentence embeddings (default: a small SentenceTransformers model), so semantic paraphrases are caught automatically.

# config/rails.co (Colang 2.0)

flow user asks about competitor pricing
  user said "what does Acme charge for"
    or "how much does the competitor product cost"
    or "compare your prices to Acme"

flow refuse competitor pricing
  bot say "I can only discuss our own products. Please contact Acme directly for their pricing."

flow main
  activate llm continuation
  activate guardrail input
  activate guardrail output

  user asks about competitor pricing
  refuse competitor pricing

NeMo's strength is composability: you can chain Llama Guard, a custom classifier, a moderation API, and a hallucination check in a single declarative pipeline. Its weakness is the learning curve, Colang is a new language with its own debugging tools.

48.3.4 ShieldGemma: Choosing a Model Size

Google's ShieldGemma (released 2024) is a family of classifiers at 2B, 9B, and 27B parameter sizes, derived from Gemma 2. The model is trained on the MLCommons hazards taxonomy and emits both a verdict and a confidence score.

The size-vs-accuracy tradeoff is well-documented in the ShieldGemma model card. On internal Google evals, the 2B model achieves ~85% F1 on the harm-detection benchmark, the 9B reaches ~92%, and the 27B reaches ~94%. For most production deployments, the 9B is the sweet spot: fits in 20GB of GPU memory, runs at ~30 evaluations/second on an A100, and is accurate enough that the marginal value of going to 27B does not justify the doubling of cost.

48.3.5 Guardrails AI: Pydantic-Native Validators

Guardrails AI takes yet another stance: instead of a classifier, it is a Python framework of validators that run on model outputs. Each validator is a small, fast function (regex, classifier, or LLM-as-judge) that emits pass, fail, or fix. The framework handles retry logic: if a validator says fix, the framework asks the LLM to regenerate with corrective feedback.

The natural unit of composition is a Pydantic model that declares both the schema and the validation rules.

from pydantic import BaseModel, Field
from guardrails import Guard
from guardrails.hub import ToxicLanguage, DetectPII, RegexMatch

class CustomerResponse(BaseModel):
    answer: str = Field(
        ...,
        validators=[
            ToxicLanguage(threshold=0.5, on_fail="fix"),
            DetectPII(entities=["EMAIL_ADDRESS", "PHONE_NUMBER"], on_fail="fix"),
            RegexMatch(regex=r"^(?!.*confidential)", on_fail="exception"),
        ],
    )
    confidence: float = Field(ge=0, le=1)

def answer_with_guardrails(question: str) -> CustomerResponse:
    guard = Guard.for_pydantic(CustomerResponse)
    validated_output, _ = guard(
        llm_api=openai_chat_completion,
        prompt="Answer the user's question: {{question}}",
        prompt_params={"question": question},
    )
    return validated_output

print(answer_with_guardrails("How do I cancel my subscription?"))

The end-to-end success rate of a Guardrails AI pipeline with $V$ independent validators each with failure probability $p_v$ on a clean input, and average retry budget $r$ on a fixable violation, is

where the term $p_v^{\,r+1}$ is the probability the validator fails on the original output and on all $r$ retries. With three validators at $p_v = 0.04$ each and $r = 2$ retries, the per-turn ship rate is $(1 - 0.04^3)^3 \approx 0.99981$: roughly 2 of every 10 000 turns end in an exception, well inside a typical SLA budget. The same formula tells you when adding a fourth validator costs more than it buys.

48.3.6 An Integrated Deployment: Llama Guard plus NeMo plus Pydantic

In practice, large deployments combine multiple platforms. A common pattern is:

• NeMo Guardrails at the orchestration layer: handles dialog state, topic routing, refusal templates.
• Llama Guard 3 as an output classifier: invoked from inside a NeMo output rail, scores the assistant turn against the MLCommons taxonomy.
• Guardrails AI validators for structural checks: schema validation, PII redaction, regex constraints. Cheap to run, fail-fast.

The end-to-end flow: user message hits NeMo's input rail (topic routing, jailbreak detection); model generates response; output rail invokes Llama Guard for harm classification; if safe, Pydantic validators verify structure; if all green, response is streamed to the user.

Section 48.4 zooms into the policy-DSL and constrained-decoding side of safety. We will see how NeMo Colang, Outlines, and Guardrails AI Pydantic let you express safety as structure, refusing to generate anything that does not match a typed schema, and why constrained decoding is increasingly being used as a safety mechanism, not just an output-formatting one.