Explaining Transformers

"Every prediction has a story. Attribution methods are how we make the model tell it."

Probe, Story Extracting AI Agent

1. The Explanation Problem

When a transformer model predicts a token, its prediction results from hundreds of attention heads and MLP layers interacting across dozens of layers. Explaining this prediction means answering some form of "which input tokens mattered and why?" Different explanation methods operationalize this question differently, leading to genuinely different (and sometimes contradictory) answers.

The core tension is between faithfulness (does the explanation accurately reflect the model's actual computation?) and plausibility (does the explanation make intuitive sense to a human?). An explanation that perfectly traces the model's computation might be incomprehensible, while an intuitively appealing explanation might not accurately reflect what the model actually did.

2. Attention Rollout

Raw attention weights from a single layer show direct token-to-token attention. But information flows through multiple layers, so a token at position 5 might influence position 10 indirectly by first attending to position 7, which then attends to position 10. Attention rollout (Abnar and Zuidema, 2020) accounts for this multi-hop information flow by multiplying attention matrices across layers. Figure 18.4.1 compares raw attention with attention rollout on the same input. Code Fragment 18.4.1 shows this approach in practice.

Figure 18.4.1: Raw attention shows only direct attention at one layer. Attention rollout traces information flow across all layers by multiplying attention matrices, capturing indirect paths.

The following implementation computes attention rollout by multiplying attention matrices across all layers, accounting for residual connections at each step.

Code Fragment 18.4.1 visualizes attention patterns.

# Attention Rollout Implementation
import torch
import numpy as np

def attention_rollout(
 attentions,
 head_fusion="mean",
 discard_ratio=0.0,
):
 """
 Compute attention rollout across all layers.

 Args:
 attentions: tuple of attention tensors, one per layer
 Each has shape (batch, num_heads, seq_len, seq_len)
 head_fusion: how to combine heads ("mean", "max", "min")
 discard_ratio: fraction of lowest attention weights to zero out

 Returns:
 rollout: (seq_len, seq_len) matrix of accumulated attention
 """
 num_layers = len(attentions)
 seq_len = attentions[0].shape[-1]

 # Start with identity (each token attends to itself)
 rollout = torch.eye(seq_len)

 for layer_idx in range(num_layers):
 attn = attentions[layer_idx].squeeze(0) # (heads, seq, seq)

 # Fuse attention heads
 if head_fusion == "mean":
 attn_fused = attn.mean(dim=0)
 elif head_fusion == "max":
 attn_fused = attn.max(dim=0).values
 elif head_fusion == "min":
 attn_fused = attn.min(dim=0).values

 # Optionally discard low-attention connections
 if discard_ratio > 0:
 flat = attn_fused.flatten()
 threshold = flat.quantile(discard_ratio)
 attn_fused = attn_fused * (attn_fused >= threshold)
 # Re-normalize rows
 attn_fused = attn_fused / attn_fused.sum(dim=-1, keepdim=True)

 # Add residual connection (identity)
 attn_with_residual = 0.5 * attn_fused + 0.5 * torch.eye(seq_len)

 # Multiply with cumulative rollout
 rollout = attn_with_residual @ rollout

 return rollout

# Usage
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("gpt2", output_attentions=True)
tokenizer = AutoTokenizer.from_pretrained("gpt2")

text = "The cat sat on the mat because it was very tired"
inputs = tokenizer(text, return_tensors="pt")
outputs = model(**inputs)

rollout = attention_rollout(outputs.attentions)
tokens = tokenizer.convert_ids_to_tokens(inputs["input_ids"][0])

# Show which tokens "it" (position 7) attends to after rollout
it_idx = tokens.index("it") if "it" in tokens else 7
print(f"Attention rollout from '{tokens[it_idx]}':")
for i, (token, score) in enumerate(zip(tokens, rollout[it_idx])):
 bar = "#" * int(score.item() * 40)
 print(f" {token:10s} {bar} ({score.item():.3f})")

# Production equivalent using BertViz
from bertviz import model_view, head_view
from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained("gpt2", output_attentions=True)
tokenizer = AutoTokenizer.from_pretrained("gpt2")
inputs = tokenizer("The cat sat on the mat", return_tensors="pt")
outputs = model(**inputs)
head_view(outputs.attentions, tokenizer.convert_ids_to_tokens(inputs["input_ids"][0]))

# Quick comparison: same task in TransformerLens vs nnsight vs nnterp

# === TransformerLens: full hook access, custom API ===
from transformer_lens import HookedTransformer
tl_model = HookedTransformer.from_pretrained("gpt2-small")
logits, cache = tl_model.run_with_cache("The cat sat on")
# Access any intermediate activation:
layer5_resid = cache["blocks.5.hook_resid_post"] # (1, seq, d_model)

# === nnsight: wrap any PyTorch model, proxy API ===
from nnsight import LanguageModel
nn_model = LanguageModel("gpt2", device_map="auto")
with nn_model.trace("The cat sat on") as tracer:
 layer5_out = nn_model.transformer.h[5].output[0].save()
# Access saved activation after trace:
layer5_resid_nn = layer5_out.value # same shape

# === nnterp: lightweight, Hugging Face native ===
from nnterp import load_model, logit_lens
model, tokenizer = load_model("gpt2")
# Built-in logit lens with one call:
layer_predictions = logit_lens(model, tokenizer, "The cat sat on")
# Returns per-layer top predictions without manual unembedding

Attention rollout and the logit lens reveal where the model looks and what it predicts at each layer. But neither method tells us which attended tokens actually matter for the final output. To answer that question, we need to incorporate gradient information.

3. Gradient-Weighted Attention

Gradient-weighted attention (also called Attention × Gradient) combines attention weights with gradient information. The intuition is that attention tells us where the model looks, while gradients tell us how sensitive the output is to what the model finds there. Multiplying these signals highlights tokens that the model both attends to and that actually influence the prediction. Code Fragment 18.4.2 shows this approach in practice.

Code Fragment 18.4.2 combines attention weights with gradient information, highlighting tokens that both receive high attention and significantly influence the output.

# Gradient-Weighted Attention
import torch

def gradient_weighted_attention(
 model,
 tokenizer,
 text,
 target_pos=-1,
):
 """
 Compute gradient-weighted attention for each layer and head.

 Returns attention weights scaled by the gradient of the output
 with respect to the attention weights themselves.
 """
 model.eval()
 inputs = tokenizer(text, return_tensors="pt")
 tokens = tokenizer.convert_ids_to_tokens(inputs["input_ids"][0])

 # Forward pass with attention outputs
 outputs = model(**inputs, output_attentions=True)
 attentions = outputs.attentions # tuple of (1, heads, seq, seq)

 # Get the predicted token logit
 logits = outputs.logits[0, target_pos]
 predicted_token_id = logits.argmax()
 target_logit = logits[predicted_token_id]

 # Compute gradient of target logit w.r.t. each attention matrix
 grad_weighted = []
 for layer_attn in attentions:
 if layer_attn.requires_grad:
 grad = torch.autograd.grad(
 target_logit, layer_attn, retain_graph=True
 )[0]
 # Element-wise multiply: attention * gradient
 weighted = (layer_attn * grad).squeeze(0)
 # Average over heads
 weighted = weighted.mean(dim=0).detach()
 grad_weighted.append(weighted)

 # Stack and average across layers
 all_layers = torch.stack(grad_weighted) # (layers, seq, seq)
 combined = all_layers.mean(dim=0) # (seq, seq)

 return combined, tokens

# Alternative: enable gradients for attention
# Need to use hooks to capture attention with grad enabled
def compute_attention_gradient_attribution(model, tokenizer, text):
 """Simpler approach using hooks to capture attention gradients."""
 attention_grads = {}

 def save_attention_grad(name):
 def hook(module, grad_input, grad_output):
 attention_grads[name] = grad_output[0].detach()
 return hook

 # Register backward hooks on attention layers
 hooks = []
 for i, layer in enumerate(model.transformer.h):
 h = layer.attn.register_full_backward_hook(
 save_attention_grad(f"layer_{i}")
 )
 hooks.append(h)

 # Forward + backward
 inputs = tokenizer(text, return_tensors="pt")
 outputs = model(**inputs, output_attentions=True)
 target = outputs.logits[0, -1].max()
 target.backward()

 # Clean up hooks
 for h in hooks:
 h.remove()

 return attention_grads

4. Layer-wise Relevance Propagation (LRP)

Layer-wise Relevance Propagation redistributes the model's output score backward through the network, assigning a relevance value to each neuron at each layer. At the input layer, these relevance values become token-level attributions. LRP satisfies a conservation property: the total relevance at each layer equals the output score, ensuring nothing is lost or created during propagation.

The following simplified LRP implementation propagates relevance backward through linear layers using the epsilon rule for numerical stability.

# Layer-wise Relevance Propagation for Transformers (simplified)
import torch
import torch.nn as nn

class TransformerLRP:
 """Simplified LRP for transformer models."""

 def __init__(self, model, epsilon=1e-6):
 self.model = model
 self.epsilon = epsilon
 self.activations = {}

 def register_hooks(self):
 """Register forward hooks to capture activations."""
 self.hooks = []
 for name, module in self.model.named_modules():
 if isinstance(module, nn.Linear):
 h = module.register_forward_hook(
 self._save_activation(name)
 )
 self.hooks.append(h)

 def _save_activation(self, name):
 def hook(module, input, output):
 self.activations[name] = {
 "input": input[0].detach(),
 "output": output.detach(),
 "weight": module.weight.detach(),
 }
 return hook

 def propagate_linear(self, relevance, layer_name):
 """LRP-epsilon rule for a linear layer."""
 act = self.activations[layer_name]
 z = act["input"] @ act["weight"].T # pre-activation
 z = z + self.epsilon * z.sign() # stabilize
 s = relevance / z
 c = s @ act["weight"]
 relevance_input = act["input"] * c
 return relevance_input

 def attribute(self, text, tokenizer):
 """Compute LRP attribution for input tokens."""
 self.register_hooks()

 inputs = tokenizer(text, return_tensors="pt")
 with torch.no_grad():
 outputs = self.model(**inputs)

 # Start with the output logits as initial relevance
 logits = outputs.logits[0, -1]
 relevance = torch.zeros_like(logits)
 relevance[logits.argmax()] = logits.max()

 # Propagate backward through the network
 # (simplified: in practice, need to handle attention specially)
 for name in reversed(list(self.activations.keys())):
 if name.startswith("lm_head") or "mlp" in name:
 relevance = self.propagate_linear(relevance, name)

 # Clean up
 for h in self.hooks:
 h.remove()

 return relevance

5. Perturbation-Based Explanations

Perturbation methods explain predictions by measuring how the output changes when parts of the input are modified or removed. Unlike gradient-based methods (which measure local sensitivity), perturbation methods measure actual counterfactual impact: what would the model predict if this token were absent? Code Fragment 18.4.4 shows this approach in practice.

The following implementations measure token importance through direct removal: leave-one-out tests individual tokens, while sliding window occlusion captures multi-token patterns.

# Perturbation-based attribution methods
import torch
import numpy as np

def leave_one_out_attribution(model, tokenizer, text):
 """
 Measure each token's importance by removing it and
 observing the change in prediction confidence.
 """
 inputs = tokenizer(text, return_tensors="pt")
 tokens = tokenizer.convert_ids_to_tokens(inputs["input_ids"][0])

 # Get baseline prediction
 with torch.no_grad():
 baseline_logits = model(**inputs).logits[0, -1]
 baseline_probs = torch.softmax(baseline_logits, dim=-1)
 predicted_id = baseline_probs.argmax()
 baseline_prob = baseline_probs[predicted_id].item()

 # Remove each token and measure impact
 attributions = []
 for i in range(len(tokens)):
 # Create input with token i replaced by padding/mask
 perturbed_ids = inputs["input_ids"].clone()
 perturbed_ids[0, i] = tokenizer.pad_token_id or 0

 with torch.no_grad():
 perturbed_logits = model(perturbed_ids).logits[0, -1]
 perturbed_prob = torch.softmax(perturbed_logits, dim=-1)[predicted_id].item()

 # Attribution = drop in probability when token is removed
 attribution = baseline_prob - perturbed_prob
 attributions.append(attribution)

 return np.array(attributions), tokens

def sliding_window_occlusion(
 model, tokenizer, text, window_size=3
):
 """
 Occlude a sliding window of tokens to find important regions.
 More robust than single-token removal for capturing multi-token patterns.
 """
 inputs = tokenizer(text, return_tensors="pt")
 tokens = tokenizer.convert_ids_to_tokens(inputs["input_ids"][0])
 seq_len = len(tokens)

 with torch.no_grad():
 baseline_logits = model(**inputs).logits[0, -1]
 predicted_id = baseline_logits.argmax()
 baseline_score = baseline_logits[predicted_id].item()

 region_scores = []
 for start in range(seq_len - window_size + 1):
 perturbed_ids = inputs["input_ids"].clone()
 for j in range(start, start + window_size):
 perturbed_ids[0, j] = tokenizer.pad_token_id or 0

 with torch.no_grad():
 perturbed_logits = model(perturbed_ids).logits[0, -1]
 score = perturbed_logits[predicted_id].item()

 impact = baseline_score - score
 region_scores.append({
 "start": start,
 "end": start + window_size,
 "tokens": tokens[start:start + window_size],
 "impact": impact,
 })

 # Sort by impact
 region_scores.sort(key=lambda x: x["impact"], reverse=True)
 return region_scores

6. Comparing Explanation Methods

Different explanation methods can produce substantially different attributions for the same prediction. Choosing the right method requires understanding what each method measures and what properties matter for your use case. Figure 18.4.2 positions the major methods on a faithfulness versus computational cost axis.

Figure 18.4.2: Approximate positioning of explanation methods on faithfulness vs. computational cost. Perturbation-based methods are most faithful but most expensive; raw attention is cheapest but least faithful.

Code Fragment 18.4.5 runs multiple attribution methods on the same input and computes rank correlations to measure agreement between them.

# Unified comparison framework for explanation methods
import torch
import numpy as np
from typing import Dict, Callable, List

def compare_attribution_methods(
 model,
 tokenizer,
 text: str,
 methods: Dict[str, Callable],
) -> Dict[str, np.ndarray]:
 """
 Run multiple attribution methods on the same input
 and compare their outputs.
 """
 results = {}
 tokens = tokenizer.convert_ids_to_tokens(
 tokenizer(text, return_tensors="pt")["input_ids"][0]
 )

 for name, method_fn in methods.items():
 attributions = method_fn(model, tokenizer, text)
 # Normalize to [0, 1] for comparison
 attr_min = attributions.min()
 attr_max = attributions.max()
 if attr_max > attr_min:
 normalized = (attributions - attr_min) / (attr_max - attr_min)
 else:
 normalized = np.zeros_like(attributions)
 results[name] = normalized

 # Compute agreement metrics
 method_names = list(results.keys())
 print(f"Attribution comparison for: '{text}'")
 print(f"Predicted next token: {get_prediction(model, tokenizer, text)}")
 print()

 # Rank correlation between methods
 from scipy.stats import spearmanr
 print("Spearman rank correlations:")
 for i, name_a in enumerate(method_names):
 for name_b in method_names[i+1:]:
 corr, pval = spearmanr(results[name_a], results[name_b])
 print(f" {name_a} vs {name_b}: rho={corr:.3f} (p={pval:.3f})")

 # Top-3 tokens per method
 print("\nTop-3 most important tokens per method:")
 for name, attrs in results.items():
 top_3 = np.argsort(attrs)[-3:][::-1]
 top_tokens = [(tokens[i], attrs[i]) for i in top_3]
 top_str = ", ".join(f"'{t}' ({s:.2f})" for t, s in top_tokens)
 print(f" {name:20s}: {top_str}")

 return results

# Run comparison
methods = {
 "raw_attention": lambda m, t, x: extract_raw_attention(m, t, x),
 "rollout": lambda m, t, x: compute_rollout(m, t, x),
 "integrated_gradients": lambda m, t, x: integrated_gradients(m, t, x)[0],
 "leave_one_out": lambda m, t, x: leave_one_out_attribution(m, t, x)[0],
}

results = compare_attribution_methods(
 model, tokenizer,
 "The Eiffel Tower is located in the city of",
 methods
)

7. Interpretability Tools Ecosystem (2025)

The interpretability research community has built a rich ecosystem of open-source tools over the past three years. Choosing the right tool depends on your goal: are you doing mechanistic circuit analysis, exploring SAE features, or building a production explanation pipeline? This section surveys the major tools and helps you match tools to use cases.

Why does the tooling matter? Interpretability research is only as reproducible and accessible as its tooling. A brilliant mechanistic finding that requires custom infrastructure to replicate has limited impact. The tools listed below have lowered the barrier to entry, enabling researchers and practitioners to run experiments that previously required months of infrastructure work.

7.1 Production XAI Libraries: Captum, LIME, and BertViz

The tools above (TransformerLens, SAELens, nnsight) serve the mechanistic interpretability community, where the goal is understanding internal model computations. A complementary set of tools addresses the production explainability problem: generating human-readable explanations of individual predictions for end users, auditors, or regulatory compliance. These libraries treat the model as a function (sometimes a black box) and explain its input-output behavior rather than its internal circuits.

Captum: Meta's Attribution Toolkit

Captum is Meta's comprehensive model interpretability library for PyTorch. It implements over a dozen attribution methods under a unified API, making it straightforward to compare different explanation approaches on the same prediction. For transformer models, the most commonly used methods are Layer Integrated Gradients (attributing to the embedding layer), Layer Gradient x Activation, and Layer Conductance (which measures the importance of individual neurons in a specific layer).

Captum's strength is its breadth: it covers gradient-based methods (Integrated Gradients, DeepLift, GradientSHAP), perturbation-based methods (Feature Ablation, Shapley Value Sampling, LIME via the Lime wrapper), and layer-level methods (Layer Conductance, Internal Influence). This means you can compare multiple explanation strategies on the same model without switching libraries.

# Comprehensive Captum attribution for a transformer classifier
import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer
from captum.attr import (
 LayerIntegratedGradients,
 LayerGradientXActivation,
 LayerConductance,
 visualization as viz,
)

model_name = "distilbert-base-uncased-finetuned-sst-2-english"
model = AutoModelForSequenceClassification.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)
model.eval()

text = "The movie was surprisingly entertaining despite a weak script"
inputs = tokenizer(text, return_tensors="pt")
input_ids = inputs["input_ids"]
baseline_ids = torch.zeros_like(input_ids) # PAD token baseline

# Wrap the forward function for Captum
def forward_func(input_ids):
 outputs = model(input_ids)
 return outputs.logits[:, 1] # positive sentiment logit

# Method 1: Layer Integrated Gradients (most common for transformers)
lig = LayerIntegratedGradients(forward_func, model.distilbert.embeddings)
attrs_ig, delta = lig.attribute(
 input_ids, baselines=baseline_ids,
 n_steps=50, return_convergence_delta=True,
)
# Convergence delta should be small (< 0.05); large values indicate
# that n_steps is too low for accurate integration.

# Method 2: Gradient x Activation (faster, less theoretically grounded)
lga = LayerGradientXActivation(forward_func, model.distilbert.embeddings)
attrs_gxa = lga.attribute(input_ids)

# Method 3: Layer Conductance (neuron-level importance in a specific layer)
lc = LayerConductance(forward_func, model.distilbert.transformer.layer[3])
attrs_cond = lc.attribute(input_ids, baselines=baseline_ids, n_steps=20)

# Summarize per-token attributions
tokens = tokenizer.convert_ids_to_tokens(input_ids[0])
attrs_sum = attrs_ig.sum(dim=-1).squeeze(0).detach().numpy()
print("Integrated Gradients attribution per token:")
for tok, score in zip(tokens, attrs_sum):
 bar = "#" * int(min(abs(score) * 10, 40))
 sign = "+" if score > 0 else "-"
 print(f" {tok:20s} {sign}{bar:40s} {score:+.4f}")

LIME for Language Models

LIME (Local Interpretable Model-agnostic Explanations) explains individual predictions by fitting a simple interpretable model (typically a sparse linear model) to the behavior of the complex model in the neighborhood of a specific input. For text, LIME works by randomly removing words from the input, observing how the model's prediction changes, and fitting a linear model that approximates the local decision boundary.

LIME's key advantage is that it is entirely model-agnostic: it treats the model as a black box and requires only the ability to call the model's prediction function. This makes it applicable to API-based LLMs where you have no access to gradients or internal activations. The tradeoff is that LIME's perturbation strategy (removing tokens) can create out-of-distribution inputs that a language model has never seen during training, potentially producing misleading attributions.

# LIME explanation for a text classifier (model-agnostic)
from lime.lime_text import LimeTextExplainer
import numpy as np

# Works with any model that returns class probabilities
def predict_proba(texts):
 """Prediction function that LIME will call repeatedly."""
 results = []
 for text in texts:
 inputs = tokenizer(text, return_tensors="pt",
 truncation=True, max_length=512)
 with torch.no_grad():
 logits = model(**inputs).logits[0]
 probs = torch.softmax(logits, dim=-1).numpy()
 results.append(probs)
 return np.array(results)

explainer = LimeTextExplainer(class_names=["negative", "positive"])

text = "The movie was surprisingly entertaining despite a weak script"
explanation = explainer.explain_instance(
 text,
 predict_proba,
 num_features=10, # top 10 most important words
 num_samples=1000, # number of perturbations to generate
)

# Display word-level importance
print("LIME feature importance (positive sentiment):")
for word, weight in explanation.as_list():
 direction = "+" if weight > 0 else "-"
 bar = "#" * int(abs(weight) * 50)
 print(f" {word:20s} {direction}{bar} ({weight:+.4f})")

# LIME also provides HTML visualization:
# explanation.save_to_file("lime_explanation.html")

# For API-based LLMs (no gradient access), LIME is often the only option.
# Replace predict_proba with an API call wrapper:
#
# def predict_proba_api(texts):
# results = []
# for text in texts:
# response = client.chat.completions.create(
# model="gpt-4o-mini",
# messages=[{"role": "user", "content": f"Classify: {text}"}],
# logprobs=True,
# )
# # Extract probabilities from logprobs
# results.append(parse_logprobs(response))
# return np.array(results)

BertViz: Interactive Attention Visualization

BertViz provides interactive, browser-based visualizations of attention patterns across all layers and heads of a transformer model. It offers three visualization modes: the head view (attention from a single head as lines connecting tokens), the model view (all heads across all layers in a compact overview), and the neuron view (how individual neurons in Q, K, V contribute to attention). BertViz works in Jupyter notebooks and supports BERT, GPT-2, RoBERTa, XLNet, and other Hugging Face models.

While Section 18.1 covered attention visualization from scratch, BertViz is the production tool for this task. It is particularly useful for qualitative exploration: scanning attention patterns across layers to identify which heads attend to syntactic structure, which heads focus on positional patterns, and which heads appear to implement specific linguistic functions like coreference resolution or subject-verb agreement.

# BertViz: interactive attention visualization in Jupyter
# pip install bertviz
from bertviz import model_view, head_view, neuron_view
from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained("bert-base-uncased",
 output_attentions=True)
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")

text = "The bank raised interest rates after the financial crisis"
inputs = tokenizer(text, return_tensors="pt")
outputs = model(**inputs)

attention = outputs.attentions # tuple of (batch, heads, seq, seq) per layer
tokens = tokenizer.convert_ids_to_tokens(inputs["input_ids"][0])

# Model view: compact overview of all layers and heads
model_view(attention, tokens)

# Head view: detailed view for a specific layer
head_view(attention, tokens, layer=6, heads=[0, 3, 7])

# Neuron view: see Q/K/V contributions (requires special model loading)
# neuron_view(model, tokenizer, text, layer=6, head=3)

# Practical usage pattern: identify interesting heads, then investigate
# with probing classifiers (Section 18.1) or activation patching
# (Section 18.2) to confirm functional roles.

8. Evaluation of Explanation Quality

How do we know if an explanation is "good"? Several metrics have been proposed to evaluate explanation quality, each capturing different desirable properties.

Code Fragment 18.4.10 demonstrates this approach.

Code Fragment 18.4.10 evaluates whether the attributed tokens actually drive the prediction, using both sufficiency (keeping only top-k) and comprehensiveness (removing top-k) tests.

# Faithfulness evaluation for attribution methods
def evaluate_faithfulness(
 model,
 tokenizer,
 text,
 attributions,
 k_values=[1, 3, 5],
):
 """
 Evaluate faithfulness of attributions using sufficiency and comprehensiveness.
 """
 inputs = tokenizer(text, return_tensors="pt")
 tokens = tokenizer.convert_ids_to_tokens(inputs["input_ids"][0])

 with torch.no_grad():
 baseline_logits = model(**inputs).logits[0, -1]
 predicted_id = baseline_logits.argmax()
 baseline_prob = torch.softmax(baseline_logits, dim=-1)[predicted_id].item()

 sorted_indices = np.argsort(attributions)[::-1]
 results = {}

 for k in k_values:
 top_k = sorted_indices[:k]

 # Sufficiency: keep only top-k tokens
 sufficient_ids = inputs["input_ids"].clone()
 mask = torch.ones(len(tokens), dtype=torch.bool)
 mask[list(top_k)] = False
 sufficient_ids[0, mask] = tokenizer.pad_token_id or 0

 with torch.no_grad():
 suf_logits = model(sufficient_ids).logits[0, -1]
 suf_prob = torch.softmax(suf_logits, dim=-1)[predicted_id].item()
 sufficiency = suf_prob / baseline_prob # closer to 1 = better

 # Comprehensiveness: remove top-k tokens
 comp_ids = inputs["input_ids"].clone()
 for idx in top_k:
 comp_ids[0, idx] = tokenizer.pad_token_id or 0

 with torch.no_grad():
 comp_logits = model(comp_ids).logits[0, -1]
 comp_prob = torch.softmax(comp_logits, dim=-1)[predicted_id].item()
 comprehensiveness = 1 - (comp_prob / baseline_prob) # closer to 1 = better

 results[f"k={k}"] = {
 "sufficiency": sufficiency,
 "comprehensiveness": comprehensiveness,
 }

 return results

9. LLMs as Interpretability Assistants

A surprising twist in the interpretability story is that LLMs themselves have become powerful tools for explaining other models. Instead of relying solely on numerical attribution scores or heatmaps, practitioners are using language models to generate natural language explanations of model behavior, produce counterfactual analyses, and even automate the labeling of internal model features discovered through mechanistic interpretability.

7.1 Natural Language Explanations of Predictions

Given a model's input, output, and attribution scores, an LLM can synthesize a human-readable explanation: "The model predicted negative sentiment primarily because of the phrase 'deeply disappointed,' which received the highest attribution score. The word 'excellent' in the second sentence pulled toward positive sentiment but was outweighed by the negative signals." This transforms opaque numerical outputs into narratives that stakeholders can understand and critique. The key advantage is accessibility: a product manager does not need to interpret a SHAP waterfall chart if an LLM can narrate the same information in plain language.

7.2 LLM-Generated Counterfactual Explanations

Counterfactual explanations answer the question "what would need to change for the prediction to be different?" An LLM can generate these by prompting it with the original input and prediction, then asking it to produce minimal modifications that would flip the outcome. For example: "The loan application was denied because the debt-to-income ratio of 45% exceeds the threshold. The prediction would change to approved if the monthly debt payments decreased from $2,700 to below $2,000, or if annual income increased from $72,000 to above $85,000." These explanations are actionable in ways that feature importance scores are not, and they satisfy regulatory requirements in domains like finance and healthcare where model decisions must be explainable.

7.3 Automated Model Card Generation

Model cards (Mitchell et al., 2019) document a model's intended use, performance characteristics, limitations, and ethical considerations. Writing them manually is tedious and often skipped. LLMs can automate this by analyzing a model's evaluation results, training data statistics, and configuration, then generating a structured model card that covers performance breakdowns by demographic group, known failure modes, and recommended use cases. While the generated card requires human review, it reduces the documentation burden from hours to minutes and ensures that no standard section is accidentally omitted.

7.4 Auto-Labeling SAE Features with LLMs

The sparse autoencoders (SAEs) discussed in Section 18.3 decompose model activations into thousands of interpretable features, but each feature needs a human-readable label to be useful. OpenAI's "Language models can explain neurons in language models" (Bills et al., 2023) pioneered the approach of using GPT-4 to automatically describe what each neuron computes by showing it the neuron's top-activating examples and asking for a natural language summary. Neuronpedia scales this approach to the features discovered by SAEs, using LLMs to auto-label features from Gemma Scope (covered earlier in Section 18.3) and other SAE analyses. The process works as follows: collect the top 20 text examples that maximally activate a given SAE feature, present them to an LLM with the prompt "What concept or pattern do these examples share?", and store the generated label alongside the feature. Human spot-checks verify label quality, and the community can propose corrections.

7.5 Practical Workflow

A typical LLM-assisted interpretability workflow combines the techniques above. First, run standard attribution methods (Integrated Gradients, attention rollout) on a set of important predictions. Second, feed the attributions into an LLM to generate natural language explanations and counterfactuals. Third, use SAE feature analysis with LLM auto-labeling to identify higher-level circuits involved in the prediction. Fourth, compile the results into an auto-generated model card. This end-to-end pipeline makes interpretability accessible to teams that lack dedicated interpretability researchers, democratizing a practice that was previously confined to specialized labs.

Exercises