Practical Interpretability for Applications

"Interpretability that cannot leave the lab is just curiosity. Interpretability that ships to production is engineering."

Probe, Production Shipping AI Agent

1. Feature Attribution Methods

Feature attribution methods assign an importance score to each input token, answering the question: "How much did each token contribute to the model's prediction?" Unlike attention visualization (which shows where the model looks), attribution methods track the actual causal influence of each input on the output through the entire network.

1.1 Integrated Gradients

Integrated Gradients (Sundararajan et al., 2017) computes attribution by integrating the gradient of the output with respect to the input along a straight-line path from a baseline (typically the zero embedding) to the actual input. The method satisfies two desirable axioms: sensitivity (if changing an input changes the output, that input gets non-zero attribution) and implementation invariance (the attribution depends only on the function, not its implementation details).

The following implementation computes Integrated Gradients attribution by interpolating between a zero-embedding baseline and the actual input, averaging gradients along this path.

# Integrated Gradients for Token Attribution
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

def integrated_gradients(
 model,
 tokenizer,
 text,
 target_token_idx=-1,
 n_steps=50,
 internal_batch_size=5,
):
 """
 Compute Integrated Gradients attribution for each input token.

 Args:
 model: language model
 tokenizer: tokenizer
 text: input text
 target_token_idx: which output position to explain (-1 = last)
 n_steps: number of interpolation steps (higher = more accurate)

 Returns:
 attributions: per-token attribution scores
 tokens: list of input tokens
 """
 model.eval()
 inputs = tokenizer(text, return_tensors="pt")
 input_ids = inputs["input_ids"]
 tokens = tokenizer.convert_ids_to_tokens(input_ids[0])

 # Get embeddings
 embedding_layer = model.get_input_embeddings()
 input_embeds = embedding_layer(input_ids).detach()

 # Baseline: zero embeddings
 baseline = torch.zeros_like(input_embeds)

 # Interpolate between baseline and input
 alphas = torch.linspace(0, 1, n_steps).unsqueeze(1).unsqueeze(2)
 interpolated = baseline + alphas * (input_embeds - baseline)
 # Shape: (n_steps, seq_len, hidden_dim)

 # Compute gradients at each interpolation point
 all_grads = []
 for i in range(0, n_steps, internal_batch_size):
 batch = interpolated[i:i + internal_batch_size]
 batch.requires_grad_(True)

 outputs = model(inputs_embeds=batch)
 logits = outputs.logits

 # Get the predicted token's logit at target position
 target_logit = logits[:, target_token_idx, :].max(dim=-1).values
 target_logit.sum().backward()

 all_grads.append(batch.grad.detach())

 grads = torch.cat(all_grads, dim=0) # (n_steps, seq_len, hidden)

 # Integrate: average gradients, then multiply by (input - baseline)
 avg_grads = grads.mean(dim=0) # (seq_len, hidden)
 ig = (input_embeds.squeeze(0) - baseline.squeeze(0)) * avg_grads

 # Sum over hidden dimension to get per-token scores
 attributions = ig.sum(dim=-1).numpy()

 return attributions, tokens

# Example usage
model = AutoModelForCausalLM.from_pretrained("gpt2")
tokenizer = AutoTokenizer.from_pretrained("gpt2")

attrs, tokens = integrated_gradients(
 model, tokenizer, "The capital of France is"
)
print("Token attributions for next-token prediction:")
for token, score in zip(tokens, attrs):
 bar = "+" * int(abs(score) * 20)
 direction = "+" if score > 0 else "-"
 print(f" {token:15s} {direction}{bar} ({score:.4f})")

# Production equivalent using Captum
from captum.attr import LayerIntegratedGradients

lig = LayerIntegratedGradients(model, model.get_input_embeddings())
attributions, delta = lig.attribute(
 inputs=input_ids, baselines=baseline_ids,
 target=target_idx, n_steps=50, return_convergence_delta=True,
)

# Model Editing with ROME (using the rome library)
# pip install rome
from rome import ROMEHyperParams, apply_rome_to_model
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("gpt2-xl")
tokenizer = AutoTokenizer.from_pretrained("gpt2-xl")

# Before editing
prompt = "The president of the United States is"
inputs = tokenizer(prompt, return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=10)
print(f"Before: {tokenizer.decode(output[0])}")

# Define the edit
edit_request = {
 "prompt": "The president of the United States is",
 "subject": "The president of the United States",
 "target_new": " Elon Musk", # hypothetical edit
}

# Apply ROME
hparams = ROMEHyperParams.from_name("gpt2-xl")
edited_model, _ = apply_rome_to_model(
 model, tokenizer, [edit_request], hparams
)

# After editing
output = edited_model.generate(**inputs, max_new_tokens=10)
print(f"After: {tokenizer.decode(output[0])}")

# Verify specificity: other knowledge should be preserved
other_prompt = "The capital of France is"
other_inputs = tokenizer(other_prompt, return_tensors="pt")
other_output = edited_model.generate(**other_inputs, max_new_tokens=10)
print(f"Other fact: {tokenizer.decode(other_output[0])}")
# Should still say "Paris"

1.2 SHAP for Language Models

SHAP (SHapley Additive exPlanations) adapts Shapley values from cooperative game theory to feature attribution. Each token's SHAP value represents the average marginal contribution of that token across all possible subsets of input tokens. While computationally expensive for long sequences, SHAP provides theoretically grounded attributions with strong guarantees. Code Fragment 18.3.2 shows this approach in practice.

Code Fragment 18.3.2 shows how to use SHAP with a language model by wrapping the model's prediction function for the SHAP explainer API.

# SHAP-based attribution for language models
import shap

def explain_with_shap(model, tokenizer, texts, max_evals=500):
 """Use SHAP to explain model predictions."""

 # Create a SHAP explainer for the model
 def model_predict(texts_list):
 """Prediction function for SHAP."""
 results = []
 for text in texts_list:
 inputs = tokenizer(text, return_tensors="pt", truncation=True)
 with torch.no_grad():
 logits = model(**inputs).logits[0, -1]
 probs = torch.softmax(logits, dim=-1)
 results.append(probs.numpy())
 return np.array(results)

 # Partition explainer works well for text
 explainer = shap.Explainer(
 model_predict,
 tokenizer,
 output_names=tokenizer.get_vocab(),
 )

 # Compute SHAP values
 shap_values = explainer(texts, max_evals=max_evals)

 return shap_values

# Visualize SHAP attributions
shap_vals = explain_with_shap(
 model, tokenizer,
 ["The movie was absolutely wonderful and I loved every moment"]
)
shap.plots.text(shap_vals[0])

2. Representation Engineering

Representation engineering (RepE) steers model behavior by modifying internal representations at inference time. Instead of retraining the model, you identify a "control vector" in activation space that corresponds to a specific behavior (such as honesty, verbosity, or formality) and add or subtract this vector during generation to increase or decrease that behavior. Figure 18.3.1 shows how control vectors are extracted and applied. Code Fragment 18.3.3 shows this approach in practice.

Figure 18.3.1: Representation engineering extracts control vectors from contrastive prompts, then applies them at inference time to steer behavior without retraining.

The following code extracts a control vector by computing the mean activation difference between contrastive prompt pairs, then applies it during generation via a forward hook.

# Representation Engineering: Control Vectors
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

def extract_control_vector(
 model,
 tokenizer,
 positive_prompts,
 negative_prompts,
 layer_idx,
):
 """
 Extract a control vector by contrasting positive and negative prompts.

 The control vector is the mean activation difference at a specific layer.
 """
 def get_mean_activation(prompts, layer_idx):
 activations = []
 for prompt in prompts:
 inputs = tokenizer(prompt, return_tensors="pt")
 with torch.no_grad():
 outputs = model(**inputs, output_hidden_states=True)
 # Take the last token's hidden state at the target layer
 hidden = outputs.hidden_states[layer_idx][0, -1, :]
 activations.append(hidden)
 return torch.stack(activations).mean(dim=0)

 pos_mean = get_mean_activation(positive_prompts, layer_idx)
 neg_mean = get_mean_activation(negative_prompts, layer_idx)

 control_vector = pos_mean - neg_mean
 # Normalize to unit length
 control_vector = control_vector / control_vector.norm()

 return control_vector

# Example: honesty control vector
honest_prompts = [
 "I need to give an honest, truthful answer to this question:",
 "Let me think carefully and give an accurate response:",
 "I want to be straightforward and factual:",
]
dishonest_prompts = [
 "I should make up a convincing-sounding answer:",
 "Let me say whatever sounds good regardless of truth:",
 "I want to tell them what they want to hear:",
]

model = AutoModelForCausalLM.from_pretrained("gpt2")
tokenizer = AutoTokenizer.from_pretrained("gpt2")

honesty_vector = extract_control_vector(
 model, tokenizer, honest_prompts, dishonest_prompts, layer_idx=8
)

def generate_with_steering(
 model, tokenizer, prompt, control_vector, layer_idx,
 alpha=1.5, max_new_tokens=100,
):
 """Generate text while adding the control vector at each step."""
 def steering_hook(module, input, output):
 # Add control vector to hidden states
 hidden = output[0]
 hidden[:, -1, :] += alpha * control_vector
 return (hidden,) + output[1:]

 handle = model.transformer.h[layer_idx].register_forward_hook(steering_hook)
 inputs = tokenizer(prompt, return_tensors="pt")
 outputs = model.generate(**inputs, max_new_tokens=max_new_tokens)
 handle.remove()

 return tokenizer.decode(outputs[0], skip_special_tokens=True)

2.1 Representation Engineering for Runtime Control

While the control vector technique described above demonstrates the core idea of representation engineering, recent work has scaled this approach into a practical tool for runtime behavior control in production systems. The key advance is that control vectors can be precomputed offline, stored as lightweight artifacts (a single vector per behavior per layer), and applied at inference time with negligible latency overhead. This makes representation engineering a compelling alternative to prompt engineering for behavioral steering, because the control is applied at the representation level rather than through input text that consumes context window tokens.

The Representation Engineering framework (Zou et al., 2023) systematizes this approach. For a target behavior (honesty, helpfulness, harmlessness, verbosity, or any other measurable trait), the practitioner constructs 50 to 200 contrastive prompt pairs that differ only in the presence or absence of the target trait. The model processes both sets, and the mean activation difference at each layer yields a control direction. The strength of steering is governed by a scalar coefficient alpha: positive alpha amplifies the trait, negative alpha suppresses it, and zero leaves the model unchanged. Crucially, multiple control vectors can be composed by simple addition: applying both an "honesty" and a "conciseness" vector simultaneously produces responses that are both more truthful and more brief.

In production, representation engineering enables several runtime control patterns that are difficult to achieve through prompting alone. A content moderation system can apply a "safety" control vector with high alpha for user-facing responses while using a lower alpha for internal reasoning steps. A multilingual deployment can steer formality level per locale without maintaining separate system prompts. An A/B testing framework can vary the "creativity" coefficient across user cohorts to measure its impact on engagement, with each variant differing only in a scalar multiplier rather than requiring distinct prompt templates.

The tooling ecosystem for representation engineering is maturing. The repeng library provides a high-level API for extracting and applying control vectors to Hugging Face models. The repeng pipeline handles contrastive data generation, activation extraction, PCA-based direction finding, and hook-based inference steering in a unified workflow. Integration with serving frameworks is still in early stages, but custom forward hooks in vLLM and SGLang can apply control vectors with minimal performance impact (typically less than 2% latency increase) because the operation is a single vector addition per layer per token.

3. Model Editing: ROME and MEMIT

Model editing techniques surgically modify specific factual associations stored in model weights without affecting other knowledge. ROME (Rank-One Model Editing) targets a single feed-forward layer to update one fact. MEMIT (Mass-Editing Memory In a Transformer) extends this to edit thousands of facts simultaneously.

# Testing CoT faithfulness via truncation experiments
from openai import OpenAI

client = OpenAI()

def test_cot_faithfulness(question: str, num_truncation_points: int = 5):
 """
 Test whether CoT reasoning is causally necessary
 by progressively truncating the reasoning chain.
 """
 # Step 1: Get full CoT response
 full_response = client.chat.completions.create(
 model="gpt-4o",
 messages=[{
 "role": "user",
 "content": f"Think step by step, then give your final answer.\n\n{question}"
 }],
 temperature=0
 ).choices[0].message.content

 # Extract the reasoning and final answer
 # (Assumes "Final answer:" delimiter in response)
 parts = full_response.split("Final answer:")
 if len(parts) < 2:
 return {"error": "Could not parse CoT structure"}

 reasoning = parts[0].strip()
 original_answer = parts[1].strip()

 # Step 2: Truncate reasoning at various points
 sentences = reasoning.split(". ")
 results = []

 for fraction in [0.0, 0.25, 0.5, 0.75, 1.0]:
 n_sentences = max(1, int(len(sentences) * fraction))
 truncated = ". ".join(sentences[:n_sentences]) + "."

 # Ask model to answer given partial reasoning
 truncated_response = client.chat.completions.create(
 model="gpt-4o",
 messages=[{
 "role": "user",
 "content": (
 f"Question: {question}\n\n"
 f"Partial reasoning: {truncated}\n\n"
 f"Based on this reasoning, what is the final answer?"
 )
 }],
 temperature=0
 ).choices[0].message.content

 results.append({
 "fraction": fraction,
 "matches_original": truncated_response.strip() == original_answer,
 "truncated_answer": truncated_response.strip()[:100]
 })

 # If answer is stable even with heavy truncation,
 # the CoT may not be causally necessary
 return {
 "original_answer": original_answer,
 "faithfulness_results": results,
 "likely_faithful": not results[0]["matches_original"]
 # If 0% reasoning still gets same answer, CoT is not needed
 }

# Test on different task types
math_result = test_cot_faithfulness("What is 17 * 23 + 45 / 9?")
common_sense = test_cot_faithfulness(
 "Would a candle still burn in a room filled with pure nitrogen?"
)

3.1 Beyond Factual Edits: Behavioral Editing and Newer Methods

ROME and MEMIT focus on factual knowledge (e.g., "The CEO of X is Y"), but model editing research has expanded to behavioral editing, which modifies how a model responds rather than what facts it stores. Behavioral edits can adjust tone, safety refusals, reasoning strategies, or language style. These are harder to localize because behavior is distributed across many layers rather than concentrated in specific MLP neurons.

MEND (Model Editor Networks with Gradient Decomposition) takes a different approach from ROME entirely. Instead of identifying and modifying a specific layer, MEND trains a small hypernetwork that learns to transform a standard fine-tuning gradient into a targeted edit. Given an edit example (input, old output, desired new output), MEND decomposes the gradient using a low-rank factorization and applies a learned transformation. This makes MEND faster at edit time (a single forward pass through the hypernetwork), but it requires a training phase to learn the editing function.

3.2 Limitations and Failure Modes (2025 Perspective)

By 2025, the knowledge editing literature has documented several concerning failure modes that limit practical deployment:

• Ripple effects: Editing "The capital of Australia is Sydney" may also change answers to "What is the largest city in New South Wales?" or "Where is the Sydney Opera House?" in unpredictable ways. These cascading side effects are difficult to test exhaustively.
• Editing collapse: Sequential edits degrade model quality. After roughly 100 to 200 sequential ROME edits, models show measurable degradation on unrelated benchmarks. This "editing collapse" appears because each rank-one update slightly distorts the layer's representation space, and these distortions compound.
• Inconsistent generalization: An edit may succeed on the exact prompt used to define it ("Who is the CEO of OpenAI?") but fail on paraphrases ("Tell me who leads OpenAI") or downstream reasoning ("The CEO of OpenAI announced..."). Edits are often more brittle than they appear in initial evaluations.

Practical guidance: Model editing is best suited for correcting a small number (fewer than 50) of specific factual errors when retraining is impractical. For larger-scale knowledge updates, retrieval-augmented generation (where the knowledge base is updated externally) or periodic fine-tuning remain more reliable. For behavioral changes, RLHF and representation engineering (Section 2 above) are generally preferable to direct weight editing.

4. Concept Erasure

Concept erasure removes specific information from model representations, ensuring the model cannot use that information for any downstream task. Unlike model editing (which changes a fact to a different value), concept erasure eliminates the information entirely. Applications include removing protected attributes (gender, race) from embeddings to prevent discriminatory predictions. Code Fragment 18.3.5 shows this approach in practice.

The following code uses LEACE to erase a binary concept from hidden states, then validates the erasure by training linear probes before and after.

# Concept Erasure with LEACE
# pip install concept-erasure
from concept_erasure import LeaceFitter
import torch

def erase_concept(
 hidden_states: torch.Tensor,
 concept_labels: torch.Tensor,
) -> torch.Tensor:
 """
 Erase a binary concept from hidden states using LEACE.

 LEACE (LEAst-squares Concept Erasure) finds the linear subspace
 that encodes the concept and projects it out, guaranteeing
 that no linear classifier can recover the concept from the
 resulting representations.
 """
 fitter = LeaceFitter.fit(hidden_states, concept_labels)
 erased = fitter.transform(hidden_states)
 return erased

# Example: erase gender information from embeddings
# hidden_states: (num_samples, hidden_dim)
# gender_labels: (num_samples,) binary labels

erased_states = erase_concept(hidden_states, gender_labels)

# Verify: train a linear probe on erased representations
from sklearn.linear_model import LogisticRegression

probe_before = LogisticRegression().fit(
 hidden_states.numpy(), gender_labels.numpy()
)
probe_after = LogisticRegression().fit(
 erased_states.numpy(), gender_labels.numpy()
)

print(f"Gender accuracy before erasure: {probe_before.score(X_test, y_test):.3f}")
print(f"Gender accuracy after erasure: {probe_after.score(X_test_erased, y_test):.3f}")
# After erasure, accuracy should be ~50% (random chance)

5. The Chain-of-Thought Faithfulness Debate

When a language model generates a chain-of-thought (CoT) reasoning trace before producing an answer, a natural question arises: does the CoT faithfully reflect the model's actual internal computation, or is it a post-hoc rationalization that sounds plausible but does not match what the model actually did? This question has profound implications for AI safety, because if CoT is unfaithful, then monitoring a model's "reasoning" provides a false sense of transparency. The prompting strategies from Section 11.2 rely on CoT to improve outputs; here we examine whether those reasoning traces can also be trusted as explanations.

Why does this matter? Many safety and alignment proposals depend on the ability to inspect a model's reasoning. If the model says "I am recommending this action because of reasons X, Y, and Z" but internally computed the answer based on entirely different features, then human oversight based on CoT inspection is fundamentally compromised.

5.1 Evidence for Unfaithful CoT

Turpin et al. (2023) demonstrated that CoT reasoning in large language models is susceptible to systematic biases that the models fail to acknowledge. When presented with multiple-choice questions where one answer was suggested by its position (e.g., always option A) or by a sycophantic cue ("I think the answer is B"), models would shift their answers toward the biased option while generating CoT traces that never mentioned the bias. The model would construct seemingly logical reasoning for the biased answer without disclosing that the position or suggestion influenced its choice. This showed that CoT can be a confabulation: a plausible story generated to justify a conclusion reached for hidden reasons.

Lanham et al. (2023) further investigated CoT faithfulness through a series of intervention experiments. They truncated CoT reasoning at various points (early, middle, late) and measured whether the model's final answer changed. If CoT were fully faithful, removing the reasoning should degrade the answer. They found a mixed picture: for some tasks (arithmetic, simple logic), truncating CoT significantly harmed performance, suggesting the reasoning was genuinely used. For other tasks (commonsense QA, sentiment), truncating CoT had minimal effect, suggesting the model had already "decided" its answer before or independently of the CoT trace.

5.2 The 2025 Updates: Anthropic and METR

In early 2025, both Anthropic and METR (Model Evaluation and Threat Research) published updated findings on reasoning faithfulness in frontier models. Anthropic's study examined Claude 3.5 Sonnet's extended thinking traces and found that faithfulness varied significantly by task type. For mathematical reasoning, over 80% of the reasoning steps were causally necessary (removing them changed the answer). For ethical reasoning and safety-relevant decisions, the faithfulness rate dropped below 50%, meaning the model often generated plausible-sounding ethical arguments that did not reflect its actual decision process.

METR's evaluation focused on agentic settings where models plan multi-step actions. They found that models performing well on benchmarks sometimes produced CoT traces that described a strategy different from the one they actually executed. In some cases, the model's stated plan was conservative and safe, while its actual actions took shortcuts that the CoT never mentioned.

6. Interpretability for Debugging

Beyond research, interpretability tools serve as practical debugging instruments for model evaluation and observability. When a model produces incorrect or unexpected outputs, these tools help diagnose the root cause by identifying which components contributed to the error and what information the model relied on. Figure 18.3.2 outlines a systematic debugging workflow using these tools.

Figure 18.3.2: A debugging workflow using interpretability tools. Attribution identifies the contributing tokens, activation patching localizes responsible components, and editing or steering fixes the issue.

# pip install nnsight
from nnsight import LanguageModel

model = LanguageModel("meta-llama/Llama-3.1-8B-Instruct")

# Read hidden states at a specific layer during a forward pass
with model.trace("The capital of France is") as tracer:
 hidden = model.model.layers[16].output[0].save()

print(f"Layer 16 activations: {hidden.value.shape}")

# Intervene: zero out an attention head to test its causal role
with model.trace("The capital of France is") as tracer:
 model.model.layers[16].self_attn.o_proj.output[:, :, :128] = 0
 logits = model.lm_head.output.save()

Exercises