Mechanistic Interpretability

"You cannot claim to understand a system until you can take it apart and put it back together."

Probe, Reverse Engineering AI Agent

This section covers five interconnected ideas. We start with the residual stream view, a conceptual reframing that makes transformer internals analyzable (building on the transformer architecture from Section 04.1). We then tackle superposition, the core challenge that makes neurons hard to interpret. Next, sparse autoencoders provide the solution by extracting clean features from tangled (see the embedding space concepts in Section 19.1 for how similar representations are used in retrieval) representations. Activation patching then lets us test which components causally drive specific behaviors. Finally, we survey the tooling (TransformerLens, nnsight) that makes this research practical.

1. The Residual Stream View

Mechanistic interpretability starts with a conceptual reframing of the transformer architecture. Instead of viewing transformers as a sequence of layers that process data in order, the "residual stream" view treats the residual connection as a central communication channel. Each attention head and MLP layer reads from this stream, performs a computation, and writes its result back to the stream by adding it to the running total. Figure 18.2.1 visualizes this shared communication channel.

Figure 18.2.3: The residual stream view of a transformer. Attention heads and MLP layers read from and write to a shared stream, accumulating their contributions through addition.

Lesson: Activation patching localizes model behaviors to specific components; targeted interventions on identified circuits are far more precise than global debiasing approaches that risk collateral damage.

Circuit-level analysis assumes we can identify what each neuron or component represents. But a deeper challenge emerges when we try: individual neurons often respond to multiple, seemingly unrelated concepts. This is not a bug in our analysis methods; it is a fundamental property of how neural networks encode information.

2. Superposition and Polysemanticity

A central challenge in interpreting neural networks is that individual neurons are often polysemantic: a single neuron may activate for multiple unrelated concepts. For example, a neuron might fire for both "legal terminology" and "the color blue." This happens because of superposition: the model encodes more features than it has neurons by representing features as directions in activation space (similar to how embedding spaces encode meaning as directions) rather than individual neurons.

Superposition is an efficient compression strategy. If a model has 4096 neurons per layer but needs to track 100,000 features, it can represent each feature as a direction in the 4096-dimensional space. As long as features rarely co-occur, they can share neurons with minimal interference. This means the "true" features of the model are not individual neurons but rather directions in activation space, which are linear combinations of neurons. Figure 18.2.2 contrasts the clean (no superposition) case with the realistic (superposition) case.

Figure 18.2.4: Without superposition, each neuron represents one feature (clean but limited). With superposition, multiple features share neurons, creating polysemantic neurons that are harder to interpret.

3. Sparse Autoencoders (SAEs)

Sparse autoencoders are the primary tool for disentangling superposed representations. An SAE takes the model's activations as input, encodes them into a much higher-dimensional but sparse representation, and then decodes back to the original activation space. The key constraint is sparsity: only a small fraction of the SAE's latent dimensions are active for any given input. Each active dimension corresponds to a single interpretable feature. Code Fragment 18.2.1 shows this approach in practice.

The following implementation defines a sparse autoencoder that projects transformer activations into a higher-dimensional sparse space, where each active dimension corresponds to an interpretable feature.

# Sparse Autoencoder for Mechanistic Interpretability
import torch
import torch.nn as nn
import torch.nn.functional as F

class SparseAutoencoder(nn.Module):
 """
 Sparse Autoencoder for extracting interpretable features
 from transformer activations.

 Architecture:
 - Encoder: project d_model -> d_sae (expansion)
 - ReLU sparsity
 - Decoder: project d_sae -> d_model (reconstruction)

 The expansion factor (d_sae / d_model) is typically 4x to 64x.
 """

 def __init__(self, d_model, d_sae, l1_coeff=5e-3):
 super().__init__()
 self.d_model = d_model
 self.d_sae = d_sae
 self.l1_coeff = l1_coeff

 # Encoder and decoder
 self.encoder = nn.Linear(d_model, d_sae, bias=True)
 self.decoder = nn.Linear(d_sae, d_model, bias=True)

 # Initialize decoder weights to unit norm
 with torch.no_grad():
 self.decoder.weight.data = F.normalize(
 self.decoder.weight.data, dim=0
 )

 def encode(self, x):
 """Encode activations into sparse feature space."""
 # Subtract decoder bias for centering
 x_centered = x - self.decoder.bias
 # Encode and apply ReLU for sparsity
 z = F.relu(self.encoder(x_centered))
 return z

 def decode(self, z):
 """Reconstruct activations from sparse features."""
 return self.decoder(z)

 def forward(self, x):
 z = self.encode(x) # sparse features
 x_hat = self.decode(z) # reconstruction
 return x_hat, z

 def loss(self, x):
 """Combined reconstruction + sparsity loss."""
 x_hat, z = self.forward(x)

 # Reconstruction loss (MSE)
 recon_loss = F.mse_loss(x_hat, x)

 # Sparsity loss (L1 on activations)
 l1_loss = z.abs().mean()

 return recon_loss + self.l1_coeff * l1_loss, {
 "reconstruction": recon_loss.item(),
 "sparsity": l1_loss.item(),
 "alive_features": (z > 0).any(dim=0).sum().item(),
 "avg_active": (z > 0).float().mean().item(),
 }

# Training an SAE on GPT-2 MLP activations
def train_sae_on_model(
 model_name="gpt2",
 layer_idx=6,
 component="mlp",
 expansion_factor=8,
 num_tokens=10_000_000,
 batch_size=4096,
 lr=3e-4,
):
 """Train a sparse autoencoder on transformer activations."""
 from transformers import AutoModelForCausalLM, AutoTokenizer

 model = AutoModelForCausalLM.from_pretrained(model_name)
 tokenizer = AutoTokenizer.from_pretrained(model_name)
 d_model = model.config.n_embd
 d_sae = d_model * expansion_factor

 sae = SparseAutoencoder(d_model, d_sae)
 optimizer = torch.optim.Adam(sae.parameters(), lr=lr)

 # Collect activations using hooks
 activations_buffer = []

 def hook_fn(module, input, output):
 activations_buffer.append(output.detach())

 # Register hook on the target layer
 if component == "mlp":
 handle = model.transformer.h[layer_idx].mlp.register_forward_hook(hook_fn)
 else:
 handle = model.transformer.h[layer_idx].attn.register_forward_hook(hook_fn)

 # Training loop (simplified)
 for step in range(num_tokens // batch_size):
 # Get a batch of activations
 batch_acts = get_activation_batch(
 model, tokenizer, batch_size
 )

 loss, metrics = sae.loss(batch_acts)
 optimizer.zero_grad()
 loss.backward()
 optimizer.step()

 # Normalize decoder weights after each step
 with torch.no_grad():
 sae.decoder.weight.data = F.normalize(
 sae.decoder.weight.data, dim=0
 )

 if step % 100 == 0:
 print(
 f"Step {step}: recon={metrics['reconstruction']:.4f}, "
 f"l1={metrics['sparsity']:.4f}, "
 f"alive={metrics['alive_features']}/{d_sae}"
 )

 handle.remove()
 return sae

# Using TransformerLens to trace a simple circuit
# This demonstrates the conceptual approach behind attribution graphs
import transformer_lens
from transformer_lens import HookedTransformer, ActivationCache
import torch

model = HookedTransformer.from_pretrained("gpt2-small")

prompt = "The Eiffel Tower is in"
target = " Paris"
target_token = model.to_single_token(target)

# Run the model and cache all activations
logits, cache = model.run_with_cache(prompt)

# Trace attribution: which components at each layer
# contribute most to the target token's logit?
per_head_direct_effect = cache.stack_head_results(
 layer=-1, pos_slice=-1
)
# Project each head's output through the unembedding
per_head_logit_attr = (
 per_head_direct_effect @ model.W_U[:, target_token]
)

print("Direct logit attribution by attention head:")
for layer in range(model.cfg.n_layers):
 for head in range(model.cfg.n_heads):
 attr = per_head_logit_attr[layer, 0, head].item()
 if abs(attr) > 0.5: # show only significant heads
 print(f" L{layer}H{head}: {attr:+.3f}")

# The heads with largest positive attribution form the
# "factual recall circuit" for this specific prompt.
# Attribution graphs extend this to trace through SAE
# features at every layer, producing a full computational DAG.

# Production equivalent using SAELens
from sae_lens import SAETrainingRunner, LanguageModelSAERunnerConfig

cfg = LanguageModelSAERunnerConfig(
 model_name="gpt2", hook_point="blocks.6.mlp.hook_post",
 d_sae=768 * 8, l1_coefficient=5e-3,
)
sae = SAETrainingRunner(cfg).run()

4. Activation Patching

Activation patching (also called causal tracing or interchange intervention) is a technique for identifying which components of a model are responsible for a specific behavior. The idea is to run the model on two inputs (a "clean" input that triggers the behavior and a "corrupted" input that does not), and then selectively replace activations from one run with activations from the other. If replacing a specific component's activation restores the original behavior, that component is causally important. Code Fragment 18.2.2 shows this approach in practice.

The following code uses TransformerLens to perform activation patching, identifying which attention heads are causally responsible for a specific factual prediction.

# Activation Patching with TransformerLens
import transformer_lens
from transformer_lens import HookedTransformer, utils
import torch

# Load model with TransformerLens
model = HookedTransformer.from_pretrained("gpt2-small")

# Example: Which components know that "The Eiffel Tower is in" -> "Paris"?
clean_prompt = "The Eiffel Tower is in"
corrupted_prompt = "The Colosseum is in" # different answer: "Rome"

# Get clean logits for the target token
clean_logits, clean_cache = model.run_with_cache(clean_prompt)
target_token = model.to_single_token(" Paris")
clean_logit_diff = (
 clean_logits[0, -1, target_token]
 - clean_logits[0, -1, model.to_single_token(" Rome")]
).item()

# Get corrupted cache
_, corrupted_cache = model.run_with_cache(corrupted_prompt)

# Patch each attention head and measure the effect
results = torch.zeros(model.cfg.n_layers, model.cfg.n_heads)

for layer in range(model.cfg.n_layers):
 for head in range(model.cfg.n_heads):
 # Define hook that patches this head's output
 def patch_hook(activation, hook, layer=layer, head=head):
 # Replace corrupted activation with clean activation
 activation[:, :, head, :] = clean_cache[
 hook.name
 ][:, :, head, :]
 return activation

 # Run corrupted input with this one head patched
 hook_name = f"blocks.{layer}.attn.hook_z"
 patched_logits = model.run_with_hooks(
 corrupted_prompt,
 fwd_hooks=[(hook_name, patch_hook)],
 )

 # Measure recovery of clean behavior
 patched_diff = (
 patched_logits[0, -1, target_token]
 - patched_logits[0, -1, model.to_single_token(" Rome")]
 ).item()

 results[layer, head] = patched_diff

print("Top 5 most important heads for 'Eiffel Tower -> Paris':")
flat = results.flatten()
top_indices = flat.argsort(descending=True)[:5]
for idx in top_indices:
 layer = idx // model.cfg.n_heads
 head = idx % model.cfg.n_heads
 print(f" L{layer}H{head}: logit diff = {flat[idx]:.3f}")

5. TransformerLens and nnsight

Two primary libraries support mechanistic interpretability research on transformers. TransformerLens provides a re-implementation of common models with full hook access at every computational step. nnsight offers a different approach, allowing intervention on any PyTorch model without re-implementation.

Code Fragment 18.2.3 demonstrates this approach.

Code Fragment 18.2.3 demonstrates how nnsight wraps any PyTorch model to enable inspection and modification of internal activations during forward passes.

# nnsight: Intervening on any PyTorch model
from nnsight import LanguageModel

# Wrap any HuggingFace model
model = LanguageModel("gpt2", device_map="auto")

# Use the tracing context to inspect and modify activations
with model.trace("The cat sat on the") as tracer:
 # Access any module's output
 layer_5_output = model.transformer.h[5].output[0]

 # Save it for inspection
 layer_5_output.save()

 # You can also modify activations in-place
 # model.transformer.h[5].mlp.output[:] *= 0 # ablate MLP

# Access saved activations after the trace
print(f"Layer 5 output shape: {layer_5_output.value.shape}")

# Activation patching with nnsight
clean_text = "The Eiffel Tower is in"
corrupt_text = "The Colosseum is in"

# Get clean activations
with model.trace(clean_text) as tracer:
 clean_resid = model.transformer.h[8].output[0].save()

# Patch corrupted run with clean activations at layer 8
with model.trace(corrupt_text) as tracer:
 model.transformer.h[8].output[0][:] = clean_resid.value
 patched_logits = model.lm_head.output.save()

print(f"Patched prediction: {patched_logits.value[0, -1].argmax()}")

6. Scaled SAE Results and Automated Interpretability

Between 2024 and 2026, both Anthropic and OpenAI published landmark results demonstrating that sparse autoencoders can scale to production-sized language models. These results transformed mechanistic interpretability from a proof-of-concept on small models into a practical tool for understanding frontier systems.

6.1 Anthropic's Scaled SAE Research

Anthropic's "Scaling Monosemanticity" work (2024) trained SAEs on Claude 3 Sonnet, extracting roughly 34 million features. The results revealed features at multiple levels of abstraction: low-level features for syntactic patterns (punctuation, code formatting), mid-level features for semantic concepts (specific cities, programming languages, scientific terms), and high-level features for abstract behaviors (deception, sycophancy, safety refusal). Notably, the researchers demonstrated that individual features could be amplified or suppressed at inference time to steer model behavior. The famous "Golden Gate Bridge" feature, when amplified, caused the model to reference the bridge in nearly every response, providing a vivid demonstration of feature-level control.

Anthropic's subsequent work on Claude 3.5 Sonnet extended these findings further, showing that SAE features could identify safety-relevant circuits in the model, connecting directly to the alignment robustness questions raised in Section 17.3. Features corresponding to deceptive reasoning, harmful content generation, and instruction-following conflicts were isolated and studied. This work has direct implications for AI safety: if you can identify the features responsible for undesirable behavior, you can potentially intervene on them without retraining the entire model.

6.2 OpenAI's Approach

OpenAI's interpretability team published results on GPT-4 scale SAEs, focusing on automated feature interpretation. Their approach uses a separate LLM to generate natural language descriptions of what each SAE feature responds to, then validates those descriptions by testing whether they predict the feature's activation on held-out data. This "automated interpretability" pipeline can process millions of features without human review, though the quality of automated descriptions varies. For well-defined features (proper nouns, code constructs), automated descriptions match human judgments closely. For abstract or compositional features, automated descriptions often miss nuances that human analysis would catch.

6.3 Automated Interpretability Pipelines

As SAEs extract millions of features from large models, manual interpretation becomes infeasible. Automated interpretability pipelines address this bottleneck by using LLMs to interpret their own internal representations. A typical pipeline works as follows:

1. Feature activation collection: For each SAE feature, gather the top-activating examples (text passages where the feature fires most strongly) and low-activating examples.
2. Description generation: Feed the top-activating examples to an LLM and ask it to describe the common pattern. The LLM might output "This feature activates on references to European capital cities" or "This feature responds to Python exception handling."
3. Description validation: Use the generated description to predict which held-out examples should activate the feature. If the description is accurate, the prediction will correlate with actual activations. A correlation score above 0.7 indicates a reliable description.
4. Hierarchical clustering: Group features with related descriptions into semantic clusters, creating a navigable taxonomy of model representations.

Code Fragment 18.2.4 demonstrates this approach.

# Automated interpretability sketch
def auto_interpret_feature(feature_id, sae, dataset, interpreter_llm):
 """Generate and validate a description for one SAE feature."""
 # Collect top-activating examples
 activations = []
 for text in dataset:
 act = sae.encode(text)[feature_id]
 activations.append((act.item(), text))

 activations.sort(reverse=True)
 top_examples = [text for _, text in activations[:20]]
 low_examples = [text for _, text in activations[-20:]]

 # Ask interpreter LLM to describe the pattern
 prompt = f"""Below are text passages that strongly activate a specific
neuron in a language model. What concept or pattern does this
neuron detect?

Top activating examples:
{chr(10).join(f'- {ex[:200]}' for ex in top_examples[:10])}

Low activating examples:
{chr(10).join(f'- {ex[:200]}' for ex in low_examples[:5])}

Description:"""

 description = interpreter_llm.generate(prompt)

 # Validate: use description to predict activations on held-out data
 held_out = activations[20:120]
 predictions = interpreter_llm.score_match(description, held_out)
 correlation = compute_correlation(predictions, held_out)

 return {
 "feature_id": feature_id,
 "description": description,
 "validation_score": correlation,
 "reliable": correlation > 0.7,
 }

7. Circuit Tracing and Attribution Graphs

In March 2025, Anthropic published a landmark advance in mechanistic interpretability: circuit tracing through attribution graphs. This work moves beyond studying individual features or single attention heads and instead traces how information flows through entire circuits of transformer components. The result is a complete, interpretable computational graph showing how the model arrives at a specific output.

Why does this matter? Before circuit tracing, mechanistic interpretability could identify individual features ("this SAE feature responds to the Golden Gate Bridge") and test individual components via activation patching ("this attention head matters for factual recall"). But connecting these findings into a coherent story of how the model computes a specific answer required painstaking manual analysis. Attribution graphs automate this process, producing end-to-end computational narratives.

7.1 The Replacement Model Methodology

The key methodological innovation is the "replacement model." Instead of analyzing the original transformer directly, researchers construct a replacement model that approximates the original using only interpretable components (SAE features and attention heads). The replacement model replaces each layer's activations with their SAE-reconstructed versions, then traces attribution through the resulting graph. If the replacement model closely matches the original model's behavior, the attribution graph faithfully represents the original model's computation.

An attribution graph is a directed acyclic graph (DAG) where nodes represent SAE features at specific layers and edges represent the contribution of one feature to another. Each edge carries a weight indicating how much the upstream feature contributed to activating the downstream feature. By following the highest-weight paths from input features to the output logits, researchers can read off the "reasoning chain" that the model uses.

# Loading and exploring Gemma Scope features
# pip install sae-lens transformer-lens
from sae_lens import SAE
from transformer_lens import HookedTransformer
import torch

# Load model and corresponding Gemma Scope SAE
model = HookedTransformer.from_pretrained("gemma-2-2b")
sae = SAE.from_pretrained(
 release="gemma-scope-2b-pt-res",
 sae_id="layer_12/width_16k/average_l0_82",
)

# Run a prompt through the model and SAE
prompt = "The Golden Gate Bridge is a suspension bridge in"
_, cache = model.run_with_cache(prompt)

# Get residual stream activations at layer 12
residual = cache["blocks.12.hook_resid_post"]

# Encode through the SAE to get feature activations
feature_acts = sae.encode(residual)

# Find the top-activating features for the last token
last_token_features = feature_acts[0, -1]
top_features = torch.topk(last_token_features, k=10)

print("Top 10 active features at the last token position:")
for idx, (value, feature_id) in enumerate(
 zip(top_features.values, top_features.indices)
):
 print(f" Feature {feature_id.item()}: activation = {value.item():.3f}")
 # Look up feature descriptions on Neuronpedia:
 # https://www.neuronpedia.org/gemma-2-2b/{layer}/{feature_id}

8. Gemma Scope: Open SAE Suites

Reproducibility is a persistent challenge in mechanistic interpretability. Training sparse autoencoders on large models requires substantial compute, and minor differences in hyperparameters, training data, or random seeds can produce different feature sets. DeepMind addressed this challenge by releasing Gemma Scope: openly available SAE suites trained on the Gemma family of models.

Why do open SAE suites matter? Before Gemma Scope, every research group had to train their own SAEs, making it nearly impossible to compare results across papers. Two groups studying "safety-relevant features" might find completely different feature sets simply because they used different SAE architectures or training hyperparameters. Gemma Scope provides a shared reference point: when multiple teams analyze the same SAE features, their findings are directly comparable.

8.1 Gemma Scope 1 and 2

Gemma Scope 1 (released mid-2024) provided SAEs trained on Gemma 2 models (2B and 9B parameters) at multiple layers, with expansion factors of 16x and 65x. Gemma Scope 2 (2025) expanded the suite to include SAEs for Gemma 2 27B, additional training checkpoints (enabling study of how features emerge during pre-training), and SAEs trained on different components (MLP outputs, attention outputs, and the residual stream). All SAE weights and metadata are publicly available through Hugging Face.

9. SAE Scaling Laws, Alternative Libraries, and Transcoder Circuits

As the field of mechanistic interpretability matures, several developments are extending the core SAE methodology in important directions: scaling laws that predict feature quality, alternative training libraries, and architectural variants that go beyond the standard autoencoder formulation.

9.1 SAE Scaling Laws

Gao et al. (2024) established the first systematic study of how SAE quality scales with SAE size and training compute. Their key finding: SAE feature quality follows predictable scaling laws, analogous to the training scaling laws discussed in Section 6.3. Specifically, the loss recovered (the fraction of the model's cross-entropy loss explained by the SAE reconstruction) increases log-linearly with the number of SAE parameters. Doubling the SAE width consistently improves reconstruction quality by a predictable amount.

This result has practical implications for choosing SAE hyperparameters. Rather than training many SAE variants and comparing them empirically, researchers can use the scaling law to predict the quality of a larger SAE from the performance of smaller ones. The scaling law also reveals diminishing returns: going from 16x to 32x expansion yields a larger quality gain than going from 64x to 128x. For most practical applications, 32x to 65x expansion factors offer the best quality-to-compute tradeoff.

9.2 sparsify (EleutherAI)

While SAELens (used in the Gemma Scope examples above) is the most widely adopted SAE training library, EleutherAI's sparsify library provides an alternative with different design priorities. sparsify focuses on training efficiency and supports additional SAE architectures beyond the standard ReLU autoencoder, including gated SAEs, p-annealing schedules, and JumpReLU activations. It also provides built-in support for distributed training across multiple GPUs, which becomes necessary when training SAEs on the residual stream of models with hidden dimensions exceeding 4096.

# Training an SAE with EleutherAI's sparsify (conceptual)
# pip install sparsify-nnsight
from sparsify import SparseAutoencoder

# Configure the SAE architecture
sae_config = {
 "d_model": 2048, # Model hidden dimension
 "expansion_factor": 32, # SAE width = 32 * d_model
 "activation": "jumprelu", # JumpReLU for cleaner sparsity
 "l1_coefficient": 5e-3, # Sparsity penalty strength
 "dead_feature_threshold": 1e-8,
 "dead_feature_resampling": True,
}

# sparsify integrates with nnsight for activation collection
# and supports multi-GPU training for large models
sae = SparseAutoencoder(**sae_config)
print(f"SAE parameters: {sae_config['d_model'] * sae_config['expansion_factor'] * 2:,}")
print(f"Expected active features per token: ~50-100 (with JumpReLU)")

9.3 Transcoder Circuits

Standard SAEs decompose residual stream or MLP outputs into interpretable features. Transcoders (Dunefsky et al., 2024) take a more radical approach: they replace entire MLP layers with a sparse, interpretable approximation. Instead of encoding MLP outputs, a transcoder takes the MLP input (the residual stream before the MLP) and directly predicts a sparse decomposition of the MLP output, bypassing the original MLP computation entirely.

Why does this matter? Standard SAEs can tell you what features a layer represents, but the features are entangled with the MLP's nonlinear computation. Transcoders make the MLP computation itself interpretable: each transcoder feature represents a specific input-output mapping (e.g., "when the input contains a date pattern, add the 'temporal reference' direction to the residual stream"). This enables a cleaner form of circuit analysis where every component in the computational graph has a human-readable description.

9.4 Automated Feature Dashboards and Gemma Scope Integration

Neuronpedia provides web-based dashboards for browsing SAE features interactively, but for large-scale analysis, automated dashboards that summarize feature properties programmatically are essential. The typical automated dashboard for each feature includes: top-activating examples, an auto-generated natural language description, activation histograms, cosine similarity to related features, and downstream logit effects. Gemma Scope SAEs are directly integrated with these dashboards, enabling researchers to go from "feature 4,217 is interesting" to a full characterization within seconds.

9.5 nnsight for Flexible Activation Access

While TransformerLens (covered in Section 5 above) requires re-implemented model architectures, nnsight works with any PyTorch model by wrapping it with proxy objects that intercept activations during the forward pass. For SAE research, nnsight is particularly useful when working with models not yet supported by TransformerLens, or when the SAE needs to access activations at non-standard locations (e.g., between sub-layers within an MLP). sparsify uses nnsight as its activation collection backend, making the two libraries naturally complementary.

Exercises