Mechanistic Interpretability at Scale

"Understanding a neural network one neuron at a time is like understanding a city one brick at a time. You need to find the roads."

Frontier, Road Mapping AI Agent

1. The Superposition Hypothesis

Imagine trying to understand a city by interviewing one resident at a time. Each person plays multiple roles (parent, employee, neighbor, volunteer), and each role is shared across many people. You cannot understand the city's "transportation system" by finding the one person who is "the transportation neuron." Mechanistic interpretability faces exactly this challenge. The central challenge is superposition: neural networks represent far more features than they have dimensions. A model with a 4096-dimensional residual stream does not encode 4096 features; it may encode tens of thousands or millions of features, each represented as a direction in the high-dimensional space, with most features overlapping (sharing dimensions with other features).

This is possible because of the geometry of high-dimensional spaces. In 4096 dimensions, you can find an exponential number of approximately orthogonal directions. If the model only needs to represent a feature with moderate precision, it can pack many more features into the space than the number of dimensions, at the cost of interference between features. This is analogous to compressed sensing in signal processing.

Elhage et al. (2022) at Anthropic formalized the superposition hypothesis and demonstrated it in toy models. They showed that a small neural network trained on a sparse input distribution will learn to superpose many features into fewer dimensions, with the degree of superposition depending on the sparsity of feature activations. Features that are rarely active can be more heavily superposed because they are unlikely to interfere with each other simultaneously.

Polysemanticity

A direct consequence of superposition is polysemanticity: individual neurons respond to multiple, seemingly unrelated concepts. A single neuron might activate for both "academic citations" and "legal precedents" and "Bible verses" because all three features share a similar direction in activation space, and the neuron's activation axis happens to align with that shared direction.

Polysemanticity makes it nearly impossible to interpret neural networks one neuron at a time. The "grandmother cell" model of neural representation (one neuron per concept) is the exception, not the rule. Most neurons participate in encoding many features, and most features are distributed across many neurons. Understanding what a model computes requires identifying features as directions in activation space, not as individual neurons.

2. Sparse Autoencoders for Feature Discovery

Sparse Autoencoders (SAEs) have become the primary tool for extracting interpretable features from neural network activations. The core idea is simple: train an autoencoder with a much larger hidden dimension than the input, with a sparsity penalty that encourages most hidden units to be zero for any given input. The non-zero hidden units then correspond to the active features for that input.

Formally, given an activation vector $\mathbf{x} \in \mathbb{R}^d$ from a transformer layer, the SAE computes:

where $m \gg d$ (the hidden dimension is much larger than the input dimension, often 8x to 64x) and the training loss combines reconstruction accuracy with a sparsity penalty:

The L1 penalty on $\mathbf{f}$ encourages sparse activations: for any given input, only a small fraction of the $m$ features are active. Each active feature corresponds to a specific direction in the original activation space (the corresponding column of $W_{\text{dec}}$), and the magnitude of the feature activation indicates how strongly that feature is present in the input.

The following code demonstrates training a basic SAE on transformer activations.

# Train a sparse autoencoder (SAE) on transformer hidden states to discover
# interpretable features. The encoder projects to a high-dimensional space
# with ReLU sparsity; the decoder reconstructs the original activation.
import torch
import torch.nn as nn

class SparseAutoencoder(nn.Module):
 """Sparse autoencoder for extracting interpretable features
 from transformer hidden states.

 Args:
 d_model: dimension of the transformer activations
 n_features: number of SAE features (typically 8x to 64x d_model)
 sparsity_coeff: weight of the L1 sparsity penalty
 """
 def __init__(self, d_model: int, n_features: int, sparsity_coeff: float = 1e-3):
 super().__init__()
 self.encoder = nn.Linear(d_model, n_features)
 self.decoder = nn.Linear(n_features, d_model, bias=True)
 self.sparsity_coeff = sparsity_coeff

 # Initialize decoder weights to unit norm (each column is a feature direction)
 with torch.no_grad():
 self.decoder.weight.data = nn.functional.normalize(
 self.decoder.weight.data, dim=0
 )

 def forward(self, x: torch.Tensor) -> dict:
 # Encode: project to high-dimensional sparse space
 features = torch.relu(self.encoder(x))

 # Decode: reconstruct the original activation
 reconstructed = self.decoder(features)

 # Losses
 reconstruction_loss = (x - reconstructed).pow(2).mean()
 sparsity_loss = features.abs().mean()
 total_loss = reconstruction_loss + self.sparsity_coeff * sparsity_loss

 return {
 "features": features,
 "reconstructed": reconstructed,
 "loss": total_loss,
 "reconstruction_loss": reconstruction_loss.item(),
 "sparsity_loss": sparsity_loss.item(),
 "n_active": (features > 0).float().sum(dim=-1).mean().item(),
 }

# Example: train an SAE on synthetic "activations"
d_model = 768
n_features = 768 * 16 # 16x expansion
sae = SparseAutoencoder(d_model, n_features, sparsity_coeff=5e-4)
optimizer = torch.optim.Adam(sae.parameters(), lr=3e-4)

# Simulate training on batches of transformer activations
for step in range(1000):
 # In practice, these come from running text through a transformer
 fake_activations = torch.randn(64, d_model)

 result = sae(fake_activations)
 result["loss"].backward()
 optimizer.step()
 optimizer.zero_grad()

 if (step + 1) % 200 == 0:
 print(
 f"Step {step+1}: "
 f"recon={result['reconstruction_loss']:.4f}, "
 f"sparsity={result['sparsity_loss']:.4f}, "
 f"active_features={result['n_active']:.1f}/{n_features}"
 )

# pip install transformer_lens
import transformer_lens as tl

model = tl.HookedTransformer.from_pretrained("gpt2-small")
logits, cache = model.run_with_cache("The Eiffel Tower is in")
# Inspect residual stream at layer 8, final token position
resid = cache["resid_post", 8][0, -1] # shape: (768,)
print(f"Residual norm: {resid.norm():.2f}")

Scaling SAEs to Frontier Models

In 2024, both Anthropic and OpenAI published results from training SAEs on their frontier models. Anthropic's work on Claude 3 Sonnet (Templeton et al., 2024) extracted millions of interpretable features, including features for specific concepts (Golden Gate Bridge, computer code, deceptive reasoning), behaviors (refusing harmful requests, being sycophantic), and languages (features that activate specifically for French, Mandarin, or Python code).

The scaling challenge is substantial. Training a SAE on a frontier model requires running billions of tokens through the model, storing the activations, and training the SAE on this dataset. The SAE itself can have billions of parameters (for a 16x expansion of a model with a 16384-dimensional residual stream, the SAE has over 4 billion parameters). The compute cost is a significant fraction of the cost of training the original model.

Despite the cost, the features discovered by large-scale SAEs are remarkably interpretable. Anthropic's team found that they could steer Claude's behavior by amplifying or suppressing specific features. Amplifying the "Golden Gate Bridge" feature caused the model to mention the bridge in every response. Suppressing "deceptive reasoning" features reduced the model's tendency to produce misleading outputs. This level of granular control was not previously possible.

3. Circuit Analysis

While SAEs identify the features (the "what"), circuit analysis identifies the computational pathways (the "how"). A circuit is a subgraph of the model's computational graph that implements a specific behavior: for example, the circuit responsible for indirect object identification in sentences like "When Mary and John went to the store, John gave a drink to ___."

The Circuit Discovery Pipeline

Identifying circuits in transformer models typically follows a multi-step process:

1. Task specification. Define a narrow behavior to investigate (e.g., "the model correctly predicts the indirect object in ditransitive sentences").
2. Activation patching. Run the model on a clean input and a corrupted input, then systematically replace activations in the corrupted run with activations from the clean run. If replacing the output of a specific attention head restores correct behavior, that head is part of the circuit.
3. Path patching. Refine the analysis by patching individual connections between components (e.g., the output of attention head 5 in layer 3 as it flows into MLP layer 4). This identifies the specific information pathways, not just the components.
4. Interpretation. For each component in the circuit, analyze what it computes using techniques like attention pattern visualization, probing classifiers, and SAE feature analysis.

Wang et al. (2023) applied this methodology to identify the "Indirect Object Identification (IOI) circuit" in GPT-2 Small, finding a circuit of 26 attention heads across multiple layers that collaborate to solve this task. The circuit includes "duplicate token heads" (detecting repeated names), "S-inhibition heads" (suppressing the subject), and "name mover heads" (copying the correct name to the output).

# pip install nnsight
from nnsight import LanguageModel

model = LanguageModel("openai-community/gpt2", device_map="auto")
with model.trace("The Eiffel Tower is in") as tracer:
 # Access hidden states at layer 6
 hidden = model.transformer.h[6].output[0].save()
print(f"Layer 6 output shape: {hidden.value.shape}")
# torch.Size([1, 6, 768])

Limitations of Circuit Analysis

Circuit analysis has produced impressive results on narrow tasks, but scaling it to the full breadth of a frontier model's capabilities faces fundamental challenges. First, most model behaviors involve diffuse computation distributed across many components, not clean, modular circuits. Second, the number of possible circuits is combinatorially large, making exhaustive search intractable. Third, circuits may overlap and interact, so understanding one circuit in isolation may not predict its behavior when other circuits are simultaneously active.

4. Practical Applications

While the theoretical ambitions of mechanistic interpretability are grand, several practical applications are already emerging.

Model Debugging

When a model produces incorrect or unexpected outputs, SAE features can help diagnose why. By examining which features are active during the problematic generation, developers can identify whether the model is attending to the wrong part of the input, activating an irrelevant feature, or suppressing a relevant one. This is analogous to using a debugger in software engineering: rather than guessing what went wrong, you can inspect the internal state.

Safety Auditing

Interpretability tools enable a new form of model safety auditing. Rather than relying solely on behavioral testing (which can miss failure modes not covered by the test set), auditors can examine the model's internal features for concerning patterns. Does the model have features related to deception? Manipulation? Weapon design? The presence of such features does not necessarily indicate a safety risk (the model may have learned them from training data without using them in problematic ways), but their absence provides evidence that specific risks are lower.

Targeted Model Editing

Once a problematic feature or circuit has been identified, it can potentially be modified without retraining the entire model. Techniques like activation steering (adding or subtracting feature directions from the residual stream during inference) and targeted fine-tuning (training only the components of a specific circuit) allow for surgical modifications. This is more efficient and predictable than full fine-tuning, which may have unintended side effects on other capabilities.

Understanding Training Dynamics

SAE features can be tracked across training checkpoints to understand how the model's internal representations evolve during training. This reveals phenomena like feature splitting (a single feature early in training splits into more specific features later), phase transitions (certain features appear suddenly at specific training steps), and feature death (features that were active early in training become permanently inactive). These observations inform training methodology and may help predict and prevent training instabilities.

5. The Scaling Challenge

The fundamental challenge facing mechanistic interpretability is scaling. Current techniques work well on models with hundreds of millions of parameters. Applying them to models with hundreds of billions of parameters requires innovations in both methodology and compute.

Key scaling challenges include:

• Feature count. Larger models have more features. Anthropic's Claude 3 Sonnet SAEs found millions of features; frontier models likely have tens of millions. Manually inspecting each feature is impossible, requiring automated interpretability methods (using LLMs to describe features, for instance).
• Circuit complexity. As models grow, circuits become more complex and involve more components. The 26-head IOI circuit in GPT-2 Small may correspond to a 200-head circuit in a frontier model, with additional complexity from increased depth and width.
• Validation. How do you verify that a proposed circuit is complete and correct? In small models, ablation studies (removing the circuit and checking that the behavior disappears) are feasible. In large models, the computational cost of comprehensive ablation is prohibitive.
• Automated interpretability. Using language models to automatically label and describe SAE features (Bills et al., 2023) is promising but introduces a meta-level interpretability challenge: how do you verify that the LLM's description of a feature is accurate?

6. Connections to Other Chapters

Mechanistic interpretability intersects with several other topics covered in this book:

• AI safety (Section 32.1). Interpretability is one of the primary tools for building safety cases for AI systems. If you can understand what a model computes and why, you can make stronger claims about its safety.
• Fine-tuning (Chapter 19). Understanding what features change during fine-tuning can explain why fine-tuning sometimes causes capability regressions or unexpected behavior changes.
• Evaluation (Section 29.1). Interpretability provides an internal complement to external evaluation. Behavioral tests tell you what the model does; interpretability tells you how and why.
• Reasoning (Section 34.5). Circuit analysis of chain-of-thought reasoning may reveal whether CoT chains are causally involved in the model's computation or are post-hoc rationalizations.