Interpretability Tooling, Evaluation, and LLM-Assisted Explanation

"A method without a tool is a lecture; a tool without a method is a toy. Interpretability needs both."

Probe, Tool-Smith AI Agent

10.5.1 Interpretability Tools Ecosystem (2025)

The interpretability research community has built a rich ecosystem of open-source tools over the past three years. Choosing the right tool depends on your goal: are you doing mechanistic circuit analysis, exploring SAE features, or building a production explanation pipeline? This section surveys the major tools and helps you match tools to use cases.

Why does the tooling matter? Interpretability research is only as reproducible and accessible as its tooling. A brilliant mechanistic finding that requires custom infrastructure to replicate has limited impact. The tools listed below have lowered the barrier to entry, enabling researchers and practitioners to run experiments that previously required months of infrastructure work.

# Same task in three frameworks: extract activations from layer 5 of GPT-2.
# Each library trades off ease of use for control over the model internals.

# (A) Plain HuggingFace transformers + a forward hook
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
tok = AutoTokenizer.from_pretrained("gpt2")
model_hf = AutoModelForCausalLM.from_pretrained("gpt2")
acts_hf = {}
def hook(module, inputs, output):
    acts_hf["layer5"] = output[0]
model_hf.transformer.h[5].register_forward_hook(hook)
model_hf(**tok("The Eiffel Tower is in", return_tensors="pt"))
print("HF activations:", acts_hf["layer5"].shape)

# (B) nnsight: pause execution mid-forward and pull values declaratively
from nnsight import LanguageModel
model_ns = LanguageModel("gpt2")
with model_ns.trace("The Eiffel Tower is in"):
    layer5_acts = model_ns.transformer.h[5].output[0].save()
print("nnsight activations:", layer5_acts.shape)

# (C) transformer_lens: built-in HookedTransformer with named hook points
from transformer_lens import HookedTransformer
model_tl = HookedTransformer.from_pretrained("gpt2")
_, cache = model_tl.run_with_cache("The Eiffel Tower is in")
print("transformer_lens activations:", cache["blocks.5.hook_resid_post"].shape)
# All three return the same tensor; the differences are in API ergonomics.

10.5.1.1 Production XAI Libraries: Captum, LIME, and BertViz

The tools above (TransformerLens, SAELens, nnsight) serve the mechanistic interpretability community, where the goal is understanding internal model computations. A complementary set of tools addresses the production explainability problem: generating human-readable explanations of individual predictions for end users, auditors, or regulatory compliance. These libraries treat the model as a function (sometimes a black box) and explain its input-output behavior rather than its internal circuits.

Captum: Meta's Attribution Toolkit

Captum is Meta's comprehensive model interpretability library for PyTorch. It implements over a dozen attribution methods under a unified API, making it straightforward to compare different explanation approaches on the same prediction. For transformer models, the most commonly used methods are Layer Integrated Gradients (attributing to the embedding layer), Layer Gradient x Activation, and Layer Conductance (which measures the importance of individual neurons in a specific layer).

Captum's strength is its breadth: it covers gradient-based methods (Integrated Gradients, DeepLift, GradientSHAP), perturbation-based methods (Feature Ablation, Shapley Value Sampling, LIME via the Lime wrapper), and layer-level methods (Layer Conductance, Internal Influence). This means you can compare multiple explanation strategies on the same model without switching libraries.

Captum on image classification (vision)

Captum is not text-only. Applied to an image classifier (a ResNet-50 or ViT, for example), Layer Integrated Gradients produces a per-pixel saliency map for any chosen target class. The recipe is exactly the same as the text case: wrap the model's forward pass, choose a baseline (typically a zero or blurred image), pick a target class index, and call .attribute(). Captum then returns a tensor of the same shape as the input image, and captum.attr.visualization.visualize_image_attr overlays the saliency map on the original image. The two-line summary is:

from captum.attr import IntegratedGradients, visualization as viz
ig = IntegratedGradients(resnet50)
attr = ig.attribute(image_tensor, target=predicted_class, n_steps=50)
viz.visualize_image_attr(attr.squeeze().cpu().permute(1,2,0).numpy(),
                          original_image, method="heat_map", sign="positive")

# Comprehensive Captum attribution for a transformer classifier
import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer
from captum.attr import (
    LayerIntegratedGradients,
    LayerGradientXActivation,
    LayerConductance,
    visualization as viz,
    )
model_name = "distilbert-base-uncased-finetuned-sst-2-english"
model = AutoModelForSequenceClassification.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)
model.eval()
text = "The movie was surprisingly entertaining despite a weak script"
inputs = tokenizer(text, return_tensors="pt")
input_ids = inputs["input_ids"]
baseline_ids = torch.zeros_like(input_ids) # PAD token baseline
# Wrap the forward function for Captum
def forward_func(input_ids):
    outputs = model(input_ids)
    return outputs.logits[:, 1] # positive sentiment logit
# Method 1: Layer Integrated Gradients (most common for transformers)
lig = LayerIntegratedGradients(forward_func, model.distilbert.embeddings)
attrs_ig, delta = lig.attribute(
    input_ids, baselines=baseline_ids,
    n_steps=50, return_convergence_delta=True,
    )
# Convergence delta should be small (< 0.05); large values indicate
# that n_steps is too low for accurate integration.
# Method 2: Gradient x Activation (faster, less theoretically grounded)
lga = LayerGradientXActivation(forward_func, model.distilbert.embeddings)
attrs_gxa = lga.attribute(input_ids)
# Method 3: Layer Conductance (neuron-level importance in a specific layer)
lc = LayerConductance(forward_func, model.distilbert.transformer.layer[3])
attrs_cond = lc.attribute(input_ids, baselines=baseline_ids, n_steps=20)
# Summarize per-token attributions
tokens = tokenizer.convert_ids_to_tokens(input_ids[0])
attrs_sum = attrs_ig.sum(dim=-1).squeeze(0).detach().numpy()
print("Integrated Gradients attribution per token:")
for tok, score in zip(tokens, attrs_sum):
    bar = "#" * int(min(abs(score) * 10, 40))
    sign = "+" if score > 0 else "-"
    print(f" {tok:20s} {sign}{bar:40s} {score:+.4f}")

LIME for Language Models

LIME (Local Interpretable Model-agnostic Explanations) explains individual predictions by fitting a simple interpretable model (typically a sparse linear model) to the behavior of the complex model in the neighborhood of a specific input. For text, LIME works by randomly removing words from the input, observing how the model's prediction changes, and fitting a linear model that approximates the local decision boundary.

LIME's key advantage is that it is entirely model-agnostic: it treats the model as a black box and requires only the ability to call the model's prediction function. This makes it applicable to API-based LLMs where you have no access to gradients or internal activations. The tradeoff is that LIME's perturbation strategy (removing tokens) can create out-of-distribution inputs that a language model has never seen during training, potentially producing misleading attributions.

import torch
# LIME explanation for a text classifier (model-agnostic)
from lime.lime_text import LimeTextExplainer
import numpy as np
# Works with any model that returns class probabilities
def predict_proba(texts):
    """Prediction function that LIME will call repeatedly."""
    results = []
    for text in texts:
        inputs = tokenizer(text, return_tensors="pt",
            truncation=True, max_length=512)
        with torch.no_grad():
            logits = model(**inputs).logits[0]
            probs = torch.softmax(logits, dim=-1).numpy()
            results.append(probs)
            return np.array(results)
        explainer = LimeTextExplainer(class_names=["negative", "positive"])
        text = "The movie was surprisingly entertaining despite a weak script"
        explanation = explainer.explain_instance(
            text,
            predict_proba,
            num_features=10, # top 10 most important words
            num_samples=1000, # number of perturbations to generate
            )
        # Display word-level importance
        print("LIME feature importance (positive sentiment):")
        for word, weight in explanation.as_list():
            direction = "+" if weight > 0 else "-"
            bar = "#" * int(abs(weight) * 50)
            print(f" {word:20s} {direction}{bar} ({weight:+.4f})")
            # LIME also provides HTML visualization:
            # explanation.save_to_file("lime_explanation.html")
            # For API-based LLMs (no gradient access), LIME is often the only option.
            # Replace predict_proba with an API call wrapper:
            #
            # def predict_proba_api(texts):
            # results = []
            # for text in texts:
            # response = client.chat.completions.create(
            # model="gpt-4o-mini",
            # messages=[{"role": "user", "content": f"Classify: {text}"}],
            # logprobs=True,
            # )
            # # Extract probabilities from logprobs
            # results.append(parse_logprobs(response))
            # return np.array(results)

BertViz: Interactive Attention Visualization

BertViz provides interactive, browser-based visualizations of attention patterns across all layers and heads of a transformer model. It offers three visualization modes: the head view (attention from a single head as lines connecting tokens), the model view (all heads across all layers in a compact overview), and the neuron view (how individual neurons in Q, K, V contribute to attention). BertViz works in Jupyter notebooks and supports BERT, GPT-2, RoBERTa, XLNet, and other Hugging Face models.

While Section 10.1 covered attention visualization from scratch, BertViz is the production tool for this task. It is particularly useful for qualitative exploration: scanning attention patterns across layers to identify which heads attend to syntactic structure, which heads focus on positional patterns, and which heads appear to implement specific linguistic functions like coreference resolution or subject-verb agreement.

# BertViz: interactive attention visualization in Jupyter
# pip install bertviz
from bertviz import model_view, head_view, neuron_view
from transformers import AutoModel, AutoTokenizer
model = AutoModel.from_pretrained("bert-base-uncased",
    output_attentions=True)
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
text = "The bank raised interest rates after the financial crisis"
inputs = tokenizer(text, return_tensors="pt")
outputs = model(**inputs)
attention = outputs.attentions # tuple of (batch, heads, seq, seq) per layer
tokens = tokenizer.convert_ids_to_tokens(inputs["input_ids"][0])
# Model view: compact overview of all layers and heads
model_view(attention, tokens)
# Head view: detailed view for a specific layer
head_view(attention, tokens, layer=6, heads=[0, 3, 7])
# Neuron view: see Q/K/V contributions (requires special model loading)
# neuron_view(model, tokenizer, text, layer=6, head=3)
# Practical usage pattern: identify interesting heads, then investigate
# with probing classifiers (Section 10.1) or activation patching
# (Section 10.2) to confirm functional roles.

10.5.2 Evaluation of Explanation Quality

How do we know if an explanation is "good"? Several metrics have been proposed to evaluate explanation quality, each capturing different desirable properties.

Code Fragment 10.5.5 demonstrates this approach.

Code Fragment 10.5.5a evaluates whether the attributed tokens actually drive the prediction, using both sufficiency (keeping only top-k) and comprehensiveness (removing top-k) tests.

import numpy as np
import torch
# Faithfulness evaluation for attribution methods
def evaluate_faithfulness(
    model,
    tokenizer,
    text,
    attributions,
    k_values=[1, 3, 5],
    ):
    """
    Evaluate faithfulness of attributions using sufficiency and comprehensiveness.
    """
    inputs = tokenizer(text, return_tensors="pt")
    tokens = tokenizer.convert_ids_to_tokens(inputs["input_ids"][0])
    with torch.no_grad():
        baseline_logits = model(**inputs).logits[0, -1]
        predicted_id = baseline_logits.argmax()
        baseline_prob = torch.softmax(baseline_logits, dim=-1)[predicted_id].item()
        sorted_indices = np.argsort(attributions)[::-1]
        results = {}
        for k in k_values:
            top_k = sorted_indices[:k]
            # Sufficiency: keep only top-k tokens
            sufficient_ids = inputs["input_ids"].clone()
            mask = torch.ones(len(tokens), dtype=torch.bool)
            mask[list(top_k)] = False
            sufficient_ids[0, mask] = tokenizer.pad_token_id or 0
            with torch.no_grad():
                suf_logits = model(sufficient_ids).logits[0, -1]
                suf_prob = torch.softmax(suf_logits, dim=-1)[predicted_id].item()
                sufficiency = suf_prob / baseline_prob # closer to 1 = better
                # Comprehensiveness: remove top-k tokens
                comp_ids = inputs["input_ids"].clone()
                for idx in top_k:
                    comp_ids[0, idx] = tokenizer.pad_token_id or 0
                    with torch.no_grad():
                        comp_logits = model(comp_ids).logits[0, -1]
                        comp_prob = torch.softmax(comp_logits, dim=-1)[predicted_id].item()
                        comprehensiveness = 1 - (comp_prob / baseline_prob) # closer to 1 = better
                        results[f"k={k}"] = {
                            "sufficiency": sufficiency,
                            "comprehensiveness": comprehensiveness,
                            }
                        return results

10.5.3 LLMs as Interpretability Assistants

A surprising twist in the interpretability story is that LLMs themselves have become powerful tools for explaining other models. Instead of relying solely on numerical attribution scores or heatmaps, practitioners are using language models to generate natural language explanations of model behavior, produce counterfactual analyses, and even automate the labeling of internal model features discovered through mechanistic interpretability.

10.5.3.1 Natural Language Explanations of Predictions

Given a model's input, output, and attribution scores, an LLM can synthesize a human-readable explanation: "The model predicted negative sentiment primarily because of the phrase 'deeply disappointed,' which received the highest attribution score. The word 'excellent' in the second sentence pulled toward positive sentiment but was outweighed by the negative signals." This transforms opaque numerical outputs into narratives that stakeholders can understand and critique. The key advantage is accessibility: a product manager does not need to interpret a SHAP waterfall chart if an LLM can narrate the same information in plain language.

10.5.3.2 LLM-Generated Counterfactual Explanations

Counterfactual explanations answer the question "what would need to change for the prediction to be different?" An LLM can generate these by prompting it with the original input and prediction, then asking it to produce minimal modifications that would flip the outcome. For example: "The loan application was denied because the debt-to-income ratio of 45% exceeds the threshold. The prediction would change to approved if the monthly debt payments decreased from $2,700 to below $2,000, or if annual income increased from $72,000 to above $85,000." These explanations are actionable in ways that feature importance scores are not, and they satisfy regulatory requirements in domains like finance and healthcare where model decisions must be explainable.

10.5.3.3 Automated Model Card Generation

Model cards (Mitchell et al., 2019) document a model's intended use, performance characteristics, limitations, and ethical considerations. Writing them manually is tedious and often skipped. LLMs can automate this by analyzing a model's evaluation results, training data statistics, and configuration, then generating a structured model card that covers performance breakdowns by demographic group, known failure modes, and recommended use cases. While the generated card requires human review, it reduces the documentation burden from hours to minutes and ensures that no standard section is accidentally omitted.

10.5.3.4 Auto-Labeling SAE Features with LLMs

The sparse autoencoders (SAEs) discussed in Section 10.3 decompose model activations into thousands of interpretable features, but each feature needs a human-readable label to be useful. OpenAI's "Language models can explain neurons in language models" (Bills et al., 2023) pioneered the approach of using GPT-4 to automatically describe what each neuron computes by showing it the neuron's top-activating examples and asking for a natural language summary. Neuronpedia scales this approach to the features discovered by SAEs, using LLMs to auto-label features from Gemma Scope (covered earlier in Section 10.3) and other SAE analyses. The process works as follows: collect the top 20 text examples that maximally activate a given SAE feature, present them to an LLM with the prompt "What concept or pattern do these examples share?", and store the generated label alongside the feature. Human spot-checks verify label quality, and the community can propose corrections.

10.5.3.5 Practical Workflow

A typical LLM-assisted interpretability workflow combines the techniques above. First, run standard attribution methods (Integrated Gradients, attention rollout) on a set of important predictions. Second, feed the attributions into an LLM to generate natural language explanations and counterfactuals. Third, use SAE feature analysis with LLM auto-labeling to identify higher-level circuits involved in the prediction. Fourth, compile the results into an auto-generated model card. This end-to-end pipeline makes interpretability accessible to teams that lack dedicated interpretability researchers, democratizing a practice that was previously confined to specialized labs.

Exercises