When and Why to Fine-Tune

"Before you fine-tune, ask yourself: have I tried writing a better prompt? If yes, have I tried RAG? If yes, welcome. You may proceed."

Finetune, Gatekeeping AI Agent

1. The Adaptation Spectrum

When a pre-trained language model does not meet your needs out of the box, you have several options for adapting it. These options form a spectrum from lightweight (no training required) to heavyweight (full model retraining). Understanding where each technique sits on this spectrum is essential for making cost-effective decisions.

1.1 Prompting, RAG, and Fine-Tuning

The three primary approaches to model adaptation differ in their complexity, cost, and the types of improvements they can deliver. Prompt engineering is the simplest: you craft instructions that guide the model toward the desired behavior at inference time. RAG augments the model with external knowledge by retrieving relevant documents and injecting them into the prompt. Fine-tuning modifies the model weights themselves through additional training on task-specific data. Figure 14.1.1 places these approaches on the adaptation spectrum.

1.2 The Decision Framework

The following decision framework helps you determine which approach to try first. The key insight is that you should start with the lightest approach that could work and only move to heavier approaches when you have evidence that simpler methods fall short.

Code Fragment 14.1.2 encodes this decision framework as a function, walking through each criterion in priority order: try prompting first, then RAG, and fine-tune only when lighter approaches genuinely fall short.

# Decision framework: choose the lightest adaptation that meets requirements
# Evaluates prompting, RAG, and fine-tuning in ascending order of commitment
def choose_adaptation_strategy(task):
 """Decision framework for choosing between prompting, RAG, and fine-tuning."""

 # Step 1: Can prompting solve it?
 if task.can_be_described_in_prompt:
 baseline = evaluate_with_prompting(task)
 if baseline.meets_quality_threshold:
 return "prompting" # Simplest solution that works

 # Step 2: Is the gap about missing knowledge?
 if task.requires_external_knowledge:
 if task.knowledge_changes_frequently:
 return "RAG" # Dynamic knowledge needs retrieval
 if task.knowledge_is_static and task.dataset_size > 10_000:
 return "fine-tuning" # Large static knowledge: bake it in

 # Step 3: Is the gap about behavior or style?
 if task.requires_specific_style or task.requires_specific_format:
 if few_shot_examples_in_prompt_work:
 return "prompting" # Few-shot can handle simple format changes
 return "fine-tuning" # Complex style/format needs weight updates

 # Step 4: Is the gap about latency or cost?
 if task.latency_budget_ms < 200 or task.cost_per_query_budget < 0.001:
 return "fine-tuning" # Smaller fine-tuned model is faster and cheaper

 # Step 5: Combine approaches
 return "RAG + fine-tuning" # Many production systems use both

Code Fragment 14.1.2 configures LoRA adapters.

# Quick comparison: resource requirements
def estimate_training_resources(
 model_size_billions: float,
 method: str = "full", # "full", "lora", "qlora"
 precision: str = "fp16"
) -> dict:
 """Estimate GPU memory and storage for fine-tuning."""
 bytes_per_param = {"fp32": 4, "fp16": 2, "bf16": 2, "int8": 1, "int4": 0.5}

 param_bytes = bytes_per_param.get(precision, 2)
 model_memory_gb = model_size_billions * 1e9 * param_bytes / (1024**3)

 if method == "full":
 # Model + gradients + optimizer states (AdamW: 2x for momentum)
 training_memory_gb = model_memory_gb * 4 # Rough 4x multiplier
 trainable_params = model_size_billions * 1e9
 checkpoint_gb = model_memory_gb
 elif method == "lora":
 # Frozen model + small adapter gradients/optimizer
 trainable_params = model_size_billions * 1e9 * 0.01 # ~1% of params
 training_memory_gb = model_memory_gb + 2 # Base + adapter overhead
 checkpoint_gb = 0.1 # Adapter only
 elif method == "qlora":
 # 4-bit quantized model + adapter
 model_memory_gb = model_size_billions * 1e9 * 0.5 / (1024**3)
 trainable_params = model_size_billions * 1e9 * 0.01
 training_memory_gb = model_memory_gb + 2
 checkpoint_gb = 0.1

 return {
 "method": method,
 "model_size": f"{model_size_billions}B",
 "training_memory_gb": round(training_memory_gb, 1),
 "trainable_params": f"{trainable_params/1e6:.1f}M",
 "checkpoint_size_gb": round(checkpoint_gb, 1),
 "min_gpu": "A100 80GB" if training_memory_gb > 40 else "A100 40GB"
 if training_memory_gb > 20 else "RTX 4090 24GB"
 if training_memory_gb > 16 else "RTX 3090 24GB"
 }

# Compare methods for a 7B model
for method in ["full", "lora", "qlora"]:
 result = estimate_training_resources(7.0, method=method)
 print(f"{method:6s}: {result['training_memory_gb']:5.1f} GB, "
 f"{result['trainable_params']:>8s} params, "
 f"checkpoint: {result['checkpoint_size_gb']} GB")

4. Catastrophic Forgetting

Catastrophic forgetting is the phenomenon where a model, after being fine-tuned on a specific task, loses its ability to perform well on other tasks it could previously handle. This happens because gradient updates that improve performance on the fine-tuning data can overwrite weights that encode general knowledge.

4.1 Symptoms and Causes

The most common symptoms of catastrophic forgetting include degraded performance on general benchmarks (MMLU, HellaSwag), loss of instruction-following ability, increased repetition or degenerate outputs, and inability to handle prompts outside the fine-tuning distribution. The primary causes are training for too many epochs, using a learning rate that is too high, training on a dataset that is too narrow in distribution, and failing to include regularization. Figure 14.1.4 shows how task-specific and general performance diverge over training. Code Fragment 14.1.3 shows this approach in practice.

4.2 Mitigation Strategies

Code Fragment 14.1.3 packages the main forgetting mitigation strategies into a configuration class, covering learning rate scheduling, data mixing ratios, layer freezing, and regularization.

# Strategies for mitigating catastrophic forgetting
from dataclasses import dataclass
from typing import List, Optional

@dataclass
class ForgettingMitigationConfig:
 """Configuration for preventing catastrophic forgetting."""

 # 1. Learning rate: use a low learning rate for fine-tuning
 learning_rate: float = 2e-5 # 10x lower than pre-training

 # 2. Short training: fewer epochs reduce overwriting
 num_epochs: int = 3 # Rarely need more than 3-5

 # 3. Data mixing: include general-purpose data
 task_data_ratio: float = 0.7 # 70% task-specific
 general_data_ratio: float = 0.3 # 30% general (e.g., OpenAssistant)

 # 4. Regularization
 weight_decay: float = 0.01
 max_grad_norm: float = 1.0

 # 5. Evaluation on general benchmarks during training
 eval_general_benchmarks: bool = True
 general_eval_datasets: List[str] = None

 def __post_init__(self):
 if self.general_eval_datasets is None:
 self.general_eval_datasets = [
 "mmlu", # General knowledge
 "hellaswag", # Commonsense reasoning
 "arc_easy", # Science questions
 ]

 def get_data_mix(self, task_samples: int) -> dict:
 """Calculate how many general samples to mix in."""
 general_samples = int(
 task_samples * self.general_data_ratio / self.task_data_ratio
 )
 return {
 "task_samples": task_samples,
 "general_samples": general_samples,
 "total": task_samples + general_samples,
 "effective_task_ratio": task_samples / (task_samples + general_samples)
 }

config = ForgettingMitigationConfig()
mix = config.get_data_mix(task_samples=5000)
print(f"Task: {mix['task_samples']}, General: {mix['general_samples']}, "
 f"Total: {mix['total']}")

4.3 Machine Unlearning

While catastrophic forgetting causes unintentional loss of knowledge, machine unlearning pursues the opposite goal: deliberately removing the influence of specific training data from a trained model. The primary regulatory motivation comes from the GDPR's "right to be forgotten" and the CCPA's data deletion requirements, which grant individuals the right to have their data removed from systems that processed it. For LLMs trained on billions of web-crawled documents, complying with a deletion request is non-trivial: simply removing the data from the training set does not remove its influence from the model weights.

Exact vs. Approximate Unlearning

Exact unlearning means retraining the model from scratch on the original dataset minus the data to be forgotten. This provides a provable guarantee (the resulting model is identical to one that never saw the data), but it is computationally prohibitive for large models. Training a 70B-parameter model costs millions of dollars; retraining for each deletion request is not feasible.

SISA (Sharded, Isolated, Sliced, Aggregated) training makes exact unlearning practical by partitioning the training data into disjoint shards. Each shard trains an independent sub-model, and the final model aggregates their predictions. When a data point must be forgotten, only the shard containing it needs retraining. The cost drops from retraining the entire model to retraining 1/N of it (where N is the number of shards). SISA was designed for traditional ML models; adapting it to LLM pretraining remains an open challenge because of the sequential, curriculum-dependent nature of language model training.

Approximate Unlearning for LLMs

Gradient ascent-based unlearning is the most common approximate approach for LLMs. The idea is simple: if training minimized the loss on the target data, unlearning maximizes it. The model is fine-tuned for a small number of steps with the loss function negated on the data to be forgotten, pushing the model away from those examples. This is often combined with a retention loss on a held-out general dataset to prevent the model from degrading on unrelated tasks.

However, approximate unlearning offers no formal guarantee that the data's influence is fully removed. Membership inference attacks can sometimes still detect traces of "unlearned" data, and the degree of removal varies depending on how memorized the data was. Highly duplicated or distinctive training examples leave deeper traces that are harder to erase.

The Editing-Unlearning Conflict

Machine unlearning interacts poorly with knowledge editing (Section 18.3). Sequential knowledge edits can inadvertently reintroduce information that unlearning removed, because both techniques modify overlapping regions of the model's weight space. Organizations applying both techniques must carefully sequence operations and verify that unlearning guarantees are preserved after subsequent edits.

5. Continual Pre-Training vs. Instruction Fine-Tuning

Fine-tuning comes in two distinct flavors that serve different purposes. Continual pre-training (also called domain-adaptive pre-training) extends the original pre-training objective on domain-specific text. Instruction fine-tuning (also called supervised fine-tuning or SFT) trains the model to follow instructions and produce specific outputs. Understanding the difference is critical for choosing the right approach.

5.1 Continual Pre-Training

Continual pre-training uses the same next-token prediction objective as the original pre-training, but on a domain-specific corpus. The model learns the vocabulary, concepts, and reasoning patterns of the target domain without any explicit instruction/output pairs. This is useful when the model lacks fundamental domain knowledge.

5.2 Instruction Fine-Tuning (SFT)

Instruction fine-tuning trains the model on input/output pairs where each input is a user instruction or query and each output is the desired response. This teaches the model to follow instructions, produce specific output formats, and adopt particular behaviors. Most practical fine-tuning falls into this category.

The boundary between prompting and fine-tuning is blurring with techniques like in-context learning distillation, which compresses few-shot prompting behavior into model weights. Research on task arithmetic suggests that fine-tuning creates interpretable weight deltas that can be composed, negated, or scaled to control model behavior without retraining.

An open question is whether future models will make fine-tuning obsolete through sufficiently powerful in-context learning, or whether weight-level adaptation will always offer efficiency advantages.