"The secret of getting ahead is getting started. The secret of fine-tuning is knowing when to stop."
Finetune, Wisely Restrained AI Agent
You have data (Chapter 17). Now you fine-tune. This chapter is the canonical home for fine-tuning fundamentals: when to fine-tune at all (vs. prompting or RAG), full fine-tuning vs. parameter-efficient methods, catastrophic forgetting, and the 5-question decision tree (see Section 16.1) that the rest of the book cross-references when fine-tuning comes up.
Chapter Overview
A small fine-tuned model often beats GPT-4 on the narrow task you actually care about, at a fraction of the latency and cost. That is the practitioner secret behind most production LLM products in 2026: a 7B Llama or Qwen, fine-tuned on a few thousand high-quality examples, runs your support classifier or your contract extractor with higher accuracy and a fraction of the bill that an API generalist incurs. The catch is knowing when fine-tuning actually wins, and when prompting or RAG is the cheaper answer.
This chapter covers the complete fine-tuning workflow from first principles. You will learn when fine-tuning is the right approach (and when prompting or RAG is a better alternative), how to prepare high-quality training data in the correct format, and how to run supervised fine-tuning with Hugging Face TRL. The chapter also covers API-based fine-tuning through providers like OpenAI and Google, fine-tuning for embedding and classification tasks, and strategies for adapting models to handle longer contexts.
Fine-tuning transforms a general-purpose LLM into a specialist for your domain. This chapter covers the full workflow: data preparation, training configuration, catastrophic forgetting mitigation, and evaluation. It provides the foundation for the parameter-efficient methods in Chapter 19 and alignment techniques in Chapter 20.
- Apply a decision framework to choose between prompting, RAG, and fine-tuning for a given task, informed by evaluation metrics
- Prepare training datasets in standard formats (Alpaca, ShareGPT, ChatML) with appropriate splits and balancing strategies
- Execute supervised fine-tuning using Hugging Face Trainer and TRL, selecting appropriate hyperparameters for learning rate, batch size, warmup, and weight decay
- Use provider APIs (OpenAI, Google Vertex AI) for fine-tuning and evaluate trade-offs between ease, control, and cost
- Fine-tune encoder and decoder models for representation learning and embedding tasks
- Add and train classification heads for single-label, multi-label, and token-level classification tasks
- Apply context extension techniques (RoPE scaling, position interpolation) to adapt models for longer input sequences
- Monitor training with W&B and TensorBoard, diagnose common issues, and mitigate catastrophic forgetting; prepare models for alignment workflows
Prerequisites
- Chapter 1: Tokenization (understanding how text is converted to tokens)
- Chapter 3: Transformer Architecture (self-attention, positional encoding, feed-forward layers)
- Chapter 6: Pretraining (training objectives, loss functions, training dynamics)
- Chapter 11: LLM APIs (API usage, structured outputs)
- Chapter 15: Synthetic Data Generation (creating training data at scale)
- Familiarity with PyTorch, gradient descent, and basic training loops
Sections
- 16.1 When and Why to Fine-Tune Fine-tuning is powerful, but it is not always the right tool. Entry
- 16.2 Data Preparation for Fine-Tuning Data quality is the single biggest lever in fine-tuning. Entry
- 16.3 Supervised Fine-Tuning (SFT) Supervised fine-tuning (SFT) is the core technique for teaching a pretrained model to follow instructions and produce specific outputs. Intermediate
- 16.4 Fine-Tuning via Provider APIs Not every team needs to manage their own GPU cluster. Intermediate
- 16.5 Fine-Tuning for Representation Learning Embeddings are the backbone of modern search, retrieval, and recommendation systems. Intermediate
- 16.6 Fine-Tuning for Classification & Sequence Tasks Classification is the most common fine-tuning task in production NLP. Advanced
- 16.7 Adapting Models for Long Text Most real-world documents are longer than models were trained to handle. Advanced
Objective
Take a small base model (GPT2-medium or Pythia-410M) and run a full fine-tune on a domain corpus. You will measure two things that matter: (a) gains on your target task, (b) loss on a held-out general benchmark. This is the canonical "what fine-tuning costs you" experiment, and the answer motivates LoRA in Chapter 17.
Steps
- Step 1: Choose corpus + baseline. Pick a domain: arXiv abstracts (CS subset, ~10k docs) or PubMed. Score the base model's perplexity on a held-out 1k-doc test split. Also score on WikiText-2 (general baseline). Save both numbers.
- Step 2: Prepare data. Tokenize with the model's own tokenizer, chunk to 512 tokens, mask only the response in instruction format (loss only on completion tokens). Use the
trlDataCollatorForCompletionOnlyLM. - Step 3: Train. Run
Trainerwithper_device_train_batch_size=4,gradient_accumulation_steps=8,learning_rate=5e-5,num_train_epochs=2. Use a single GPU (T4 is enough for 350M). Log loss every 50 steps. - Step 4: Re-measure both perplexities. Target domain ppl should drop 20 to 40%. WikiText-2 ppl should rise (catastrophic forgetting). Quantify both.
- Step 5: Forgetting-mitigation A/B. Retrain with 10% of the original pretraining-style data mixed in. Re-measure: WikiText-2 ppl should rise less. This is the simplest forgetting-mitigation recipe.
- Step 6: Compare with PEFT (preview of Ch 17). Train a LoRA on the same setup and re-measure both ppls. You should see most of the domain gain with less forgetting. This sets up why Ch 17 exists.
Expected Output
Expected time: 4 to 5 hours (most is GPU training time). Difficulty: intermediate. Artifact: a 2x2 ppl table (base / FT / FT+replay / LoRA) on (domain / general).
What's Next?
Next: Chapter 17: Parameter-Efficient Fine-Tuning (PEFT). Full fine-tuning of a 70B model needs eight A100s and burns through your budget on the first run. What if you could match its quality by updating less than 1% of the parameters? Chapter 17 covers LoRA, QLoRA, adapter layers, and prefix tuning, the methods that let you adapt frontier-scale models on a single consumer GPU.