Part IV

Part IV: LLM Training and Adaptation

Generating data, fine-tuning models, distilling knowledge, and aligning with human preferences.

Chapter opener illustration: Part IV: LLM Training and Adaptation.

"Give me a lever long enough and a fulcrum on which to place it, and I shall move the world."

FinetuneFinetune, Gradient-Loving AI Agent

Part Overview

Part IV is the heart of the book for practitioners who want to customize LLMs. You will learn to generate synthetic training data, fine-tune models (full and parameter-efficient), distill large models into smaller ones, merge model weights, and align models with human preferences via RLHF, DPO, Constitutional AI, and RLVR. This is the most technically dense part of the book; take your time with the labs.

Chapters: 5 (Chapters 15 through 19). Builds on API and prompting skills from Part III and supplies the trained models used in Part V and beyond. The part closes with a Tools of the Trade chapter on the transformers / trl / peft / axolotl / lit-gpt training stack.

Big Picture

Off-the-shelf models only get you so far. Part IV teaches you to bend LLMs to your needs through synthetic data, fine-tuning, distillation, and alignment, turning general-purpose models into specialized tools you can trust.

Chapter 15 Synthetic Data Generation & LLM Simulation
Chapter 16 Fine-Tuning Fundamentals
Chapter 17 Parameter-Efficient Fine-Tuning, Distillation & Model Merging
Chapter 18 Alignment: RLHF, DPO & Preference Tuning
Chapter 19 Tools of the Trade: Training & Adaptation Stack

What's Next?

This part begins with Chapter 15: Synthetic Data Generation & LLM Simulation. Each chapter builds on the previous one, so we recommend reading Part IV in order.