LoRA & QLoRA

"Why update a billion parameters when a few thousand will do? Efficiency is not laziness; it is elegance under constraint."

LoRA, Elegantly Lazy AI Agent

1. The Full Fine-Tuning Problem

When you fine-tune a model with full parameter updates, every weight in the model gets a gradient, an optimizer state (momentum and variance for Adam), and a copy of the updated weight. For a 7B parameter model in FP16, that means 14 GB just for the weights, plus roughly 42 GB for optimizer states, totaling over 56 GB of GPU memory. Scaling to 13B or 70B models makes this prohibitively expensive.

The key insight behind parameter-efficient methods is that the weight changes during fine-tuning are low-rank. Research (Aghajanyan et al., 2021) has shown that when you compute the difference between a fine-tuned model and its pretrained base (the "task-specific delta"), this delta matrix has a very low intrinsic dimensionality, meaning it can be well-approximated by a matrix of much lower rank. Most of the information in the update can be captured by a much smaller matrix. Think of it like image compression: a JPEG discards high-frequency details that humans barely notice, keeping the image recognizable at a fraction of the file size. Similarly, LoRA discards the high-rank components of the weight update, keeping only the most important directions of change.

Why does LoRA work with so few parameters? The deep intuition is that fine-tuning does not need to change the model's fundamental capabilities; it only needs to steer existing capabilities toward a specific task. The pretrained weight matrices encode rich, general-purpose representations built from trillions of tokens. The task-specific adaptation (learning to follow instructions, adopting a particular output format, specializing in a domain) requires only small directional adjustments to these representations. Mathematically, these adjustments occupy a low-dimensional subspace of the full weight space, which is precisely what low-rank matrices capture. This is why rank-4 or rank-8 LoRA frequently matches full fine-tuning: the effective dimensionality of the task adaptation is genuinely small, regardless of the model's total parameter count.

2. LoRA Mathematics

2.1 The Core Decomposition

LoRA (Low-Rank Adaptation) works by expressing the weight update as a product of two small matrices. For a pretrained weight matrix W of dimension d × k, instead of computing a full update ΔW (also d × k), LoRA decomposes it as:

W' = W + ΔW = W + BA

where B is d × r and A is r × k, with the rank r being much smaller than both d and k. Typical values of r range from 4 to 64, while d and k are typically 4096 or larger. This means the number of trainable parameters drops from d × k (e.g., 16.7 million for a 4096 × 4096 matrix) to r × (d + k) (e.g., 131,072 for r=16). Figure 15.1.1 visualizes this decomposition.

Figure 15.1.3: LoRA decomposes the weight update into two small trainable matrices B and A, reducing parameters by ~128x. Source: Sebastian Raschka, 2023. Parameter-Efficient LLM Finetuning With Low-Rank Adaptation (LoRA).

2.2 Initialization and Scaling

LoRA uses a specific initialization strategy: matrix A is initialized with a random Gaussian distribution, and matrix B is initialized to all zeros. This means that at the start of training, BA = 0, and the model behaves exactly like the pretrained base. Training gradually moves the model away from this starting point.

The scaling factor α (alpha) controls the magnitude of the LoRA update. The actual update applied is:

W' = W + (α / r) · BA

The ratio α/r acts as a learning rate multiplier for the LoRA weights. A common convention is to set α = 2r (so the effective multiplier is 2), but the optimal value depends on the task. Increasing α relative to r makes the adaptation more aggressive; decreasing it keeps the model closer to the pretrained weights.

2.3 Why Low-Rank Works

The effectiveness of low-rank adaptation rests on a remarkable empirical finding: the "intrinsic dimensionality" of fine-tuning updates is far lower than the full parameter count would suggest. When researchers analyzed the singular value decomposition of ΔW matrices from full fine-tuning runs, they found that a small number of singular values capture the vast majority of the update's information content. In many cases, ranks as low as 4 or 8 capture over 90% of the useful signal.

This makes intuitive sense. Fine-tuning typically adapts a model to a specific domain or task format. The knowledge required for this adaptation (new terminology, output format preferences, domain-specific reasoning patterns) is a small modification relative to the vast general knowledge encoded in the pretrained weights.

3. LoRA Hyperparameters in Practice

3.1 Rank (r) Selection

3.2 Target Module Selection

Not all weight matrices benefit equally from LoRA adaptation. The standard practice is to apply LoRA to the attention projection matrices: q_proj, k_proj, v_proj, and o_proj. Research and practice have converged on the recommendation to also include the MLP layers (gate_proj, up_proj, down_proj) for best results, though this increases trainable parameters. Code Fragment 15.1.7 shows this approach in practice.

# Configure LoRA adapter parameters: rank, alpha, target modules
# Lower rank reduces trainable parameters; alpha scales the adapter contribution
from peft import LoraConfig, get_peft_model, TaskType

# Standard configuration: attention layers only
lora_config_basic = LoraConfig(
 task_type=TaskType.CAUSAL_LM,
 r=16,
 lora_alpha=32, # alpha = 2r
 lora_dropout=0.05,
 target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
 bias="none",
)

# Recommended: attention + MLP layers for best quality
lora_config_full = LoraConfig(
 task_type=TaskType.CAUSAL_LM,
 r=16,
 lora_alpha=32,
 lora_dropout=0.05,
 target_modules=[
 "q_proj", "k_proj", "v_proj", "o_proj",
 "gate_proj", "up_proj", "down_proj",
 ],
 bias="none",
)

# Apply LoRA to a model
model = get_peft_model(model, lora_config_full)
model.print_trainable_parameters()

Code Fragment 15.1.2 configures LoRA adapters.

# Set up parameter-efficient fine-tuning with LoRA adapters
# Freeze the base model and train only the low-rank decomposition matrices
from peft import PeftModel, AutoPeftModelForCausalLM
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

# === Option A: Load adapter separately ===
base_model = AutoModelForCausalLM.from_pretrained(
 "meta-llama/Meta-Llama-3-8B",
 torch_dtype=torch.bfloat16,
 device_map="auto",
)
# Load different adapters dynamically
model = PeftModel.from_pretrained(base_model, "./lora-medical")
# Switch to another adapter
model.load_adapter("./lora-legal", adapter_name="legal")
model.set_adapter("legal")

# === Option B: Merge and save ===
model = AutoPeftModelForCausalLM.from_pretrained(
 "./lora-medical",
 torch_dtype=torch.bfloat16,
 device_map="auto",
)
merged_model = model.merge_and_unload()
merged_model.save_pretrained("./llama3-medical-merged")
tokenizer.save_pretrained("./llama3-medical-merged")

# === Option C: Merge QLoRA (requires dequantization) ===
# Load the QLoRA model in full precision for merging
model = AutoPeftModelForCausalLM.from_pretrained(
 "./qlora-output",
 torch_dtype=torch.float16,
 device_map="auto",
 low_cpu_mem_usage=True,
)
merged = model.merge_and_unload()
merged.save_pretrained("./merged-model")

print("Merged model saved. Upload to HF Hub or serve with vLLM.")

7. LoRA Hyperparameter Tuning Guide

Finding optimal LoRA hyperparameters requires systematic experimentation. Here is a practical guide based on what works across a wide range of tasks and model sizes.

8. The PEFT Library Ecosystem

The Hugging Face peft library provides a unified interface for all parameter-efficient methods. Beyond basic LoRA, it supports loading adapters from the Hub, combining adapters, and quantized training workflows. Code Fragment 15.1.2 shows this approach in practice.

# Configure LoRA adapter parameters: rank, alpha, target modules
# Lower rank reduces trainable parameters; alpha scales the adapter contribution
from peft import (
 PeftModel,
 PeftConfig,
 get_peft_model,
 LoraConfig,
 TaskType,
 AutoPeftModelForCausalLM,
)

# Load a LoRA adapter from Hugging Face Hub
model = AutoPeftModelForCausalLM.from_pretrained(
 "username/my-lora-adapter", # Adapter repo on HF Hub
 device_map="auto",
 torch_dtype=torch.bfloat16,
)

# Inspect adapter configuration
config = PeftConfig.from_pretrained("username/my-lora-adapter")
print(f"Base model: {config.base_model_name_or_path}")
print(f"Rank: {config.r}, Alpha: {config.lora_alpha}")
print(f"Target modules: {config.target_modules}")

# Push adapter to Hub (only saves the small adapter weights)
model.push_to_hub("username/my-lora-adapter")
# Adapter size: typically 50-200 MB vs 14+ GB for full model

QLoRA: 4-Bit Quantized LoRA

QLoRA (Dettmers et al., 2023) combines LoRA with 4-bit NF4 quantization of the base model, reducing memory by roughly 4x compared to standard LoRA in FP16. The bitsandbytes library handles the quantization transparently through BitsAndBytesConfig. The frozen base weights are stored in 4-bit precision, while the LoRA adapter matrices train in BF16. This enables fine-tuning a 70B model on a single 48 GB GPU. Code Fragment 15.1.10 shows the setup.

# pip install bitsandbytes peft transformers
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
import torch

# 4-bit NF4 quantization config
bnb_config = BitsAndBytesConfig(
 load_in_4bit=True,
 bnb_4bit_quant_type="nf4",
 bnb_4bit_compute_dtype=torch.bfloat16,
 bnb_4bit_use_double_quant=True, # quantize the quantization constants
)

model = AutoModelForCausalLM.from_pretrained(
 "meta-llama/Llama-3.1-8B-Instruct",
 quantization_config=bnb_config,
 device_map="auto",
)
model = prepare_model_for_kbit_training(model) # freeze + enable gradients

lora_config = LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05,
 target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
 "gate_proj", "up_proj", "down_proj"],
 bias="none", task_type="CAUSAL_LM")
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# trainable params: 13,631,488 || all params: 4,544,075,776 || trainable%: 0.30

pip install transformers trl peft datasets accelerate torch bitsandbytes

# Load a small model and wrap with LoRA: only the low-rank adapter
# matrices are trainable; all original weights stay frozen.
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import LoraConfig, get_peft_model, TaskType
import torch

model_name = "HuggingFaceTB/SmolLM2-135M-Instruct"
tokenizer = AutoTokenizer.from_pretrained(model_name)
if tokenizer.pad_token is None:
 tokenizer.pad_token = tokenizer.eos_token

base_model = AutoModelForCausalLM.from_pretrained(
 model_name, torch_dtype=torch.float16, device_map="auto"
)

# TODO: Create a LoraConfig with rank=16, alpha=32
lora_config = LoraConfig(
 r=16,
 lora_alpha=32,
 target_modules=["q_proj", "v_proj"],
 task_type=TaskType.CAUSAL_LM,
 lora_dropout=0.05,
 bias="none"
)

model = get_peft_model(base_model, lora_config)
model.print_trainable_parameters()

# Compare trainable parameter counts across LoRA ranks (4 to 64).
# Shows how rank controls the expressiveness vs. efficiency tradeoff.
import pandas as pd

ranks = [4, 8, 16, 32, 64]
results = []

for r in ranks:
 base = AutoModelForCausalLM.from_pretrained(
 model_name, torch_dtype=torch.float16, device_map="auto")
 config = LoraConfig(r=r, lora_alpha=r * 2,
 target_modules=["q_proj", "v_proj"],
 task_type=TaskType.CAUSAL_LM, lora_dropout=0.05, bias="none")
 pm = get_peft_model(base, config)

 # TODO: Extract trainable and total parameter counts
 trainable = sum(p.numel() for p in pm.parameters() if p.requires_grad)
 total = sum(p.numel() for p in pm.parameters())
 results.append({"rank": r, "trainable": trainable,
 "total": total, "pct": f"{trainable/total*100:.4f}%"})
 del pm, base
 torch.cuda.empty_cache()

df = pd.DataFrame(results)
print(df.to_string(index=False))

# Train the LoRA adapter (r=16) on 300 instruction examples.
# Only adapter weights update; base model gradients are skipped.
from trl import SFTTrainer, SFTConfig
from datasets import load_dataset

dataset = load_dataset("HuggingFaceH4/no_robots", split="train")
dataset = dataset.shuffle(seed=42).select(range(300))

def format_chat(example):
 return {"text": tokenizer.apply_chat_template(
 example["messages"], tokenize=False, add_generation_prompt=False)}

formatted = dataset.map(format_chat)

# TODO: Configure training (use higher LR than full fine-tuning)
training_args = SFTConfig(
 output_dir="./lora-smollm2-r16",
 num_train_epochs=3,
 per_device_train_batch_size=4,
 gradient_accumulation_steps=2,
 learning_rate=2e-4, # 10x higher than full FT
 logging_steps=10,
 max_seq_length=512,
 fp16=True,
 warmup_ratio=0.1,
 save_strategy="epoch",
 report_to="none",
)

trainer = SFTTrainer(model=model, args=training_args,
 train_dataset=formatted, processing_class=tokenizer)
result = trainer.train()
print(f"Training loss: {result.training_loss:.4f}")

# Save the tiny adapter checkpoint, then merge it back into the
# base model for deployment without the PEFT dependency.
import os
from peft import PeftModel

# Save adapter only
model.save_pretrained("./lora-adapter-r16")
adapter_size = sum(
 os.path.getsize(os.path.join("./lora-adapter-r16", f))
 for f in os.listdir("./lora-adapter-r16")
 if f.endswith(('.safetensors', '.bin')))
print(f"Adapter size: {adapter_size / 1024:.1f} KB")

# Reload base model and merge adapter
base_fresh = AutoModelForCausalLM.from_pretrained(
 model_name, torch_dtype=torch.float16, device_map="auto")
peft_model = PeftModel.from_pretrained(base_fresh, "./lora-adapter-r16")
merged_model = peft_model.merge_and_unload()
print("Adapter merged successfully!")

# Compare merged model outputs against base model on test prompts.
# Look for improved instruction-following after LoRA fine-tuning.
test_prompts = [
 "Explain photosynthesis to a 10-year-old.",
 "Write a short email declining a meeting politely.",
 "List 3 advantages of renewable energy.",
]

for prompt in test_prompts:
 messages = [{"role": "user", "content": prompt}]
 fmt = tokenizer.apply_chat_template(
 messages, tokenize=False, add_generation_prompt=True)
 inputs = tokenizer(fmt, return_tensors="pt").to(merged_model.device)
 with torch.no_grad():
 outputs = merged_model.generate(
 **inputs, max_new_tokens=200, temperature=0.7, do_sample=True)
 response = tokenizer.decode(
 outputs[0][inputs['input_ids'].shape[1]:], skip_special_tokens=True)
 print(f"Prompt: {prompt}\nResponse: {response[:300]}\n{'-'*50}")

# Complete LoRA lab solution: load model, apply adapter, train,
# save/merge, and compare outputs before and after fine-tuning.
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import LoraConfig, get_peft_model, PeftModel, TaskType
from trl import SFTTrainer, SFTConfig
from datasets import load_dataset
import torch, pandas as pd, os

model_name = "HuggingFaceTB/SmolLM2-135M-Instruct"
tokenizer = AutoTokenizer.from_pretrained(model_name)
if tokenizer.pad_token is None: tokenizer.pad_token = tokenizer.eos_token

# Step 1: Apply LoRA
base_model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.float16, device_map="auto")
lora_config = LoraConfig(r=16, lora_alpha=32, target_modules=["q_proj","v_proj"],
 task_type=TaskType.CAUSAL_LM, lora_dropout=0.05, bias="none")
model = get_peft_model(base_model, lora_config)
model.print_trainable_parameters()

# Step 2: Compare ranks
for r in [4, 8, 16, 32, 64]:
 b = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.float16, device_map="auto")
 c = LoraConfig(r=r, lora_alpha=r*2, target_modules=["q_proj","v_proj"],
 task_type=TaskType.CAUSAL_LM, lora_dropout=0.05, bias="none")
 pm = get_peft_model(b, c)
 t = sum(p.numel() for p in pm.parameters() if p.requires_grad)
 a = sum(p.numel() for p in pm.parameters())
 print(f"r={r}: {t:,} trainable / {a:,} total = {t/a*100:.4f}%")
 del pm, b; torch.cuda.empty_cache()

# Step 3: Train
ds = load_dataset("HuggingFaceH4/no_robots", split="train").shuffle(seed=42).select(range(300))
def fmt(ex): return {"text": tokenizer.apply_chat_template(ex["messages"], tokenize=False, add_generation_prompt=False)}
formatted = ds.map(fmt)

args = SFTConfig(output_dir="./lora-smollm2-r16", num_train_epochs=3, per_device_train_batch_size=4,
 gradient_accumulation_steps=2, learning_rate=2e-4, logging_steps=10, max_seq_length=512,
 fp16=True, warmup_ratio=0.1, save_strategy="epoch", report_to="none")
trainer = SFTTrainer(model=model, args=args, train_dataset=formatted, processing_class=tokenizer)
trainer.train()

# Step 4: Save and merge
model.save_pretrained("./lora-adapter-r16")
base_fresh = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.float16, device_map="auto")
merged = PeftModel.from_pretrained(base_fresh, "./lora-adapter-r16").merge_and_unload()

# Step 5: Test
for p in ["Explain photosynthesis to a 10-year-old.", "Write a short email declining a meeting.", "List 3 advantages of renewable energy."]:
 msgs = [{"role":"user","content":p}]
 f = tokenizer.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
 i = tokenizer(f, return_tensors="pt").to(merged.device)
 with torch.no_grad():
 o = merged.generate(**i, max_new_tokens=200, temperature=0.7, do_sample=True)
 print(f"Q: {p}\nA: {tokenizer.decode(o[0][i['input_ids'].shape[1]:], skip_special_tokens=True)[:300]}\n")