Advanced PEFT Methods

"One adapter is good. Six adapters, each specialized and hot-swappable, is a production strategy."

LoRA, Hot-Swapping AI Agent

1. DoRA: Weight-Decomposed Low-Rank Adaptation

DoRA (2024) improves on LoRA by decomposing the pretrained weight into its magnitude and direction components before applying the low-rank update. Specifically, a weight vector w is decomposed as w = m · (w / ||w||), where m is a learnable magnitude scalar and the direction is updated via standard LoRA. This decomposition aligns more closely with how full fine-tuning actually modifies weights, resulting in better performance for the same rank.

In practice, DoRA consistently outperforms LoRA by 1-3% across benchmarks when using the same rank and target modules, with only a marginal increase in trainable parameters (the additional magnitude vectors are tiny). The training speed is nearly identical to LoRA. For deployment considerations, the Section 09.1 apply equally to DoRA-adapted models. Figure 15.2.1 compares the LoRA and DoRA weight update mechanisms.

The following implementation (Code Fragment 15.2.2) shows how to enable DoRA with a single configuration flag.

# Configure QLoRA adapter parameters on top of the quantized base
# The adapter trains in float16 while the base stays in 4-bit NormalFloat
from peft import LoraConfig, get_peft_model

# DoRA configuration: enable the use_dora flag
dora_config = LoraConfig(
 r=16,
 lora_alpha=32,
 target_modules=[
 "q_proj", "k_proj", "v_proj", "o_proj",
 "gate_proj", "up_proj", "down_proj",
 ],
 lora_dropout=0.05,
 use_dora=True, # Enable DoRA
 task_type="CAUSAL_LM",
)

model = get_peft_model(base_model, dora_config)
model.print_trainable_parameters()
# Slightly more params than LoRA due to magnitude vectors,
# but typically ~1-3% better accuracy at same rank.

Code Fragment 15.2.5 shows how Prefix Tuning prepends learnable vectors to each attention layer.

# Prefix Tuning: prepend learnable key-value vectors to each attention layer
# The model attends to these "virtual tokens" alongside real input tokens
from peft import PrefixTuningConfig, get_peft_model, TaskType

# Prefix Tuning configuration
prefix_config = PrefixTuningConfig(
 task_type=TaskType.CAUSAL_LM,
 num_virtual_tokens=30, # Number of prefix tokens
 prefix_projection=True, # Use MLP to project prefix (more stable)
 encoder_hidden_size=1024, # Hidden size of projection MLP
)

# Wrap base model with trainable LoRA adapters
model = get_peft_model(base_model, prefix_config)
model.print_trainable_parameters()
# Typically 0.1-0.5% of total parameters

Code Fragment 15.2.4 demonstrates the bottleneck adapter pattern using LLaMA-Adapter style chapters.

# Adapter layers: insert small bottleneck modules between transformer layers
# Uses LLaMA-Adapter style via PEFT's AdaptionPromptConfig
from peft import AdaptionPromptConfig, get_peft_model

# Note: For bottleneck adapters, use the adapters library
# from adapters import AutoAdapterModel

# Example with LLaMA-Adapter style (via PEFT)
adapter_config = AdaptionPromptConfig(
 adapter_len=10, # Length of adapter prompt
 adapter_layers=30, # Number of layers to add adapters
 task_type="CAUSAL_LM",
)

model = get_peft_model(base_model, adapter_config)
model.print_trainable_parameters()

5. IA3: Infused Adapter by Inhibiting and Amplifying Inner Activations

IA3 (few-shot parameter-efficient fine-tuning Is All You Need, 2022) takes parameter efficiency to the extreme. Instead of learning new matrices or inserting new layers, IA3 learns only three rescaling vectors that modulate the keys, values, and intermediate activations in the feedforward layers. The total number of trainable parameters is typically 10x smaller than LoRA.

The tradeoff is that IA3's limited capacity makes it best suited for simple adaptation tasks (format changes, style transfer) rather than complex domain adaptation. It excels in few-shot settings where overfitting is a concern. Code Fragment 15.2.5 demonstrates IA3 configuration.

# IA3 configuration: learns only rescaling vectors
# Extreme parameter efficiency at the cost of limited adaptation capacity
from peft import IA3Config, get_peft_model, TaskType

ia3_config = IA3Config(
 task_type=TaskType.CAUSAL_LM,
 target_modules=["k_proj", "v_proj", "down_proj"],
 feedforward_modules=["down_proj"],
)

# Wrap base model with IA3 rescaling vectors
model = get_peft_model(base_model, ia3_config)
model.print_trainable_parameters()
# trainable params: ~500K for a 7B model (0.007%)

6. Comprehensive PEFT Method Comparison

Why this matters: The proliferation of PEFT methods (DoRA, AdaLoRA, IA3, GaLore) is not just academic variety; each addresses a specific limitation of vanilla LoRA. DoRA handles magnitude-direction decomposition for better convergence. AdaLoRA allocates rank adaptively, putting more parameters where they help most. IA3 achieves extreme parameter efficiency (orders of magnitude fewer parameters than LoRA) at the cost of some task performance. The practical guidance is: start with standard LoRA, and only explore these variants when you have a specific bottleneck (convergence issues, memory constraints, or multi-task serving requirements).

7. Multi-Adapter Serving

One of LoRA's most powerful production features is the ability to serve many adapters from a single base model. This enables multi-tenant deployments where each customer or task gets its own fine-tuned behavior without duplicating the base model weights, a significant advantage for inference optimization. Two main systems support this at scale: LoRAX (formerly LoRAX by Predibase) and S-LoRA.

7.1 Architecture Overview

Figure 15.2.3 illustrates how a single base model serves multiple LoRA adapters dynamically at request time.

7.2 LoRAX and S-LoRA

LoRAX (Predibase) is a production-grade serving system that can host hundreds of fine-tuned LoRA adapters on a single GPU. It keeps the base model in GPU memory and dynamically loads adapter weights per request. Key features include adapter weight caching, batched inference across different adapters, and automatic adapter management.

S-LoRA (from UC Berkeley) takes a more research-oriented approach, using unified paging to manage adapter memory and custom CUDA kernels for batched LoRA computation. S-LoRA can serve thousands of adapters simultaneously, with adapters stored in a tiered memory system (GPU, CPU, disk) and paged in on demand.

8. Choosing the Right PEFT Method

With so many PEFT options available, the decision can feel overwhelming. Here is a practical decision framework based on your constraints and requirements.

9. GaLore: Gradient Low-Rank Projection

While LoRA reduces the number of trainable parameters by adding low-rank adapters, GaLore (Gradient Low-Rank Projection) takes a fundamentally different approach: it reduces optimizer memory by projecting gradients into a low-rank subspace. This distinction matters because optimizer states (momentum and variance in Adam) typically consume two to three times the memory of the parameters themselves. GaLore enables full-parameter training of large models on consumer hardware, something that LoRA alone cannot achieve because LoRA still freezes the base model and only updates the adapter.

9.1 How GaLore Works

During training, GaLore periodically computes the SVD of the gradient matrix for each weight layer, retaining only the top-r singular vectors. The optimizer states (Adam's first and second moments) are maintained in this reduced r-dimensional space rather than the full parameter space. Every T steps (typically T=200), the projection matrices are recomputed from the current gradient to track the evolving optimization landscape. The key insight is that gradient matrices during LLM training tend to be approximately low-rank, so very little information is lost by this projection. Code Fragment 15.2.6 provides a conceptual implementation.

# GaLore conceptual implementation
import torch

class GaLoreProjector:
 """Project gradients to low-rank subspace for memory-efficient training."""

 def __init__(self, rank: int, update_freq: int = 200):
 self.rank = rank
 self.update_freq = update_freq
 self.step = 0
 self.projector = None

 def project(self, grad: torch.Tensor) -> torch.Tensor:
 """Project full gradient to low-rank subspace."""
 if self.step % self.update_freq == 0:
 # Recompute projection via SVD
 U, S, Vh = torch.linalg.svd(grad, full_matrices=False)
 self.projector = U[:, :self.rank]

 self.step += 1
 # Project gradient: (d, r) @ (r, d) is never formed explicitly
 return self.projector.T @ grad # shape: (rank, d_out)

 def back_project(self, low_rank_update: torch.Tensor) -> torch.Tensor:
 """Map low-rank update back to full parameter space."""
 return self.projector @ low_rank_update

In practice, the galore_torch library wraps this into a drop-in optimizer replacement:

# Library shortcut: GaLore optimizer (pip install galore-torch)
from galore_torch import GaLoreAdamW8bit

optimizer = GaLoreAdamW8bit(
 model.parameters(),
 lr=1e-5,
 rank=128, # low-rank projection dimension
 update_proj_gap=200, # recompute SVD every 200 steps
)
# Use this optimizer in any standard training loop; no other changes needed.

The memory savings are substantial. For a 7B parameter model, standard Adam requires roughly 42 GB of optimizer state memory (two copies of all parameters in float32). With GaLore at rank 128, the optimizer states shrink to approximately 2 to 4 GB, enabling full-parameter training of 7B models on a single 24 GB GPU. The authors demonstrated that GaLore can pre-train LLaMA models up to 7B parameters on a single GPU with no loss in quality compared to full-rank Adam.

10. rsLoRA: Rank-Stabilized LoRA

Standard LoRA initializes the low-rank matrices A and B such that the adapter output is scaled by a fixed factor α/r, where α is a hyperparameter and r is the rank. This scaling creates a practical problem: when you change the rank, the effective magnitude of the adapter's contribution changes, requiring you to re-tune the learning rate and α for each rank setting. This makes rank selection tedious and error-prone.

rsLoRA (rank-stabilized LoRA) addresses this by changing the scaling factor from α/r to α/√r. This seemingly small modification has a significant theoretical and practical impact. The √r scaling ensures that the adapter's output magnitude remains stable as the rank changes, because the variance of the product BA scales proportionally with r under random initialization. With α/√r scaling, doubling the rank does not double the adapter's contribution; it increases it by only √2, which is the correct normalization for maintaining stable training dynamics. Code Fragment 15.2.6 compares the two scaling approaches.

# rsLoRA vs standard LoRA scaling comparison
import torch
import math

def lora_forward(x, A, B, alpha, rank, use_rslora=False):
 """Compare standard LoRA and rsLoRA scaling."""
 lora_output = x @ A @ B # shape: (batch, d_out)

 if use_rslora:
 # rsLoRA: scale by alpha / sqrt(rank)
 scaling = alpha / math.sqrt(rank)
 else:
 # Standard LoRA: scale by alpha / rank
 scaling = alpha / rank

 return lora_output * scaling

# Demonstrate stability across ranks
d_in, d_out, alpha = 4096, 4096, 16.0
x = torch.randn(1, d_in)

for rank in [4, 16, 64, 256]:
 A = torch.randn(d_in, rank) * 0.01
 B = torch.randn(rank, d_out) * 0.01

 std_out = lora_forward(x, A, B, alpha, rank, use_rslora=False)
 rs_out = lora_forward(x, A, B, alpha, rank, use_rslora=True)

 print(f"rank={rank:3d} standard_norm={std_out.norm():.4f}"
 f" rslora_norm={rs_out.norm():.4f}")

The practical benefit is straightforward: with rsLoRA, you can change the rank without retuning other hyperparameters. A learning rate and α that work well at rank 16 will also work well at rank 64 or rank 256. This makes hyperparameter search much faster (complementing the training loop fundamentals from Section 14.3), because you can tune the learning rate at a low rank (which trains quickly) and then scale up the rank for higher quality without adjusting anything else. rsLoRA is available in PEFT via the use_rslora=True parameter in LoraConfig.