Advanced PEFT Methods

"One adapter is good. Six adapters, each specialized and hot-swappable, is a production strategy."

LoRA, Hot-Swapping AI Agent

17.2.1 DoRA: Weight-Decomposed Low-Rank Adaptation

DoRA (2024) improves on LoRA by decomposing the pretrained weight into its magnitude and direction components before applying the low-rank update. Specifically, a weight vector w is decomposed as w = m · (w / ||w||), where m is a learnable magnitude scalar and the direction is updated via standard LoRA. This decomposition aligns more closely with how full fine-tuning actually modifies weights, resulting in better performance for the same rank.

Think of pointing a flashlight: LoRA must learn both which direction to aim and how bright the beam should be using the same dial. If you want to brighten the beam without rotating it, LoRA still has to use some of its low-rank capacity to keep the angle steady, wasting parameters. DoRA gives you two dials: one for the angle (low-rank, like LoRA) and a separate scalar for the brightness. Now the rank budget can spend all its capacity on rotation, and the magnitude scalar adjusts intensity independently. That separation is why DoRA at rank 16 typically matches LoRA at rank 32 with fewer trainable parameters.

The original DoRA paper (Liu et al., 2024) reports gains of 1-3% over LoRA across commonsense reasoning, visual-instruction, and image-text understanding tasks at matched rank and target modules, with only a marginal increase in trainable parameters (the additional magnitude vectors are tiny). Independent replications find the gap is task-dependent and sometimes within noise; the safest claim is "DoRA dominates LoRA on the original benchmark suite and is rarely worse," not that it always wins. Training speed is nearly identical to LoRA. For deployment, the quantization techniques covered later in the chapter apply equally to DoRA-adapted models; the diagram below compares the LoRA and DoRA weight-update mechanisms side by side.

Figure 17.2.1: DoRA separates weight magnitude (m) from direction, applying LoRA only to the directional component.

The following implementation (Code Fragment 17.2.3) shows how to enable DoRA with a single configuration flag.

pip install bitsandbytes peft transformers trl
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
import torch
bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
                         bnb_4bit_compute_dtype=torch.bfloat16,
                         bnb_4bit_use_double_quant=True)
base = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.1-70B-Instruct",
                                            quantization_config=bnb, device_map="auto")
base = prepare_model_for_kbit_training(base)
model = get_peft_model(base, LoraConfig(r=16, lora_alpha=32,
                                        target_modules="all-linear",
                                        task_type="CAUSAL_LM"))

# Configure QLoRA adapter parameters on top of the quantized base
# The adapter trains in float16 while the base stays in 4-bit NormalFloat
from peft import LoraConfig, get_peft_model

# DoRA configuration: enable the use_dora flag
dora_config = LoraConfig(
 r=16,
 lora_alpha=32,
 target_modules=[
 "q_proj", "k_proj", "v_proj", "o_proj",
 "gate_proj", "up_proj", "down_proj",
 ],
 lora_dropout=0.05,
 use_dora=True, # Enable DoRA
 task_type="CAUSAL_LM",
)

model = get_peft_model(base_model, dora_config)
model.print_trainable_parameters()
# Slightly more params than LoRA due to magnitude vectors,
# but typically ~1-3% better accuracy at same rank.

Code Fragment 17.2.6 shows how Prefix Tuning prepends learnable vectors to each attention layer.

# Prefix Tuning: prepend learnable key-value vectors to each attention layer
# The model attends to these "virtual tokens" alongside real input tokens
from peft import PrefixTuningConfig, get_peft_model, TaskType

# Prefix Tuning configuration
prefix_config = PrefixTuningConfig(
 task_type=TaskType.CAUSAL_LM,
 num_virtual_tokens=30, # Number of prefix tokens
 prefix_projection=True, # Use MLP to project prefix (more stable)
 encoder_hidden_size=1024, # Hidden size of projection MLP
)

# Wrap base model with trainable LoRA adapters
model = get_peft_model(base_model, prefix_config)
model.print_trainable_parameters()
# Typically 0.1-0.5% of total parameters

A bottleneck adapter is the small trainable module Houlsby et al. (2019) insert after the feed-forward (and sometimes after the attention) sub-layer of each transformer block. Given a hidden state $h \in \mathbb{R}^d$, the adapter down-projects to a small bottleneck dimension $r \ll d$, applies a non-linearity, projects back to $d$, and adds the result residually:

$$h' = h + W_{\text{up}}\, \sigma\!\big(W_{\text{down}}\, h\big), \qquad W_{\text{down}} \in \mathbb{R}^{r \times d},\ W_{\text{up}} \in \mathbb{R}^{d \times r}$$

The down-up sandwich is initialised so that $W_{\text{up}} \approx 0$, which makes the adapter the identity at the start of training; gradients then carve out a per-task perturbation in the $2 d r$ parameters of $(W_{\text{down}}, W_{\text{up}})$ while the base weights stay frozen. For a 7B model with $d = 4096$ and $r = 16$, each adapter holds $\approx 131$k parameters, so a full set across 32 layers costs roughly 0.06% of the base. Code Fragment 17.2.5 demonstrates the bottleneck adapter pattern using LLaMA-Adapter style chapters, and Code Fragment 17.2.4b shows the authentic Houlsby-style configuration via the adapters library.

# Houlsby-style bottleneck adapters via the adapters library
# (the modern successor to adapter-transformers / AdapterHub).
from adapters import AutoAdapterModel, SeqBnConfig
from transformers import AutoTokenizer

model = AutoAdapterModel.from_pretrained("bert-base-uncased")
tok = AutoTokenizer.from_pretrained("bert-base-uncased")

# Sequential bottleneck: W_down (d -> r), ReLU, W_up (r -> d), residual add.
adapter_cfg = SeqBnConfig(reduction_factor=16, non_linearity="relu")
model.add_adapter("legal-clauses", config=adapter_cfg)
model.add_classification_head("legal-clauses", num_labels=3)
model.train_adapter("legal-clauses")   # freezes base, trains only the bottleneck

# Standard HF Trainer loop trains ~0.9M params on a 110M-param BERT.

# Adapter layers: insert small bottleneck modules between transformer layers
# Uses LLaMA-Adapter style via PEFT's AdaptionPromptConfig
from peft import AdaptionPromptConfig, get_peft_model

# Note: For bottleneck adapters, use the adapters library
# from adapters import AutoAdapterModel

# Example with LLaMA-Adapter style (via PEFT)
adapter_config = AdaptionPromptConfig(
 adapter_len=10, # Length of adapter prompt
 adapter_layers=30, # Number of layers to add adapters
 task_type="CAUSAL_LM",
)

model = get_peft_model(base_model, adapter_config)
model.print_trainable_parameters()

17.2.2 IA3: Infused Adapter by Inhibiting and Amplifying Inner Activations

IA3 (few-shot parameter-efficient fine-tuning Is All You Need, 2022) takes parameter efficiency to the extreme. Instead of learning new matrices or inserting new layers, IA3 learns only three rescaling vectors that modulate the keys, values, and intermediate activations in the Transformer architecture. The total number of trainable parameters is typically 10x smaller than LoRA.

The tradeoff is that IA3's limited capacity makes it best suited for simple adaptation tasks (format changes, style transfer) rather than complex domain adaptation. It excels in few-shot settings where overfitting is a concern. Code Fragment 17.2.6a demonstrates IA3 configuration.

# IA3 configuration: learns only rescaling vectors
# Extreme parameter efficiency at the cost of limited adaptation capacity
from peft import IA3Config, get_peft_model, TaskType

ia3_config = IA3Config(
 task_type=TaskType.CAUSAL_LM,
 target_modules=["k_proj", "v_proj", "down_proj"],
 feedforward_modules=["down_proj"],
)

# Wrap base model with IA3 rescaling vectors
model = get_peft_model(base_model, ia3_config)
model.print_trainable_parameters()
# trainable params: ~500K for a 7B model (0.007%)

17.2.3 Comprehensive PEFT Method Comparison

Why this matters: The proliferation of PEFT methods (DoRA, AdaLoRA, IA3, GaLore) is not just academic variety; each addresses a specific limitation of vanilla LoRA. DoRA handles magnitude-direction decomposition for better convergence. AdaLoRA allocates rank adaptively, putting more parameters where they help most. IA3 achieves extreme parameter efficiency (orders of magnitude fewer parameters than LoRA) at the cost of some task performance. The practical guidance is: start with standard LoRA, and only explore these variants when you have a specific bottleneck (convergence issues, memory constraints, or multi-task serving requirements).

17.2.4 Multi-Adapter Serving

One of LoRA's most powerful production features is the ability to serve many adapters from a single base model. This enables multi-tenant deployments where each customer or task gets its own fine-tuned behavior without duplicating the base model weights, a significant advantage for inference optimization. Two main systems support this at scale: LoRAX (formerly LoRAX by Predibase) and S-LoRA.

17.2.4.1 Architecture Overview

Figure 17.2.2a illustrates how a single base model serves multiple LoRA adapters dynamically at request time.

Figure 17.2.2b: Multi-adapter serving loads one base model and dynamically applies per-request LoRA adapters.

17.2.4.2 LoRAX and S-LoRA

LoRAX (Predibase) is a production-grade serving system that can host hundreds of fine-tuned LoRA adapters on a single GPU. It keeps the base model in GPU memory and dynamically loads adapter weights per request. Key features include adapter weight caching, batched inference across different adapters, and automatic adapter management.

S-LoRA (from UC Berkeley) takes a more research-oriented approach, using unified paging to manage adapter memory and custom CUDA kernels for batched LoRA computation. S-LoRA can serve thousands of adapters simultaneously, with adapters stored in a tiered memory system (GPU, CPU, disk) and paged in on demand.

17.2.5 Choosing the Right PEFT Method

With so many PEFT options available, the decision can feel overwhelming. Here is a practical decision framework based on your constraints and requirements.

17.2.6 GaLore: Gradient Low-Rank Projection

While LoRA reduces the number of trainable parameters by adding low-rank adapters, GaLore (Gradient Low-Rank Projection) takes a fundamentally different approach: it reduces optimizer memory by projecting gradients into a low-rank subspace. This distinction matters because optimizer states (momentum and variance in Adam) typically consume two to three times the memory of the parameters themselves. GaLore enables full-parameter training of large models on consumer hardware, something that LoRA alone cannot achieve because LoRA still freezes the base model and only updates the adapter.

17.2.6.1 How GaLore Works

During training, GaLore periodically computes the SVD of the gradient matrix for each weight layer, retaining only the top-r singular vectors. The optimizer states (Adam's first and second moments) are maintained in this reduced r-dimensional space rather than the full parameter space. Every T steps (typically T=200), the projection matrices are recomputed from the current gradient to track the evolving optimization landscape. The key insight is that gradient matrices during LLM training tend to be approximately low-rank, so very little information is lost by this projection. Code Fragment 17.2.7 provides a conceptual implementation.

# GaLore conceptual implementation
import torch

class GaLoreProjector:
    """Project gradients to low-rank subspace for memory-efficient training."""
    def __init__(self, rank: int, update_freq: int = 200):
        self.rank = rank
        self.update_freq = update_freq
        self.step = 0
        self.projector = None

    def project(self, grad: torch.Tensor) -> torch.Tensor:
        """Project full gradient to low-rank subspace."""
        if self.step % self.update_freq == 0:
            # Recompute projection via SVD
            U, S, Vh = torch.linalg.svd(grad, full_matrices=False)
            self.projector = U[:, :self.rank]
        self.step += 1
        # Project gradient: (d, r) @ (r, d) is never formed explicitly
        return self.projector.T @ grad  # shape: (rank, d_out)

    def back_project(self, low_rank_update: torch.Tensor) -> torch.Tensor:
        """Map low-rank update back to full parameter space."""
        return self.projector @ low_rank_update

In practice, the galore_torch library wraps this into a drop-in optimizer replacement:

# Library shortcut: GaLore optimizer (pip install galore-torch)
from galore_torch import GaLoreAdamW8bit
optimizer = GaLoreAdamW8bit(
    model.parameters(),
    lr=1e-5,
    rank=128, # low-rank projection dimension
    update_proj_gap=200, # recompute SVD every 200 steps
)
# Use this optimizer in any standard training loop; no other changes needed.

The memory savings are substantial. For a 7B parameter model, standard Adam requires roughly 42 GB of optimizer state memory (two copies of all parameters in float32). With GaLore at rank 128, the optimizer states shrink to approximately 2 to 4 GB, enabling full-parameter training of 7B models on a single 24 GB GPU. The authors demonstrated that GaLore can pretrain LLaMA models up to 7B parameters on a single GPU with no loss in quality compared to full-rank Adam.

17.2.7 rsLoRA: Rank-Stabilized LoRA

Standard LoRA initializes the low-rank matrices A and B such that the adapter output is scaled by a fixed factor α/r, where α is a hyperparameter and r is the rank. This scaling creates a practical problem: when you change the rank, the effective magnitude of the adapter's contribution changes, requiring you to re-tune the learning rate and α for each rank setting. This makes rank selection tedious and error-prone.

rsLoRA (rank-stabilized LoRA) addresses this by changing the scaling factor from α/r to α/√r. This seemingly small modification has a significant theoretical and practical impact. The √r scaling ensures that the adapter's output magnitude remains stable as the rank changes, because the variance of the product BA scales proportionally with r under random initialization. With α/√r scaling, doubling the rank does not double the adapter's contribution; it increases it by only √2, which is the correct normalization for maintaining stable training dynamics. Code Fragment 17.2.7b compares the two scaling approaches.

# rsLoRA vs standard LoRA scaling comparison
import torch
import math
def lora_forward(x, A, B, alpha, rank, use_rslora=False):
    """Compare standard LoRA and rsLoRA scaling."""
    lora_output = x @ A @ B # shape: (batch, d_out)
    if use_rslora:
        # rsLoRA: scale by alpha / sqrt(rank)
        scaling = alpha / math.sqrt(rank)
    else:
        # Standard LoRA: scale by alpha / rank
        scaling = alpha / rank
    return lora_output * scaling

# Demonstrate stability across ranks
d_in, d_out, alpha = 4096, 4096, 16.0
x = torch.randn(1, d_in)
for rank in [4, 16, 64, 256]:
    A = torch.randn(d_in, rank) * 0.01
    B = torch.randn(rank, d_out) * 0.01
    std_out = lora_forward(x, A, B, alpha, rank, use_rslora=False)
    rs_out = lora_forward(x, A, B, alpha, rank, use_rslora=True)
    print(f"rank={rank:3d} standard_norm={std_out.norm():.4f}"
        f" rslora_norm={rs_out.norm():.4f}")

The practical benefit is straightforward: with rsLoRA, you can change the rank without retuning other hyperparameters. A learning rate and α that work well at rank 16 will also work well at rank 64 or rank 256. This makes hyperparameter search much faster (complementing the training loop fundamentals from Section 16.3), because you can tune the learning rate at a low rank (which trains quickly) and then scale up the rank for higher quality without adjusting anything else. rsLoRA is available in PEFT via the use_rslora=True parameter in LoraConfig.