PEFT Hands-On

This is the final session of the course, and the most hands-on. Let us understand exactly what LoRA changes inside the model, how QLoRA squeezes fine-tuning onto a single GPU, and then run a complete fine-tune and prove it beats the base model.

01LoRA: how it works

A layer in an LLM multiplies its input by a big weight matrix W (for example 4096 × 4096 ≈ 16.8 million numbers). LoRA freezes W and learns a small correction made of two thin matrices, B and A:

02How many parameters does LoRA train?

For a Llama-2-7B-style model (32 layers, hidden size 4096) with adapters on q_proj and v_proj:

Formula: 32 layers × 2 matrices × r × (4096 + 4096). Even at r = 64 you train under 0.5% of the weights.

03QLoRA: fine-tuning on one GPU

QLoRA (2023) loads the frozen base model in 4-bit precision and trains LoRA adapters on top in higher precision. Its key tricks:

• 4-bit NormalFloat (NF4): a 4-bit format designed for the bell-shaped distribution of neural-network weights.
• Double quantization: also compresses the quantization constants, saving more memory.
• Paged optimizers: move optimizer state to CPU memory during spikes to avoid out-of-memory crashes.

04A complete fine-tune with PEFT + TRL

This is the standard recipe with Hugging Face libraries. Run it on a GPU (for example Google Colab); replace the model and data with your own.

# pip install transformers peft trl bitsandbytes datasets accelerate
import torch
from datasets import load_dataset
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import LoraConfig
from trl import SFTTrainer, SFTConfig

base = "meta-llama/Llama-3.1-8B-Instruct"                 # any causal LM you have access to
bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
                         bnb_4bit_use_double_quant=True, bnb_4bit_compute_dtype=torch.bfloat16)
model = AutoModelForCausalLM.from_pretrained(base, quantization_config=bnb, device_map="auto")
tok = AutoTokenizer.from_pretrained(base)

lora = LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05, task_type="CAUSAL_LM",
                  target_modules=["q_proj", "k_proj", "v_proj", "o_proj"])

data = load_dataset("json", data_files={"train": "train.jsonl", "test": "test.jsonl"})   # chat "messages" format

args = SFTConfig(output_dir="ticket-classifier-lora", num_train_epochs=3, learning_rate=2e-4,
                 per_device_train_batch_size=4, gradient_accumulation_steps=4,
                 logging_steps=20, eval_strategy="epoch", bf16=True)

trainer = SFTTrainer(model=model, args=args, train_dataset=data["train"], eval_dataset=data["test"],
                     peft_config=lora, processing_class=tok)
trainer.train()
trainer.save_model("ticket-classifier-lora")             # saves only the small adapter

05Evaluate against the base model

A fine-tune is only worth it if it beats the base model on data it has never seen. Use the held-out test set from Session 39:

Merge and serve

import torch
from peft import AutoPeftModelForCausalLM

model = AutoPeftModelForCausalLM.from_pretrained("ticket-classifier-lora", torch_dtype=torch.bfloat16)
merged = model.merge_and_unload()                         # fold B·A into W: no extra latency
merged.save_pretrained("ticket-classifier-merged")

Or keep adapters separate and load the right one per request. Serving engines like vLLM can serve many LoRA adapters on one base model (recall the “swap adapters” idea from the GenAI lessons).

06Summary

• LoRA freezes W and learns a low-rank update B·A, usually under 1% of the weights.
• Rank, alpha and target modules are the main knobs.
• QLoRA = 4-bit NF4 base + LoRA adapters, so big models fit on one GPU.
• Train with PEFT + TRL SFTTrainer; always compare with the base model on a held-out set.
• Merge adapters for simple serving, or keep them separate to swap per task.

07Quick quiz

08References

• LoRA: Low-Rank Adaptation of Large Language Models (Hu et al., 2021)
• QLoRA: Efficient Finetuning of Quantized LLMs (Dettmers et al., 2023)
• Hugging Face TRL SFTTrainer
• Hugging Face PEFT