Adaptation, Quantization & Evaluation

You have a pre-trained model. Three practical questions come next: How do I make it good at my task? How do I make it small enough to run? And how do I know if it is actually good? This session answers all three.

01Transfer learning and domain adaptation

Transfer learning means reusing what a model learned on general text for a new task. You don’t start from zero; you start from a model that already knows language.

02Prompting vs RAG vs PEFT vs full fine-tuning

This is one of the most common questions in real projects. Use this matrix:

03Quantization: making models smaller

Model weights are normally stored as 16-bit floating-point numbers. Quantization stores them with fewer bits, like 8 or 4, using a scale factor. The model becomes 2–4× smaller and often faster, with a small loss in quality.

# pip install transformers accelerate bitsandbytes
import torch
from transformers import AutoModelForCausalLM, BitsAndBytesConfig

bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
                         bnb_4bit_compute_dtype=torch.float16)
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-7B-Instruct",
                                             quantization_config=bnb, device_map="auto")
print(f"{model.get_memory_footprint() / 1e9:.1f} GB")   # roughly 5 GB instead of ~15 GB

04Evaluation metrics

Perplexity: how surprised is the model?

Perplexity is the exponential of the average negative log-probability the model gave to the real tokens. Lower is better. A perplexity of 2 roughly means the model was, on average, choosing between 2 equally likely options.

BLEU, ROUGE and METEOR: comparing with a reference

Where automatic metrics mislead

These metrics only count matching words. Look at this real calculation. The reference is “The card has been blocked for your safety.”

This is why teams always add human evaluation: people rate outputs for correctness, helpfulness, fluency and safety, often by comparing two answers side by side. A cheaper middle path is LLM-as-a-judge, where a strong model grades answers using a clear rubric, checked regularly against human ratings.

# pip install rouge-score sacrebleu
from rouge_score import rouge_scorer
import sacrebleu

ref = "The card has been blocked for your safety."
cands = ["Your card is now blocked for safety.",
         "The card has been blocked for your account."]

scorer = rouge_scorer.RougeScorer(["rouge1", "rouge2", "rougeL"], use_stemmer=True)
for c in cands:
    r = scorer.score(ref, c)
    bleu = sacrebleu.sentence_bleu(c, [ref]).score
    print(c, {k: round(v.fmeasure, 2) for k, v in r.items()}, "BLEU", round(bleu, 1))

Explainability

To understand why a model gave an answer, teams use attention maps (Session 19), feature-attribution tools like SHAP, LIME or integrated gradients that score how much each input word mattered, and for LLMs, asking for citations to retrieved sources (RAG). In regulated fields like banking, explanations and audit logs are often a requirement.

05Summary

• Transfer learning reuses a pre-trained model; domain adaptation teaches it your field’s language.
• Start with prompting, add RAG for knowledge, use PEFT for behaviour, and full fine-tuning only for big changes.
• Quantization shrinks models 2–4×; INT8 is nearly lossless, INT4 trades a little quality; GGUF runs models on laptops.
• Perplexity measures surprise; BLEU/ROUGE/METEOR measure word overlap and can mislead, so add human or LLM-judge evaluation.

06Quick quiz

07References

• Dettmers et al., LLM.int8() (2022) · QLoRA (2023)
• Papineni et al., BLEU (2002) · Lin, ROUGE (2004)
• llama.cpp and GGUF