GenAI Course · Module 6 · Session 22

Transformer Variants

⏱ 12 min read🎯 Intermediate🖼 2 diagrams🧪 1 live demo🐍 Python code

The original transformer had both an encoder and a decoder. Since then, people found you can also use only one half. Today almost every model belongs to one of three families. Knowing which family fits which job is one of the most practical skills in GenAI.

In this session

The three families: encoder-only, decoder-only, encoder-decoder
Key models in each family and what they are good at
T5’s text-to-text idea and BART’s denoising idea
A quick task → family chooser

01Three families

One architecture, three ways to use it.

Family	Reads text	Trained to	Best at
Encoder-only	both directions	fill in masked words	understanding: classify, extract, search
Decoder-only	left to right	predict the next word	generating: chat, write, code, reasoning
Encoder-decoder	encoder both ways, decoder left to right	turn one text into another	translation, summarisation, rewriting

02Encoder-only: BERT and friends

These models read the full input at once and produce a rich vector for every token. They do not write long text, but they are small, fast and excellent at understanding. We covered them in detail in the BERT lesson.

Model	Key idea
BERT (2018)	masked language modelling + next sentence prediction
RoBERTa (2019)	no NSP, more data, longer training
DistilBERT (2019)	40% smaller, 60% faster, keeps ~97% accuracy
DeBERTa (2020)	separate content and position in attention

03Decoder-only: the GPT family

Drop the encoder and cross-attention, keep masked self-attention, and train on one simple task: predict the next token. Scale this up with huge data, and you get today’s LLMs.

Same basic design, much bigger models and data, plus instruction tuning and RLHF.

Why did decoder-only win for chat? One model, one training objective, and it can do almost any task when you describe it in a prompt. That flexibility, at scale, beat specialised designs for general assistants.

04Encoder-decoder: T5 and BART

T5 (Google, 2019) treats every task as text in → text out. You simply add a prefix: translate English to German: …, summarize: …. One model, many tasks.
BART (Facebook, 2019) is trained to repair damaged text: words are masked, deleted or shuffled, and the model must rebuild the original. This makes it very good at summarisation.

three_families.py

from transformers import pipeline

fill = pipeline("fill-mask", model="distilbert-base-uncased")                 # encoder-only
gen  = pipeline("text-generation", model="gpt2")                               # decoder-only
t5   = pipeline("text2text-generation", model="t5-small")                       # encoder-decoder

print(fill("Mumbai is a [MASK] city.")[0]["token_str"])
print(gen("Learning GenAI is", max_new_tokens=10)[0]["generated_text"])
print(t5("translate English to German: The weather is nice today.")[0]["generated_text"])

major (top prediction; may differ) Learning GenAI is … (generated text changes every run) Das Wetter ist heute schön. (sample output)

05Choosing the right family

Demo · Which family should I use?

Rule of thumb: need a label, a score or a vector → encoder-only. Need free text or a conversation → decoder-only. Need to transform one text into another with a compact model → encoder-decoder.

06Summary

Encoder-only models understand (BERT family).
Decoder-only models generate (GPT family and most LLMs).
Encoder-decoder models transform text (T5, BART).
Pick by the shape of the output: label/vector, free text, or transformed text.

07Quick quiz

1. Which family is best for classifying support tickets cheaply?

2. GPT models are…

3. T5’s main idea is…

4. BART is pre-trained by…

5. You need sentence vectors for a RAG retriever. Pick a family.

08References

T5: Exploring the Limits of Transfer Learning (Raffel et al., 2019)
BART (Lewis et al., 2019)
Language Models are Few-Shot Learners (GPT-3) (Brown et al., 2020)

Created by Ajeetkumar