BERT Explained: How It Works and Its Variants

BERT is the model that taught computers to really understand a sentence. Google released it in 2018, and even today it quietly powers search, spam filters, chatbots and fraud detection. In this lesson, let us understand BERT step by step, with simple examples, pictures you can watch and code you can run.

01The simple idea behind BERT

Remember fill-in-the-blanks questions in school? To fill the blank, you never read only the words before it. You read the whole line, and the words after the blank also give you a clue. That is exactly how BERT learns.

Older models read only from left to right, like reading with one eye closed. When they reached the word “bank” in “I deposited cash at the bank”, they could only look backwards. BERT looks both ways. Try it:

When BERT came out, it broke records on 11 language tests, including GLUE and SQuAD (a reading-comprehension test). In 2019, Google started using it to understand search queries better.

02What is inside BERT

BERT is made by stacking the same building block again and again. This block is called a Transformer encoder layer. Words go in at the bottom as numbers; after passing through all the layers, each word comes out as a rich vector that carries its meaning in this sentence.

Inside one encoder layer

Each layer does two jobs. First, self-attention: every word “talks” to every other word and collects useful information. Second, a small feed-forward network processes each word on its own. Shortcut (residual) connections and normalisation keep training stable.

03How BERT reads your sentence

Before the text goes in, BERT adds a few special tokens:

• [CLS] at the start. Its final vector is used when we want one answer for the whole text, like “fraud or not fraud”.
• [SEP] at the end of a sentence, or between two sentences.
• [MASK] to hide a word during training.

Then every token gets three embeddings added together: which token it is, which sentence it belongs to (A or B), and its position in the line. Press Build to watch it:

WordPiece: breaking big words into pieces

BERT’s dictionary has about 30,000 pieces. A rare word is split into smaller known pieces; pieces that continue a word start with ##. So BERT never gets stuck on an unknown word.

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
print(tokenizer.tokenize("Tokenization handles unseen words"))
print(tokenizer("I love learning AI")["input_ids"])

04How BERT learns: pre-training

BERT learned from English Wikipedia (about 2.5 billion words) and a collection of books (about 800 million words). Nobody labelled this data. BERT makes its own practice questions, using two games.

Game 1 · Masked Language Modelling (fill in the blanks)

In every sentence, BERT picks 15% of the tokens to predict. But it does not always hide them the same way:

Now you play the game. Pick a sentence and see what BERT would guess:

Game 2 · Next Sentence Prediction

BERT gets two sentences, A and B. Half the time B really comes after A (IsNext); half the time B is a random sentence (NotNext). BERT must say which. Later research (RoBERTa) found this game does not help much, so many newer models skip it.

05Fine-tuning: teaching BERT your task

Pre-training is costly and is done once by big labs. You simply download that trained BERT and give it a little extra training on your own data. This is called fine-tuning.

For each task, BERT uses a different part of its output:

from transformers import AutoTokenizer, AutoModelForSequenceClassification, Trainer, TrainingArguments
from datasets import load_dataset

tok = AutoTokenizer.from_pretrained("bert-base-uncased")
model = AutoModelForSequenceClassification.from_pretrained("bert-base-uncased", num_labels=2)

data = load_dataset("csv", data_files={"train": "train.csv", "test": "test.csv"})   # columns: text, label
data = data.map(lambda b: tok(b["text"], truncation=True, padding="max_length", max_length=128), batched=True)

args = TrainingArguments("bert-fraud", num_train_epochs=3, learning_rate=2e-5, per_device_train_batch_size=16)
Trainer(model=model, args=args, train_dataset=data["train"], eval_dataset=data["test"]).train()

06Try it yourself in Python

You only need two libraries. Run these in Google Colab or Jupyter.

1. Let BERT fill a blank

# pip install transformers torch
from transformers import pipeline

fill = pipeline("fill-mask", model="bert-base-uncased")
for p in fill("New Delhi is the capital of [MASK].")[:3]:
    print(p["token_str"], round(p["score"], 3))

2. Get a meaning vector for each word

import torch
from transformers import AutoTokenizer, AutoModel

tok = AutoTokenizer.from_pretrained("bert-base-uncased")
model = AutoModel.from_pretrained("bert-base-uncased")

inputs = tok("I love learning AI", return_tensors="pt")
with torch.no_grad():
    out = model(**inputs)

print(out.last_hidden_state.shape)   # one 768-number vector per token
print(out.pooler_output.shape)       # one vector for the whole sentence

07The BERT family: its variants

After BERT, many teams made improved versions. Each one fixes one weakness: speed, size, accuracy or context length.

For special domains and Indian languages

• FinBERT for financial sentiment, BioBERT for medical text, SciBERT for research papers, Legal-BERT for legal documents.
• mBERT supports 104 languages. MuRIL from Google is built for 17 Indian languages, including Hindi, Marathi, Tamil and Bengali.

08BERT vs GPT: what is the difference?

Both come from the same Transformer family, but they take opposite halves of it.

09Where BERT is used, and its limits

Where BERT shines

• Classification: spam, sentiment, customer intent, fraud SMS detection
• Named entity recognition: picking names, amounts and account numbers out of documents
• Search and ranking: better search results, and the retriever in RAG systems
• Question answering: finding the exact answer inside a passage

Limitations to remember

• It cannot write long, free-form answers. It understands and scores text.
• The original model reads only 512 tokens at a time, so long documents must be cut into parts.
• You need to fine-tune it for each new task.
• Training from scratch is very costly, so always start from a public checkpoint.

10Quick quiz: check your understanding

11Summary

• BERT reads the whole sentence at once, so every word understands its context from both sides.
• It is a stack of Transformer encoder layers (12 in Base, 24 in Large).
• It learned by filling in blanks on billions of words; no labels needed.
• You fine-tune it with a small dataset for your own task.
• Choose the right family member: DistilBERT for speed, RoBERTa/DeBERTa for accuracy, ModernBERT for long text, Sentence-BERT for search.

12References

• Devlin et al., BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (2018)
• Liu et al., RoBERTa (2019) · Sanh et al., DistilBERT (2019)
• Hugging Face BERT documentation