Encoder–Decoder (Seq2Seq)

Translation, summarisation and chatbots all turn one sequence into another of a different length. The sequence-to-sequence (seq2seq) model (2014) does this with two RNNs: an encoder that reads and a decoder that writes. It was a breakthrough, but it has one weak spot, the context vector. We train a real model and watch that weakness appear, which sets up Module 5.

01The seq2seq architecture

• The encoder (an RNN, LSTM or GRU) reads the input token by token. Its final hidden state is the context vector c.
• The decoder (another RNN) starts from c and a special start token, and generates the output one token at a time, feeding each output back in as the next input.
• Input and output lengths can differ: 4 Hindi words → 4 English words here, but it could be 4 → 7.

02Teacher forcing and generation

Let us train a tiny GRU seq2seq model to reverse digit sequences (3 1 4 1 5 → 5 1 4 1 3). We train on lengths 3–12, then test on several lengths:

# A tiny encoder-decoder (GRU) that learns to REVERSE a sequence of digits: 3 1 4 -> 4 1 3
import torch, torch.nn as nn
torch.manual_seed(0); torch.set_num_threads(1)     # reproducible results
V, SOS, H = 12, 10, 64                 # digits 0-9, 10 = start token, 11 = padding

class Seq2Seq(nn.Module):
    def __init__(self):
        super().__init__()
        self.emb = nn.Embedding(V, 32)
        self.encoder = nn.GRU(32, H, batch_first=True)
        self.decoder = nn.GRU(32, H, batch_first=True)
        self.out = nn.Linear(H, V)
    def forward(self, src, tgt_in):
        _, context = self.encoder(self.emb(src))          # the WHOLE input squeezed into one vector
        dec, _ = self.decoder(self.emb(tgt_in), context)   # teacher forcing: feed the right previous token
        return self.out(dec)
    @torch.no_grad()
    def generate(self, src):
        _, h = self.encoder(self.emb(src)); tok = torch.full((src.size(0), 1), SOS); outs = []
        for _ in range(src.size(1)):
            d, h = self.decoder(self.emb(tok), h); tok = self.out(d).argmax(-1); outs.append(tok)
        return torch.cat(outs, 1)

def batch(n, length):
    src = torch.randint(0, 10, (n, length)); tgt = src.flip(1)
    return src, torch.cat([torch.full((n, 1), SOS), tgt[:, :-1]], 1), tgt

model = Seq2Seq(); opt = torch.optim.Adam(model.parameters(), 3e-3); lossf = nn.CrossEntropyLoss()
for step in range(3000):
    L = torch.randint(3, 13, (1,)).item()                  # train on lengths 3-12
    src, tgt_in, tgt = batch(64, L)
    loss = lossf(model(src, tgt_in).reshape(-1, V), tgt.reshape(-1))
    opt.zero_grad(); loss.backward(); opt.step()

s = torch.tensor([[3, 1, 4, 1, 5]]); print("input", s.tolist()[0], "-> output", model.generate(s).tolist()[0])
for L in [4, 8, 12, 16, 20]:
    src, _, tgt = batch(500, L)
    acc = (model.generate(src) == tgt).all(1).float().mean().item()
    print(f"length {L:2d}: {acc * 100:5.1f}% of sequences reversed perfectly")

03The context-vector bottleneck

Look at the accuracy: perfect at length 4, about 70% at length 8, and under 10% at length 12, even though length 12 was in the training data. Lengths 16 and 20 were never seen in training, so near 0% is expected there. Everything the decoder knows must squeeze through one vector of 64 numbers.

04What this implies for the next module

The fix began with a simple idea: instead of one context vector, let the decoder look back at all encoder hidden states and decide which ones matter for the current word. That is attention. Then the 2017 paper “Attention Is All You Need” asked: what if we drop the RNN entirely and use only attention?

05Summary

• Seq2seq = encoder (reads) + decoder (writes), joined by a context vector.
• Training uses teacher forcing; inference generates one token at a time.
• A single fixed-size vector is a bottleneck: our model fell from 100% to under 10% as length grew from 4 to 12.
• RNN seq2seq is also slow (sequential) and still weak on long-range memory.
• Attention and then transformers were built to fix exactly these problems.

06Quick quiz

07References

• Sequence to Sequence Learning with Neural Networks (Sutskever et al., 2014)
• On the Properties of Neural Machine Translation: Encoder–Decoder Approaches (Cho et al., 2014)
• Neural Machine Translation by Jointly Learning to Align and Translate (Bahdanau et al., 2015)
• PyTorch: NLP from scratch, translation with seq2seq