Why RNNs Couldn't Scale to Modern LLMs

I tried to remember the beginning of this sentence, but my gradients vanished somewhere around the fourth word.

Attn, Gradient-Starved AI Agent

2.1.1 The Sequence Modeling Problem

In Chapter 1, you learned how text is converted into sequences of token IDs. Now those sequences need to be processed. How does a neural network read a sentence one token at a time and build an understanding of the whole? Language is inherently sequential. The meaning of a word depends on the words that came before it: "bank" means something entirely different in "river bank" versus "bank account." Any neural network that processes language must therefore have a mechanism for incorporating context from earlier positions in a sequence.

A standard feedforward network (an MLP) cannot do this naturally. It takes a fixed-size input and produces a fixed-size output with no notion of ordering or memory. If you feed it one word at a time, it has no way to remember what came before. If you concatenate an entire sentence into one large input vector, you lose the ability to handle sentences of different lengths, and you learn position-specific weights rather than general sequential patterns.

The recurrent neural network (RNN) solves this by introducing a hidden state that acts as a form of memory. At each time step, the network reads one input token and combines it with its previous hidden state to produce a new hidden state. This allows information to flow forward through time, step by step.

2.1.2 RNN Fundamentals

To feel the pain that motivated attention, we have to first feel the recurrence itself. The vanilla RNN cell, just three matrix multiplications and a tanh, built a decade of sequence modeling. Every later complication (LSTM gates, GRUs, residual connections) addressed a problem that the vanilla cell makes immediately visible.

The Vanilla RNN Cell

An RNN processes a sequence $(x_{1}, x_{2}, ..., x_{T})$ one element at a time. At each time step $t$, the RNN computes:

This hidden state is then linearly projected to produce the output at each time step:

Here, $h_{t}$ is the hidden state at time $t$, $W_{hh}$ is the recurrent weight matrix (hidden-to-hidden), $W_{xh}$ is the input weight matrix, and $W_{hy}$ maps the hidden state to the output. Crucially, these weight matrices are shared across all time steps, which is what allows the network to generalize to sequences of any length.

Figure 2.1.2: An RNN unrolled through time. The same weights are applied at every step. Hidden states (red arrows) carry information forward through the sequence.

Let us implement a vanilla RNN cell in PyTorch to make this concrete:

# Vanilla RNN cell from scratch: h_t = tanh(W_hh @ h + W_xh @ x + b).
# Shows how each time step mixes the previous hidden state with new input.
import torch
import torch.nn as nn
class VanillaRNNCell(nn.Module):
    """A single RNN cell: h_t = tanh(W_hh @ h_{t-1} + W_xh @ x_t + b)"""
    def __init__(self, input_size, hidden_size):
        super().__init__()
        self.W_xh = nn.Linear(input_size, hidden_size)
        self.W_hh = nn.Linear(hidden_size, hidden_size, bias=False)
        self.hidden_size = hidden_size
    def forward(self, x_t, h_prev):
        # Combine input and previous hidden state, then apply tanh
        return torch.tanh(self.W_xh(x_t) + self.W_hh(h_prev))
        # Process a sequence of 5 tokens, each with embedding dim 10, hidden dim 20
        cell = VanillaRNNCell(input_size=10, hidden_size=20)
        seq = torch.randn(5, 10) # 5 time steps, 10-dim input each
        h = torch.zeros(20) # initial hidden state
        # Step through the sequence
        for t in range(seq.size(0)):
            h = cell(seq[t], h)
            print(f"Step {t}: h norm = {h.norm():.4f}, h[:3] = {h[:3].tolist()}")

# Production equivalent: process an entire sequence in one call
import torch, torch.nn as nn
rnn = nn.RNN(input_size=10, hidden_size=20, batch_first=True)
seq = torch.randn(1, 5, 10) # (batch=1, seq_len=5, input_dim=10)
output, h_final = rnn(seq) # output: all hidden states; h_final: last one
print(f"All hidden states: {output.shape}") # (1, 5, 20)
print(f"Final hidden state: {h_final.shape}") # (1, 1, 20)

# Compare parameter counts: RNN vs LSTM vs GRU for the same
# input_size and hidden_size. LSTM has ~4x more parameters than RNN.
import torch
import torch.nn as nn
# Compare parameter counts
input_size, hidden_size = 128, 256
rnn = nn.RNN(input_size, hidden_size, batch_first=True)
lstm = nn.LSTM(input_size, hidden_size, batch_first=True)
gru = nn.GRU(input_size, hidden_size, batch_first=True)
for name, model in [("RNN", rnn), ("LSTM", lstm), ("GRU", gru)]:
    params = sum(p.numel() for p in model.parameters())
    print(f"{name:4s}: {params:>8,} parameters")
    # Forward pass comparison
    x = torch.randn(1, 50, input_size)
    rnn_out, _ = rnn(x)
    lstm_out, _ = lstm(x)
    gru_out, _ = gru(x)
    print(f"\nAll output shapes: {rnn_out.shape}") # Same for all three

Notice how each hidden state depends on the previous one. The final hidden state $h_{4}$ has (in principle) been influenced by every token in the sequence. This is the memory mechanism of the RNN. But as we will soon see, "in principle" and "in practice" diverge dramatically for long sequences.

2.1.3 The Vanishing and Exploding Gradient Problem

Training an RNN means computing gradients through time via backpropagation through time (BPTT). To update the weights, we need to know how the loss at time step $T$ depends on the hidden state at time step $t$. By the chain rule:

This is a product of $(T - t)$ Jacobian matrices. Each factor contains the recurrent weight matrix $W_{hh}$ multiplied by the derivative of tanh (which is always between 0 and 1). When this product involves many factors, two things can happen:

(The following gradient analysis is optional for intuition-first readers. The key takeaway is that multiplying many matrices together causes gradients to either vanish or explode.)

• Vanishing gradients: If the largest singular value of each Jacobian is less than 1, the product shrinks exponentially toward zero. Gradients from distant time steps become negligible, and the network cannot learn long-range dependencies.
• Exploding gradients: If the largest singular value exceeds 1, the product grows exponentially. Gradients become astronomically large, causing unstable updates. (This is typically handled with gradient clipping, but it is a symptom rather than a cure.)

Let us demonstrate this empirically by measuring gradient magnitudes through an RNN:

# Vanishing gradient demo: feed a long sequence through an RNN and
# measure how the gradient at the first time step shrinks to near zero.
import torch
import torch.nn as nn
# Create a simple RNN and a long sequence
rnn = nn.RNN(input_size=32, hidden_size=64, batch_first=True)
x = torch.randn(1, 100, 32, requires_grad=True) # 100-step sequence
h0 = torch.zeros(1, 1, 64)
# Forward pass: collect all hidden states
output, _ = rnn(x, h0)
# Use the LAST hidden state to compute a scalar loss
loss = output[0, -1, :].sum()
loss.backward()
# Measure gradient magnitude at each time step
grad_norms = x.grad[0].norm(dim=1)
print("Gradient norms (selected time steps):")
for t in [0, 10, 25, 50, 75, 99]:
    print(f" Step {t:3d}: {grad_norms[t]:.6f}")
    print(f"Ratio (step 99 / step 0): {grad_norms[99] / grad_norms[0]:.1f}x")

The gradient at step 0 is nearly 400,000 times smaller than at step 99. The network can barely "feel" the influence of early tokens when updating its weights. This is the vanishing gradient problem in action.

Figure 2.1.3: Gradient magnitude vs. time step in a vanilla RNN. Gradients from early time steps are exponentially smaller than those from recent steps, making it nearly impossible to learn long-range dependencies.

2.1.4 LSTM and GRU: Gated Recurrent Networks

The vanishing gradient problem makes vanilla RNNs nearly useless for long-range dependencies. We need a mechanism that can selectively preserve information across many time steps while still learning what to remember and what to forget. This is exactly what gated recurrent networks provide.

The Long Short-Term Memory (LSTM) cell, introduced by Hochreiter and Schmidhuber in 1997, was specifically designed to combat the vanishing gradient problem. Its key innovation is the cell state, a dedicated memory highway that runs through the entire sequence with only linear interactions.

The LSTM Cell

Figure 2.1.5a: The LSTM cell. The cell state (green line at top) acts as a memory highway with only additive interactions, allowing gradients to flow across many time steps. Three gates control information flow: the forget gate decides what to discard, the input gate decides what to store, and the output gate decides what to reveal as the hidden state.

An LSTM cell maintains two vectors: a hidden state $h_{t}$ and a cell state $c_{t}$. Three gates control the flow of information:

1. Forget gate $f_{t}$: Decides what to discard from the cell state.
   $f_{t} = \sigma (W_{f} [h_{t-1}, x_{t}] + b_{f})$
2. Input gate $i_{t}$: Decides what new information to store.
   $i_{t} = \sigma (W_{i} [h_{t-1}, x_{t}] + b_{i})$
3. Output gate $o_{t}$: Decides what to output from the cell state.
   $o_{t} = \sigma (W_{o} [h_{t-1}, x_{t}] + b_{o})$

The cell state update is:

The updated cell state is then filtered through the output gate to produce the hidden state that the rest of the network sees:

The GRU Cell

The Gated Recurrent Unit (GRU), proposed by Cho et al. in 2014, simplifies the LSTM by merging the cell state and hidden state into a single vector and using only two gates:

1. Update gate $z_{t}$: Controls how much of the old state to keep (combining the LSTM's forget and input gates).
2. Reset gate $r_{t}$: Controls how much of the old state to use when computing the candidate hidden state.

2.1.5 Bidirectional RNNs

A standard (unidirectional) RNN reads the sequence left to right. The hidden state at position $t$ encodes information from tokens 1 through $t$, but knows nothing about tokens after position $t$. For many tasks (machine translation, named entity recognition, sentiment analysis), the context on both sides of a word matters.

A bidirectional RNN addresses this by running two independent RNNs over the sequence: one forward (left to right) and one backward (right to left). At each position, the two hidden states are concatenated to form a representation that encodes both past and future context:

2.1.6 The Encoder-Decoder Architecture (Seq2Seq)

Many important NLP tasks require mapping a variable-length input sequence to a variable-length output sequence: machine translation (English to French), summarization (article to summary), and question answering (question to answer). The sequence-to-sequence (seq2seq) architecture, introduced by Sutskever et al. in 2014, handles this with two RNNs:

1. Encoder: Reads the input sequence token by token and compresses it into a single fixed-length vector: the final hidden state $h_{T}$. In the original seq2seq literature, this vector is sometimes called the "context vector," but we will reserve that term for the attention-weighted sum introduced in Section 2.2. Here, we call it the encoder summary vector.
2. Decoder: Receives the encoder summary vector as its initial hidden state and generates the output sequence one token at a time, feeding each generated token as input to the next step.

Figure 2.1.6: The encoder-decoder (seq2seq) architecture. The entire source sentence must be compressed into a single fixed-length context vector, creating an information bottleneck that limits performance on long sequences.

The Information Bottleneck

The critical weakness of this architecture is the information bottleneck. The encoder must compress an entire source sentence (which could be 50 words or 500 words) into a single vector of fixed dimensionality (typically 256 to 1024 dimensions). For short sentences this works reasonably well, but as sentence length increases, the model must cram more and more information into the same number of dimensions. Important details from early parts of the sentence are inevitably lost.

Empirically, Cho et al. (2014) showed that seq2seq translation quality degrades sharply for sentences longer than about 20 words. This degradation is a direct consequence of the bottleneck: the decoder simply does not have access to enough information about the source sentence.

Attention solves the information bottleneck, but it does not address a separate, equally important limitation of recurrent architectures. Even with perfect gradient flow and unlimited memory, RNNs face a structural barrier that limits their practical scalability.

2.1.7 Sequential Computation: The Parallelism Problem

Beyond the gradient and bottleneck issues, RNNs have a fundamental computational limitation: they are inherently sequential. To compute $h_{t}$, you must first compute $h_{t-1}$. There is no way to parallelize across time steps.

On modern GPUs, which contain thousands of cores designed for parallel computation, this is devastating. A feedforward layer can process all positions simultaneously, but an RNN must process them one by one. For a sequence of length $n$, an RNN requires $O(n)$ sequential operations, while a fully parallelizable architecture requires only $O(1)$ (at the cost of more total computation per step).

# Benchmark RNN vs LSTM vs GRU wall-clock time on a 512-step sequence.
# The sequential bottleneck becomes clear compared to parallel alternatives.
import torch, torch.nn as nn, time
seq_len, batch, inp, hid = 512, 32, 256, 256
x = torch.randn(batch, seq_len, inp)
rnn = nn.LSTM(inp, hid, batch_first=True)
linear = nn.Linear(inp, hid) # Feedforward: processes all positions in parallel
# Time the sequential RNN
t0 = time.perf_counter()
for _ in range(100):
    _ = rnn(x)
    rnn_time = (time.perf_counter() - t0) / 100
    # Time the parallel feedforward
    t0 = time.perf_counter()
    for _ in range(100):
        _ = linear(x)
        ff_time = (time.perf_counter() - t0) / 100
        print(f"LSTM forward pass: {rnn_time*1000:.1f} ms")
        print(f"Linear forward pass: {ff_time*1000:.1f} ms")
        print(f"Speedup: {rnn_time/ff_time:.1f}x")

The feedforward layer is over 14 times faster, and this gap widens dramatically on GPUs and with longer sequences. This sequential bottleneck is one of the primary reasons the field moved toward attention-based architectures that can process all positions in parallel.

2.1.8 Putting It All Together: A Seq2Seq Translation Model

Let us tie everything together by building a minimal seq2seq model. This will crystallize both the architecture and its limitations, setting the stage for the attention mechanism in Section 2.2.

# Seq2seq architecture: an LSTM encoder compresses input into a context
# vector, and a decoder LSTM generates output tokens one at a time.
import torch
import torch.nn as nn
class Seq2SeqEncoder(nn.Module):
    def __init__(self, vocab_size, emb_dim, hidden_dim):
        super().__init__()
        self.embedding = nn.Embedding(vocab_size, emb_dim)
        self.rnn = nn.LSTM(emb_dim, hidden_dim, batch_first=True)
    def forward(self, src):
        # src: (batch, src_len) token indices
        embedded = self.embedding(src) # (batch, src_len, emb_dim)
        outputs, (h, c) = self.rnn(embedded) # h, c: (1, batch, hidden)
        return outputs, (h, c)
class Seq2SeqDecoder(nn.Module):
    def __init__(self, vocab_size, emb_dim, hidden_dim):
        super().__init__()
        self.embedding = nn.Embedding(vocab_size, emb_dim)
        self.rnn = nn.LSTM(emb_dim, hidden_dim, batch_first=True)
        self.fc = nn.Linear(hidden_dim, vocab_size)
    def forward(self, tgt, hidden):
        # tgt: (batch, 1) single token at a time
        embedded = self.embedding(tgt)
        output, hidden = self.rnn(embedded, hidden)
        prediction = self.fc(output) # (batch, 1, vocab_size)
        return prediction, hidden
        # Build model
        enc = Seq2SeqEncoder(vocab_size=5000, emb_dim=64, hidden_dim=128)
        dec = Seq2SeqDecoder(vocab_size=6000, emb_dim=64, hidden_dim=128)
        # Simulate encoding "the cat sat"
        src_tokens = torch.tensor([[12, 457, 89]]) # batch=1, len=3
        enc_outputs, (h, c) = enc(src_tokens)
        # Decode three tokens autoregressively
        dec_input = torch.tensor([[1]]) # <SOS> token
        for step in range(3):
            logits, (h, c) = dec(dec_input, (h, c))
            next_token = logits.argmax(dim=-1) # greedy decode
            print(f"Decode step {step}: token = {next_token.item()}, logits shape = {logits.shape}")
            dec_input = next_token
            print(f"\nEncoder outputs shape: {enc_outputs.shape}")
            print(f"Context vector shape: h={h.shape}, c={c.shape}")
            print("Note: decoder only sees (h, c), not enc_outputs. That is the bottleneck.")

Look at the shapes: the encoder produces hidden states for all 3 positions (torch.Size([1, 3, 128])), but the decoder receives only the final hidden state (torch.Size([1, 1, 128])). All the rich, position-specific information from the encoder is discarded. This is exactly the problem that attention will solve.

Exercises