RNN, LSTM & GRU: Sequence Modeling

Language, audio, time series, and video all share a property that image classification does not: order matters. The word "not" completely reverses the meaning of "good" in a sentiment sentence. A feedforward network has no way to express this — it sees each token in isolation, with no memory of what came before.

Recurrent Neural Networks (RNNs) solve this by threading a hidden state through the sequence, accumulating context as they process each element. LSTM and GRU extend this idea with gating mechanisms that overcome the fundamental weakness of vanilla RNNs: the vanishing gradient problem.

---

1. The Recurrent Neural Network

Architecture

An RNN processes a sequence $x_1, x_2, \ldots, x_T$ one element at a time. At each step $t$, it takes the current input $x_t$ and the previous hidden state $h_{t-1}$, and produces a new hidden state $h_t$:

$$h_t = \tanh\!\left(W_h \, h_{t-1} + W_x \, x_t + b\right)$$

The output at each step can be $y_t = W_y h_t + b_y$, or just the final hidden state $h_T$ may be used as a summary of the whole sequence.

A critical property: the weight matrices $W_h$, $W_x$, $W_y$ are shared across all time steps. The same transformation is applied at $t = 1$ and $t = T$. This weight sharing lets the network generalize to sequences of any length and keeps parameter count constant regardless of sequence length.

Unrolling Through Time

"Unrolling" is the conceptual trick of drawing one copy of the RNN cell for each time step, connected by the hidden state:

Each cell is identical — same weights, same computation — but receives a different context via $h_{t-1}$.

Many-to-Many, One-to-Many, Many-to-One

The same RNN architecture supports multiple task shapes:

---

2. Backpropagation Through Time

Training an RNN uses Backpropagation Through Time (BPTT): the unrolled network is treated as a very deep feedforward network, and gradients are propagated backward through each time step all the way to $t = 1$.

The loss $\mathcal{L}$ gradient with respect to $W_h$ at step $t$ requires multiplying through all earlier Jacobians:

$$\frac{\partial \mathcal{L}}{\partial W_h} = \sum_{t=1}^{T} \frac{\partial \mathcal{L}_t}{\partial h_t} \prod_{k=t}^{T-1} \frac{\partial h_{k+1}}{\partial h_k}$$

Each factor $\frac{\partial h_{k+1}}{\partial h_k} = \text{diag}\!\left(1 - h_k^2\right) W_h$ involves the derivative of $\tanh$, which is bounded by $[0, 1]$.

The Vanishing Gradient Problem

When these Jacobians are repeatedly multiplied, the result shrinks exponentially. After as few as 10 steps, the gradient reaching the earliest tokens is essentially zero — early inputs have no influence on learning.

---

3. Long Short-Term Memory (LSTM)

Hochreiter & Schmidhuber (1997) proposed the solution: separate the "memory" from the "working state" and control what is written and read via learned gates.

The Cell State — A Gradient Highway

The key innovation is the cell state $c_t$, a vector that runs horizontally through time with only minor, linear interactions:

$$c_t = f_t \odot c_{t-1} + i_t \odot \tilde{c}_t$$

Because the update is an addition (not a nonlinear squash), gradients flow back through $c_t$ without attenuation as long as the forget gate $f_t \approx 1$. This is the "constant error carousel" that makes LSTM work.

The Four Computations

All four computations take the same concatenated input $[h_{t-1},\, x_t]$:

Forget gate — decides what fraction of the old cell state to keep:

$$f_t = \sigma\!\left(W_f [h_{t-1}, x_t] + b_f\right)$$

Input gate — decides how much of the new candidate to write:

$$i_t = \sigma\!\left(W_i [h_{t-1}, x_t] + b_i\right)$$

Candidate cell state — the proposed new values:

$$\tilde{c}_t = \tanh\!\left(W_c [h_{t-1}, x_t] + b_c\right)$$

Output gate — filters what part of the cell state becomes the hidden state:

$$o_t = \sigma\!\left(W_o [h_{t-1}, x_t] + b_o\right), \qquad h_t = o_t \odot \tanh(c_t)$$

Interactive LSTM Cell

Adjust $h_{t-1}$, $x_t$, and $c_{t-1}$ to see how each gate responds. Step through all six computations:

---

4. Gated Recurrent Unit (GRU)

Cho et al. (2014) proposed a simplified architecture that retains the gating mechanism but eliminates the separate cell state, reducing the number of parameters by ~25%.

Two Gates Instead of Four

Reset gate — controls how much past state influences the candidate:

$$r_t = \sigma\!\left(W_r [h_{t-1}, x_t]\right)$$

Update gate — interpolates between old state and new candidate (replaces both forget and input gates):

$$z_t = \sigma\!\left(W_z [h_{t-1}, x_t]\right)$$

Candidate hidden state — the reset gate moderates the role of $h_{t-1}$:

$$\tilde{h}_t = \tanh\!\left(W [r_t \odot h_{t-1},\; x_t]\right)$$

New hidden state — a convex combination controlled by $z_t$:

$$h_t = (1 - z_t) \odot h_{t-1} + z_t \odot \tilde{h}_t$$

When $z_t = 0$ the cell ignores the input and copies the old state — equivalent to a maximally active forget gate. When $z_t = 1$ it discards all prior context and adopts the candidate.

Interactive GRU Cell

---

5. Comparison

Parameter Count

For a hidden size $d$ and input size $n$, the parameter count per gate is $(n + d) \times d + d$.

This is why GRU (3× RNN) trains notably faster than LSTM (4× RNN) at the same hidden dimension.

---

6. Beyond RNNs

Transformers (Vaswani et al., 2017) have displaced RNNs for most sequence tasks. Instead of processing tokens one-by-one, Transformers attend to all positions in parallel — eliminating the sequential bottleneck and scaling to much larger models.

However, RNNs remain relevant in specific settings:

---

Summary