Module 6 · Core Architectures

RNNs, LSTM & BiLSTM

CNNs (Module 5) spot local patterns. But sequences — sentences, DNA, time series — carry meaning in their order and in interactions between far-apart positions. This module builds the family of models that read: RNN → LSTM → BiLSTM, ending exactly where Paper 1's architecture lives.

▶

Audio recap

A ~2-minute spoken summary of this module — great for revision on the go.

6.1 Why order matters

Compare "the dog bit the man" with "the man bit the dog" — same words, opposite meaning. Sequences are like that everywhere: in language, in stock prices, and in DNA, where whether a variant is harmful can depend on bases dozens of positions away (a splice site, a regulatory motif). Two things a sequence model must handle:

Order: the same elements in a different arrangement mean something different.
Long-range interactions: position 3 can change the meaning of position 80.

A plain MLP takes one fixed-size vector and has no notion of position at all. A CNN slides small filters, so it sees fixed-size local chunks — great for motifs, blind to interactions wider than its receptive field. What we want is a model that processes a sequence the way you read a sentence: one element at a time, remembering what came before.

6.2 Recurrent neural networks: a running memory

A recurrent neural network (RNN) does exactly that. It walks through the sequence one element at a time, carrying a hidden state h_t — a vector that acts as a running summary of everything read so far. At each step it combines the previous summary with the new input:

h_t = f(h_t−1, x_t)

Crucially, the same function f — the same weights — is reused at every step, so one RNN handles sequences of any length. A useful mental picture is unrolling: imagine the loop drawn out as a chain of copies of the same cell, each passing its hidden state to the next. The unrolled RNN looks like a very deep network whose depth equals the sequence length.

The key idea The hidden state is the model's memory. After reading 60 DNA bases, h₆₀ should encode whatever about those 60 bases matters for what comes next. Everything in this module is about making that memory actually work over long distances.

6.3 The vanishing gradient problem

Training an RNN means backpropagating the loss through every step of the unrolled chain. The chain rule multiplies one slope per step — so the gradient reaching step 1 from step 100 is a product of ~100 factors. If those factors are typically a bit below 1 (which sigmoid/tanh slopes usually are), the product shrinks exponentially: 0.9¹⁰⁰ ≈ 0.00003.

The consequence: gradients from late positions barely reach early positions, so early inputs stop influencing learning. The network physically receives them but cannot learn to use them. In practice, plain RNNs forget dependencies longer than roughly 10–20 steps — hopeless for a 101-letter DNA window.

The mirror case If the per-step factors are a bit above 1, the product blows up instead — exploding gradients, which make training unstable (weights jump wildly). Exploding gradients have a crude fix (clip the gradient at a maximum size); vanishing gradients needed a new architecture.

6.4 LSTM: memory with gates

The Long Short-Term Memory (LSTM) network fixes vanishing gradients with two additions. First, alongside the hidden state it carries a cell state — think of it as a conveyor belt running the length of the sequence, on which information can ride largely untouched. Second, three learned gates control the belt. Each gate is a small sigmoid-activated layer producing values between 0 and 1 — a valve, per dimension, from "fully closed" to "fully open":

Forget gate: what to erase from the cell state.
Input gate: what new information to write onto it.
Output gate: what part of the cell state to reveal as this step's hidden state.

Metaphor: a reader with a notebook Picture reading a long report while keeping a notebook. At each paragraph you decide what to jot down (input gate), what old note to cross out because it's now irrelevant (forget gate), and which notes to consult when forming your current opinion (output gate). The notebook is the cell state: notes persist untouched until you deliberately cross them out — unlike a plain RNN, which rewrites its entire memory every single step.

Why this tames vanishing gradients: updates to the cell state are largely additive — new information is added onto the belt rather than the whole memory being squashed through a nonlinearity each step. Gradients can flow backwards along the belt across long spans without being multiplied down to zero at every step, so the LSTM can learn dependencies spanning hundreds of positions.

6.5 BiLSTM: reading both directions

An LSTM reading left-to-right only knows the past at each position. But often the context after a position matters just as much: in "I sat by the bank and watched the river," the word "river" — arriving later — is what resolves "bank." In DNA there is no privileged reading direction at all for this task: bases downstream of a variant carry as much diagnostic signal as bases upstream.

A Bidirectional LSTM (BiLSTM) runs two independent LSTMs: one reads the sequence left-to-right, the other right-to-left, and at each position their hidden states are concatenated. Every position's representation therefore summarizes the entire sequence — everything before it and everything after it.

Where Paper 1 uses this Paper 1's BiLSTM sits right after the convolutional layers. Its Eqs. 3–4 are exactly the two passes, written with a forward state d⃗_t and a backward state d⃖_t:

d⃗_t = LSTM(d_t−1, s_t−1)forward, left→right (Eq. 3)

d⃖_t = LSTM(d_t+1, s_t+1)backward, right→left (Eq. 4)

And the design is symmetric on purpose: the variant sits at the center of the 101-bp window, so the forward pass brings in 50 bp of upstream context and the backward pass 50 bp of downstream context — both directions contribute equally to the decision at the variant site.

6.6 Why CNN + BiLSTM is a natural pairing

Paper 1's architecture is a division of labor. Conv1D filters (Module 5) excel at extracting short local patterns — sequence motifs a few bases wide. The BiLSTM then reads across the whole window of detected motifs, integrating them with long-range, bidirectional context. Each component does what the other can't.

101-bp window→ one-hot encoding→ Conv1D filters (motifs)→ max pooling→ BiLSTM (context)→ attention→ dense + softmax

Paper 1's ablation logic The paper defends the hybrid empirically by removing each part:

CNN-only: strong at local motifs but has no mechanism for long-range dependencies across the window — weaker overall ROC-AUC.
BiLSTM-only: reads long-range context but lacks the local motif-extraction front end, so it works from raw bases instead of meaningful features.
Hybrid: outperforms both — evidence that both local and long-range signal genuinely matter for variant classification.

This is the classic ablation argument: each component is justified by what performance loses without it.

6.7 A first taste of attention

After the BiLSTM, Paper 1 adds one more mechanism: attention. Not every position in the window is equally informative — the bases right around the variant, or a disrupted motif 30 bp away, may matter far more than the rest. The attention layer lets the model learn which positions and features to weight most for the final decision: each feature is scored using learned Query, Key, and Value vectors, the scores are softmax-normalized into weights, and the output is the weighted combination that feeds the dense classification layer.

Module 7 shows what happens when attention stops being a garnish and becomes the whole meal — the Transformer.

Exam warm-up — say it out loud Answer each aloud in under a minute, no notes:

Explain the vanishing gradient problem to a first-year student — why does multiplying many small slopes make early inputs unlearnable?
Name the three LSTM gates and what each one decides.
Why bidirectional for DNA? What does the backward pass contribute at the variant position?
Defend the hybrid CNN+BiLSTM choice against "why not just use one?" — cite the ablation results.