Convolutional Neural Networks

5.1 The idea: sliding pattern detectors

A dense layer (Module 3) connects every input to every output — for a 101×4 DNA window that means thousands of independent weights, and a pattern learned at position 12 tells the network nothing about the same pattern at position 80. Convolution fixes both problems.

A filter (or kernel) is a small weight vector — say 3 positions long. You slide it along the input, and at each position compute a dot product between the filter and the input chunk under it. From Module 1, a dot product measures similarity: so each output value answers "how much does this spot look like my pattern?" The sequence of answers, one per position, is the activation map (or feature map).

Two consequences of using the same filter at every position (weight sharing):

• Far fewer parameters: one 3-wide filter has 3×4 weights (plus a bias) no matter how long the sequence is — versus a dense layer's weight for every input–output pair.
• Translation invariance: a pattern is detected wherever it appears. Perfect for DNA, where a disease-relevant motif can sit anywhere inside the 101-bp window — the model shouldn't have to relearn it at each of 101 positions.

5.2 Conv1D on DNA, concretely

Take a toy one-hot sequence and a toy length-3 filter that "looks for" the pattern TAA (large weights on T, then A, then A). Slide it:

A length-5 input with a length-3 filter yields 5 − 3 + 1 = 3 output positions. In general, Paper 1's Equation 1 (simplified notation) says exactly this:

Read it aloud: the activation of filter k at position t is a nonlinearity σ (ReLU) applied to the dot product of the filter's weights w_k with the one-hot input y starting at position t, plus a bias. For input length L and filter size f, there are L − f + 1 valid positions — e.g. L = 101, f = 7 gives 95 outputs.

Each filter is a learnable motif detector: gradient descent shapes its weights until it fires on whatever short subsequence helps classification. Using K filters in parallel gives K different motif detectors, producing K activation maps.

5.3 Pooling: "was the motif there at all?"

After convolution we have, per filter, an activation value at every position — often more detail than we need. Max pooling keeps only the strongest activation per filter (or per local region). Paper 1's Equation 2 is global max pooling in one symbol:

Why do this?

• Dimensionality reduction: each filter's whole map collapses toward one (or a few) numbers — fewer values for later layers to process.
• Positional robustness: the pooled value answers "did my motif appear anywhere (and how strongly)?" — small shifts in where the motif sits no longer change the output.

The trade-off: pooling discards precise location. "Motif present" survives; "motif at position 37" does not. When exact position matters, the architecture must recover it elsewhere — one reason Paper 1 also runs a BiLSTM over the sequence (Module 6).

5.4 Stacking layers & going 2-D

Stack conv layers and a hierarchy emerges: early filters detect simple, tiny patterns (in images: edges and color blobs; in DNA: short motifs), and deeper layers combine those detections into larger, more abstract structures (textures, object parts; composite regulatory patterns). Each layer's "pixels" are the previous layer's pattern-detections.

Conv2D is the same machinery for images: a small square filter (say 3×3, spanning all C channels) slides across the H×W grid in both directions, producing a 2-D activation map per filter. Everything else — weight sharing, translation invariance, pooling, stacking — carries over unchanged.

5.5 What CNNs are good and bad at

• Good: local patterns, detected cheaply (weight sharing), wherever they occur (translation invariance).
• Bad: long-range dependencies. A filter only sees f positions at once. What if two motifs 80 bp apart must both be present, and interact, for a variant to be pathogenic? A single conv layer cannot see them together, and the receptive field (how much input a deep unit can see) grows only slowly with each added layer.

5.6 Seeing what a CNN sees: Grad-CAM

A trained CNN is a black box: it says "aluminum, 97%" but not why. Grad-CAM (gradient-weighted class activation mapping) opens the box. The recipe: take the class score (e.g. the "aluminum" logit), compute its gradients flowing back into the last convolutional layer's activation maps, use those gradients to weight how much each map mattered for that class, and combine the weighted maps into a heatmap over the input showing which regions drove the decision. Intuition: "if nudging this region's activations would change the class score a lot, that region mattered."