5. Classification with Deep Learning (CNNs)

Convolution, pooling, batch normalization, dropout, regularization, optimizers, evaluation metrics

1. Limitations of Fully Connected Networks for Images

Before CNNs, neural networks used fully connected layers where every input pixel connects to every neuron. This creates serious problems for image data:

Fully connected network: every input neuron connects to every hidden neuron. For even modest images, this creates an impractical number of parameters.

2. CNN Innovations: Local Connectivity and Weight Sharing

Innovation 1: Local Connectivity

Instead of connecting each neuron to all inputs, each CNN neuron connects only to a small local region called its receptive field (e.g., 3×3 pixels). This mirrors how the visual cortex works: neurons respond to local regions of the visual field.

Left: FC layer connects to all inputs. Right: Conv layer connects each neuron to only its local receptive field.

Innovation 2: Weight Sharing

The same filter weights are applied at every spatial position. A 3×3 filter has only 9 learnable weights regardless of image size. This is the core insight that makes CNNs parameter-efficient.

The network weights literally become the convolutional filter parameters. Training learns the optimal filter values.

Parameter Count Comparison

3. The Convolution Operation

2D Discrete Convolution

A filter slides across the input, computing a dot product at each position to produce an activation (feature) map.

Worked Example (5×5 image, 3×3 kernel)

Convolution computation: position (0,0)

Image (top-left 3×3 subregion): $\begin{bmatrix}1 & 1 & 1 \\ 0 & 1 & 1 \\ 0 & 0 & 1\end{bmatrix}$, Kernel: $\begin{bmatrix}1 & 0 & 1 \\ 0 & 1 & 0 \\ 1 & 0 & 1\end{bmatrix}$

Element-wise multiply and sum:
$1\cdot1 + 1\cdot0 + 1\cdot1 + 0\cdot0 + 1\cdot1 + 1\cdot0 + 0\cdot1 + 0\cdot0 + 1\cdot1 = 4$

Output dimensions for a 5×5 input with 3×3 kernel (stride 1, no padding): $(5-3+1)\times(5-3+1) = 3\times3$.

Worked convolution: 5x5 image, 3x3 kernel, stride 1, producing a 3x3 output.

After Convolution: Activation Function

The convolution output is passed through a non-linear activation function (typically ReLU):

This is directly analogous to classical image filtering (Sobel, Gaussian), except the filter weights are learned from data rather than hand-designed.

Classical Sobel filters detect horizontal and vertical gradients. CNN filters learn to detect the features most useful for the task.

4. 3D Convolutions and Feature Maps

Why 3D?

Real images have multiple channels (RGB = 3 channels). Each CNN layer also produces multiple output channels (feature maps). Convolution is therefore inherently 3D.

A single filter spans all 3 RGB channels, producing one output feature map by summing contributions from all input channels.

Input volume (H×W×3) convolved with a 3×3×3 filter produces a 2D feature map. N filters produce an output of depth N.

Dimensions and Parameter Count

5. Pooling Layers

After convolution extracts local features, pooling layers reduce spatial resolution while increasing the effective receptive field of later layers.

Why Pooling?

1. Reduce spatial resolution: Decreases computation in subsequent layers.
2. Increase receptive field: Later filters see larger regions, detecting higher-level patterns.
3. Partial translation/scale invariance: Small input shifts do not change pooling output.

After pooling, subsequent convolution filters have larger effective receptive fields, allowing detection of larger patterns.

Max Pooling

The most common pooling operation: a window slides over each feature map and selects the maximum value at each position.

1  1  2  4
5  6  7  8
3  2  1  0
1  2  3  4

6  8
3  4

Max pooling with 2×2 window and stride 2: halves spatial dimensions in both axes.

Average Pooling

Comparison of max pooling (preserves strongest activations) and average pooling (smooths responses).

Convolution followed by pooling: spatial dimensions shrink progressively (11×11 → 9×9 → 3×3).

6. The Full CNN Pipeline

Standard Architecture Pattern

Complete CNN pipeline: Input image → stacked Conv+Pool layers → FC layers → Softmax → class probabilities.

Role of Each Stage

Example CNN classification: boat (0.94), cat (0.04), bird (0.02), dog (0.01). The softmax output is a probability distribution over all classes.

Example: 125×125×3 Input Network

Key pattern: Spatial dimensions decrease while depth (number of channels) increases as you go deeper. This represents progressive transformation from raw pixels to abstract, high-level features.

7. 1×1 Convolution

As networks grow deeper, the number of feature maps (channels) grows rapidly. 1×1 convolutions are used to reduce or expand the number of channels without affecting spatial dimensions.

1×1 convolution: reduces 192 channels to 64 while preserving H×W dimensions.

1×1 convolutions are used extensively in GoogLeNet/Inception (channel bottleneck before 3×3 and 5×5 convs) and ResNet bottleneck blocks.

8. Stride, Padding, and Output Size

Stride

Stride is the step size the filter moves between positions.

• Stride = 1: Filter moves one pixel at a time. Largest output.
• Stride = 2: Filter moves two pixels at a time. Roughly halves output dimensions.
• Stride > 1 can replace pooling for downsampling.

Stride 1 on a 5×5 input with 3×3 filter: 3×3 output. Stride 2: 2×2 output.

Padding

Padding adds extra pixels (typically zeros) around the input border before convolution.

Zero-padding: a 6×6 input padded to 8×8, convolved with 3×3 kernel (stride 1), produces a 6×6 output (same dimensions).

Output Size Formula

9. Overfitting and Regularization

The Bias-Variance Tradeoff

Three model complexity regimes: underfitting (too simple), just right (optimal capacity), overfitting (too complex).

Total generalization error = Bias + Variance. Optimal capacity minimizes both. Underfitting = high bias. Overfitting = high variance.

L1 and L2 Regularization

Regularization penalizes large weight values, encouraging simpler models that generalize better.

Decision boundaries: linear (0.83 train acc, too simple), NN without regularization (0.89, jagged/overfitting), NN with regularization (0.87, smooth/generalizable).

10. Dropout

Dropout randomly disables neurons during each training iteration with probability $p$ (typically $p = 0.5$).

Standard network (left) vs. network with dropout applied (right). Crossed-out neurons are disabled for this iteration.

Why Dropout Works

• Ensemble approximation: Each dropout pattern creates a different sub-network; training on all patterns is like training an ensemble.
• Prevents co-adaptation: Neurons cannot rely on specific other neurons being present, forcing distributed, redundant representations.
• Noise injection: Acts as regularization by preventing exact memorization.

11. Batch Normalization

The Problem: Internal Covariate Shift

During training, the distribution of each layer's inputs changes as parameters in preceding layers change. Each layer must continuously adapt to a shifting input distribution, slowing training. This is called internal covariate shift.

The Algorithm

BatchNorm placement: Linear → BatchNorm → Activation (most common) or Linear → Activation → BatchNorm.

BatchNorm: normalize to zero mean and unit variance, then apply learned scale (gamma) and shift (beta).

The input distribution is normalized, then re-scaled and re-shifted by learned gamma and beta parameters.

Benefits

12. Weight Initialization and Transfer Learning

Initialization Schemes

Transfer Learning

Use a model pretrained on a large dataset (e.g., ImageNet, 1.2M images) as the starting point for a new task.

1. Take a pretrained network (e.g., ResNet on ImageNet).
2. Freeze early layers — they already learn generic features (edges, textures, colors).
3. Replace and retrain the final FC layers for your specific task (different number of output classes).
4. Optionally fine-tune some later convolutional layers with a very small learning rate.

13. Gradient Descent Variants and Optimizers

Variants of Gradient Descent

Mini-batch GD is the standard in practice. Typical batch sizes: 32, 64, 128 images.

Momentum

Standard GD oscillates in narrow valleys or gets stuck in local minima. Momentum accumulates velocity from previous updates:

Without momentum: oscillating, slow. With momentum: smoother path, faster convergence. Think of a ball rolling down a hill accumulating speed.

Advanced Optimizers

Optimizer convergence paths on a 3D loss surface. Adam typically converges fastest and most reliably.

2D contour view: different optimizers trace different paths to the minimum. Adaptive methods (Adam, RMSprop) generally outperform vanilla SGD.

14. What Does a CNN See? Feature Map Visualization

Visualizing CNN feature maps reveals the hierarchical nature of learned representations:

Feature maps from early layers (top) show edge detectors in various orientations. Deeper layers (bottom) show increasingly abstract patterns.

Layer 1: edges and orientations. Layer 2: simple shapes and parts. Layer 3: whole face structures. Each layer builds on the previous.

3D feature map visualizations: early layers show clear spatial structure; deeper layers show more abstract, less human-interpretable patterns.

Labeled CNN stages: prediction/error, feature extraction, hierarchical feature extraction, cross-feature map learning, classification.

15. Classification Metrics

Confusion Matrix

Sorted prediction scores with a threshold: above threshold = predicted positive (TP if correct, FP if wrong); below = predicted negative.

Core Metrics

The Problem with Accuracy

Accuracy is misleading for imbalanced datasets. Example: 9,990 Class 0 samples, 10 Class 1 samples. A model predicting everything as Class 0 achieves 99.9% accuracy while completely failing to detect any Class 1 samples.

Precision-Recall Tradeoff

PR Curve and ROC Curve

PR curve: sweeping threshold from 0 to 1 traces the precision-recall tradeoff. Ideal curve stays near the top-right corner (high precision AND recall).

PR-AUC = area under the PR curve. Higher = better. Random classifier baseline depends on class ratio (not 0.5 as in ROC).

ROC curves: perfect classifier (top-left), better model (green), worse model (orange), random classifier (diagonal).

ROC curve with threshold sweep: each point corresponds to a different classification threshold.

Application-Driven Metric Selection

PR curves from 10-fold cross-validation. Low variance across folds indicates a reliable, generalizable model.

Flashcards