5.1 CNN Architecture Landscape

AlexNet, VGGNet, GoogLeNet, ResNet, DenseNet, SENet, NASNet and the evolution of deep image classifiers

1. ImageNet and the Deep Learning Revolution

The ImageNet Large Scale Visual Recognition Challenge (ILSVRC) began in 2010 and served as the primary benchmark driving the modern deep learning era. It features 1000 object categories, 1.2 million training images, 50K validation images, and 150K test images. Performance is measured by Top-1 and Top-5 error rates.

Historical Top-5 Error Progression

Top-5 error on ImageNet ILSVRC over time. AlexNet (2012) caused the inflection. ResNet (2015) crossed the estimated human threshold.

Classification accuracy scaling with data: ResNeXt-101 with and without large-scale Instagram pretraining. Weakly supervised data at scale dramatically improves accuracy.

2. AlexNet (2012)

AlexNet (Krizhevsky, Sutskever, Hinton) was the breakthrough architecture that launched the deep learning revolution in computer vision, winning ILSVRC 2012 with a margin of nearly 11 percentage points over the runner-up.

Architecture

Five convolutional layers followed by three fully connected layers:

• Conv1: 96 filters, 11×11, stride 4 → ReLU → LRN → MaxPool
• Conv2: 256 filters, 5×5 → ReLU → LRN → MaxPool
• Conv3–4: 384 filters, 3×3 → ReLU
• Conv5: 256 filters, 3×3 → ReLU → MaxPool
• FC6–7: 4096 units → ReLU → Dropout
• FC8: 1000 units → Softmax

AlexNet architecture showing the dual-GPU split and layer progression from 224×224×3 input to 1000-class softmax output.

Key Innovations

3. ZFNet (2013)

ZFNet (Zeiler and Fergus, "Visualizing and Understanding Convolutional Networks") improved AlexNet by using deconvolution-based visualization to understand what each layer learns, guiding architectural changes.

ZFNet architecture with 7×7 kernels and stride 2 in Conv1, correcting the aliasing seen in AlexNet's 11×11 stride-4 filters.

The visualization methodology was ZFNet's primary contribution, providing the field with interpretability tools that guided subsequent architecture design. Top-5 error: 14.7%.

4. VGGNet (2014)

VGGNet (Simonyan and Zisserman) demonstrated that depth with small uniform kernels was the key to better performance. It uses only 3×3 convolutions throughout the entire network.

Receptive Field Equivalence

The central insight: stacking small kernels achieves the same receptive field as a single large kernel but with fewer parameters and more non-linearities.

VGGNet architecture: stacked 3×3 convolution blocks with max-pooling between groups and three fully connected layers at the end.

VGG16 / VGG19 Structure

VGG16 (16 weight layers): 2×Conv-64 → Pool, 2×Conv-128 → Pool, 3×Conv-256 → Pool, 3×Conv-512 → Pool, 3×Conv-512 → Pool, FC-4096, FC-4096, FC-1000 → Softmax.

Top-5 error: 7.5% (single model), 6.8% (ensemble).

5. GoogLeNet / Inception v1 (2014)

GoogLeNet (Szegedy et al., "Going Deeper with Convolutions") introduced the Inception module: instead of choosing a single kernel size, apply multiple kernel sizes in parallel and concatenate the results. 22 layers deep with only ~5 million parameters.

Inception Module

The solution: add 1×1 bottleneck convolutions before the expensive 3×3 and 5×5 operations to first reduce the channel dimension.

(a) Naive inception module and (b) inception module with 1×1 bottleneck convolutions for dimensionality reduction. The 1×1 convolutions dramatically cut computation.

Full Architecture

• 22 layers deep (weight layers only).
• Average pooling before the classifier instead of multiple FC layers — drastically reduces parameters.
• Dropout 0.7 after average pooling.
• Auxiliary classifiers at intermediate layers inject gradient signal into early layers during training (removed at inference).

GoogLeNet: stacked inception modules, auxiliary classifiers at two intermediate points, and final average pooling before softmax.

6. ResNet (2015)

ResNet (He et al., "Deep Residual Learning for Image Recognition") solved the degradation problem that had prevented very deep networks from training effectively.

The Degradation Problem

In very deep plain networks, adding more layers causes accuracy to saturate and then decrease — even the training error increases. This is not overfitting and not underfitting in the conventional sense. A 56-layer plain network has higher training error than a 20-layer network.

The degradation problem: the 56-layer plain network has higher training and test error than the 20-layer network. Adding depth hurts performance without residual connections.

Residual Learning

Instead of learning the desired mapping $H(\mathbf{x})$ directly, learn the residual $F(\mathbf{x}) = H(\mathbf{x}) - \mathbf{x}$:

Why it works:

1. Easy identity mapping: If a layer is not needed, the network sets $F(\mathbf{x}) \approx 0$, making the block an identity function. This is easy to learn.
2. Gradient highway: The identity shortcut provides a direct path for gradients to flow back through the network, alleviating vanishing gradients.
3. Ensemble interpretation: ResNets can be viewed as an ensemble of many shorter networks.

Residual block: the skip connection bypasses the stacked weight layers, adding the input $\mathbf{x}$ to the learned residual $F(\mathbf{x})$.

Residual Block Variants

Left: basic residual block (two 3×3 convolutions) for ResNet-18/34. Right: bottleneck block (1×1 reduce, 3×3 process, 1×1 restore) for ResNet-50/101/152.

Architecture comparison: VGG-19 (left), 34-layer plain network (center), 34-layer residual network (right). The residual network achieves lower error despite being the same depth as the failing plain network.

Top-5 error: 4.49% (single ResNet-152), 3.57% (ensemble) — the first architecture to surpass estimated human-level performance (5.1%).

7. ResNet Variants and Inception Extensions

7.1 ResNet Pre-activation (2016)

He et al. proposed moving BN and ReLU before the convolution (pre-activation) rather than after:

• Original: Conv → BN → ReLU → Conv → BN → (+x) → ReLU
• Pre-activation: BN → ReLU → Conv → BN → ReLU → Conv → (+x)

The identity shortcut becomes a truly clean path (no BN or ReLU), enabling unimpeded gradient flow. ResNet-1001 with pre-activation: test error 4.92% vs. 7.61% original on CIFAR-10.

Pre-activation ResNet: (a) original BN-after-Conv ordering vs. (b) proposed BN-ReLU-Conv ordering. Pre-activation gives a cleaner gradient path through the shortcut.

7.2 Wide ResNet (2016)

Instead of going deeper, make the network wider (more feature maps per layer). Notation: WRN-$d$-$k$ has depth $d$ and width multiplier $k$.

• Increasing width consistently improves performance.
• Wider networks need dropout within residual blocks for regularization.
• WRN-28-10: CIFAR-10 error 4.0%, comparable to ResNet-1001 but with far fewer layers.

Ablation study: increasing width multiplier $k$ at fixed depth consistently reduces CIFAR-10/100 error. A 16-layer wide network can match very deep thin networks.

7.3 Inception v2 and v3

Szegedy et al. ("Rethinking the Inception Architecture") proposed two factorizations:

Step 1: Replace 5×5 with two stacked 3×3

Two stacked 3×3 convolutions have the same receptive field as one 5×5 but use fewer parameters ($2 \times 9 = 18$ vs. 25 per channel pair) and add an extra non-linearity.

Original 5×5 branch (left) replaced by two stacked 3×3 convolutions (right).

Step 2: Asymmetric factorization (n×n into 1×n and n×1)

A 3×3 convolution can be factorized into a 1×3 followed by a 3×1 convolution, reducing computation by ~33%.

Asymmetric factorization: $n \times n$ into $1 \times n$ then $n \times 1$ convolutions.

Inception v3 = Inception v2 + RMSprop optimizer + label smoothing regularization + factorized 7×7 in the stem + batch-normalized FC before auxiliary classifier.

Inception v3: stacked Inception-A (35×35), Inception-B (17×17), and Inception-C (8×8) blocks with reduction layers between stages.

7.4 Inception v4 and Inception-ResNet (2016)

Inception v4 introduced a cleaner, more uniform inception architecture. Inception-ResNet v1/v2 adds residual connections on top of inception blocks, enabling faster convergence.

Inception-ResNet-A module: three parallel branches combined via a 1×1 linear projection, then added to the input via a residual connection.

7.5 ResNeXt (2016)

ResNeXt (Xie et al.) introduces a new dimension: cardinality (the number of parallel transformation paths). The block follows a split-transform-aggregate pattern.

ResNet-50 bottleneck block (left) vs. ResNeXt-50 (32×4d) block (right). ResNeXt splits into 32 parallel paths of width 4, equivalent to grouped convolutions.

Increasing cardinality is more effective than increasing depth or width for improving accuracy at the same computational budget.

7.6 Squeeze-and-Excitation Networks (SENet, 2017)

SENet (Hu et al.) introduces a lightweight channel attention mechanism that re-weights feature map channels based on their global importance. It can be inserted into any existing architecture.

SE block: (1) transformation (standard conv), (2) squeeze via global average pooling, (3) excitation via two FC layers with sigmoid, (4) channel-wise scaling.

Standard ResNet block (left) vs. SE-ResNet block (right). The SE branch adds channel attention at only ~10% computational overhead.

SENet won ILSVRC 2017 with Top-5 error 3.79%.

8. DenseNet (2016)

DenseNet (Huang et al., "Densely Connected Convolutional Networks") extends ResNet's skip connections to the extreme: every layer receives feature maps from all preceding layers via concatenation.

For a network with $L$ layers there are $\frac{L(L+1)}{2}$ direct connections (vs. $L$ in a plain network).

Dense block: each layer receives the concatenation of all preceding feature maps, enabling explicit feature reuse throughout the block.

Architecture Details

• Growth rate $k$: Number of new feature maps produced per layer. Typical values: $k = 12, 24, 32$. Layer $l$ receives $k_0 + k(l-1)$ input channels.
• Dense blocks contain multiple layers with dense connectivity.
• Transition layers between blocks: BN → 1×1 Conv (channel compression) → 2×2 Average Pooling.
• DenseNet-BC: Adds a 1×1 bottleneck before each 3×3 conv (B) and compresses channels at transitions by factor $\theta = 0.5$ (C).

DenseNet full architecture: initial convolution, three dense blocks separated by transition layers, global average pooling, and linear classifier.

DenseNet vs. ResNet

DenseNet achieves comparable accuracy to ResNet with significantly fewer parameters and FLOPs, at the cost of higher memory consumption.

9. NASNet (2017)

NASNet (Zoph et al., "Learning Transferable Architectures for Scalable Image Recognition") uses machine learning to search for optimal architectures, replacing hand-crafted design with reinforcement learning.

Search Process

NAS overview: a controller RNN samples architectures, child networks are trained to get validation accuracy $R$, and the controller is updated via REINFORCE.

1. A controller RNN samples an architecture from the search space with probability $p$.
2. The proposed child network is trained to obtain validation accuracy $R$.
3. The gradient of $p$ is scaled by $R$ to update the controller (REINFORCE algorithm).
4. Repeat: the controller gradually learns to propose better architectures.

Cell-Based Search

Rather than searching for the full architecture, NASNet searches for two types of cells:

• Normal Cell: Retains spatial dimensions (same H×W).
• Reduction Cell: Downsamples spatial dimensions by factor of 2.

NASNet-A Normal Cell (left) and Reduction Cell (right) found by the controller RNN. These are stacked to build the full architecture.

Performance

• NASNet outperforms all hand-crafted networks on ImageNet at the same accuracy level.
• Lower FLOPs compared to state-of-the-art human-designed architectures.
• Top-5 error: 3.8% on ImageNet.
• Training cost: 500 GPUs for 4 days (search phase).

NASNet-A accuracy vs. number of operations and parameters on ImageNet. NASNet achieves better accuracy at lower compute than manually designed competitors.

10. Other Notable Architectures

SqueezeNet — AlexNet accuracy at 50x fewer parameters

SqueezeNet uses fire modules (squeeze 1×1 conv + expand mix of 1×1 and 3×3 convs) with bypass connections. AlexNet-level accuracy with ~50x fewer parameters, making it suitable for mobile/edge deployment.

SqueezeNet architecture with fire modules and bypass connections between fire2–fire3 and fire4–fire5.

Capsule Networks — Routing by agreement

Proposed by Hinton et al. Key differences from standard CNNs: (1) vector output (capsules) instead of scalar activation — magnitude represents probability of entity existence, orientation represents entity properties. (2) Routing by agreement instead of max-pooling, preserving spatial relationships between object parts. (3) Potentially requires less training data.

Capsule network: Conv1 extracts local features, PrimaryCaps creates capsule vectors, DigitCaps uses routing by agreement to classify.

11. Chronological Summary and Design Paradigms

Design Paradigm Evolution

Flashcards