6. Object Detection

R-CNN family, YOLO, SSD, anchor boxes, FPN, RetinaNet, focal loss, NMS, IoU, mAP

1. Problem Definition

Object detection is one of the core problems in computer vision. Given an input image, the aim is to detect and localize all objects of interest within that image, outputting a list of detections where each detection consists of:

• A class label (e.g., "dog", "car")
• A bounding box defined by coordinates $(x, y, w, h)$ or $(x_1, y_1, x_2, y_2)$
• A confidence score $P$ indicating certainty

Image classification (left) assigns a single label to the whole image. Object detection (right) localizes multiple objects with bounding boxes, class labels, and confidence scores.

Two Families of Deep Learning Detectors

2. Applications of Object Detection

Object detectors can be used alone or combined with other techniques:

• Self-driving cars: detecting pedestrians, vehicles, traffic signs, obstacles
• Object tracking: detecting objects in each frame, linking detections over time
• Face detection: for recognition pipelines, camera autofocus
• Medical imaging: detecting tumors, lesions, anomalies
• Activity recognition: detect person first, then classify action
• Manufacturing: quality inspection, defect detection on production lines
• OCR: detecting text regions before reading
• Counting: crowd analysis, retail analytics

3. Challenges in Object Detection

4. Object Localization and Bounding Box Regression

Bounding Box Formats

• Corner coordinates: $(x_1, y_1, x_2, y_2)$ — top-left and bottom-right corners
• Center + size: $(x_c, y_c, w, h)$ — center point plus width and height

Adding a Regression Head

To predict bounding box coordinates, a CNN classifier is extended with 4 regression outputs in addition to the class scores:

Multi-task network: CNN backbone feeds into both a classification head (softmax, $C$ class scores) and a bounding box regression head (4 coordinate values).

Bounding box regression training: the network predicts 4 coordinate values; L2 loss measures the difference from ground truth coordinates.

5. Two-Stage Object Detectors

Two-stage detectors split detection into two phases: (1) Region Proposal — generate candidate regions that might contain objects, then (2) Classification and Refinement — for each proposed region, classify the object and refine the bounding box.

5.1 R-CNN (2014)

R-CNN (Girshick et al.) was the first successful deep learning object detector.

R-CNN pipeline: (1) input image, (2) ~2000 region proposals via Selective Search, (3) CNN feature extraction per proposal, (4) SVM classification + bbox regression + NMS.

R-CNN architecture: each proposal is independently warped to fixed size and passed through an ImageNet-pretrained CNN.

Selective Search

Selective Search generates ~2000 region proposals per image:

1. Over-segment the image based on pixel intensities (watershed).
2. Iteratively group adjacent segments by similarity (color, texture, size, shape).
3. At each iteration, add bounding boxes of merged segments to the proposal list.
4. This creates proposals from small to large in a bottom-up hierarchical approach.

Selective search: initial over-segmentation is iteratively merged into larger regions, generating proposals at multiple scales.

5.2 Fast R-CNN (2015)

Fast R-CNN (Girshick) addresses the R-CNN speed bottleneck with one key insight: pass the whole image through the CNN once, then extract features per proposal from the shared feature map.

Fast R-CNN: the full image passes through CNN once to produce a shared feature map. RoI Pooling extracts fixed-size features for each proposal from the shared map.

Remaining bottleneck: Selective Search still runs on CPU and takes ~2 seconds per image, dominating total runtime.

5.3 Faster R-CNN (2015)

Faster R-CNN (Ren et al.) eliminates the Selective Search bottleneck by introducing a learned Region Proposal Network (RPN) that shares the backbone feature maps with the detection head.

Faster R-CNN: RPN generates proposals from shared feature maps, eliminating the CPU-bound Selective Search. RPN and detection head share the same backbone.

Region Proposal Network (RPN)

• A fully convolutional network sliding a $3 \times 3$ window over the feature map.
• At each position, it predicts $k$ anchors of different shapes/sizes (typically $k=9$: 3 scales × 3 aspect ratios).
• For each anchor: 2 objectness scores (object vs. background) and 4 bounding box offsets.

RPN sliding window: at each position on the feature map, the RPN outputs $2k$ objectness scores and $4k$ bounding box coordinates for $k$ anchors of different shapes.

Full Faster R-CNN pipeline: CNN backbone → RPN (proposals) → RoI Pooling on shared feature map → FC layers → classification + bbox regression heads.

Speed: ~7 fps (140 ms/image) — near real-time and ~140x faster than the original R-CNN.

5.4 Mask R-CNN (2017)

Mask R-CNN (He et al.) extends Faster R-CNN to simultaneously perform instance segmentation in addition to object detection.

Mask R-CNN: extends Faster R-CNN with RoI Align (replaces RoI Pooling) and an additional mask prediction branch that outputs per-class binary masks.

RoI Pooling: snapping floating-point coordinates to integer grid positions causes spatial misalignment, which is tolerable for classification but harmful for pixel-level segmentation.

RoI Align: bilinear interpolation computes exact feature values at fractional positions, preserving precise spatial alignment needed for the mask branch.

Mask R-CNN outputs three predictions per RoI: class label, refined bounding box, and a binary segmentation mask ($14 \times 14 \times C$ for $C$ classes).

Mask R-CNN head: RoI features branch into (1) classification + box regression head via average pooling, and (2) mask head via upsampling with a 1×1 conv producing one mask per class.

5.5 R-FCN (2016)

R-FCN addresses the per-RoI FC layer computation bottleneck of Faster R-CNN using position-sensitive RoI Pooling.

R-FCN: 90 position-sensitive score maps (10 classes × 9 spatial positions) encode where object parts appear. For each RoI, each grid cell pools from its corresponding position-specific map.

Advantage: all heavy computation is shared in conv layers. Per-RoI computation is just lightweight pooling and voting — making R-FCN faster than Faster R-CNN for large numbers of proposals.

6. Single-Stage Object Detectors

Single-stage detectors skip the separate region proposal step entirely. They predict bounding boxes and class scores directly from feature maps in a single forward pass, enabling real-time detection.

6.1 SSD (Single Shot MultiBox Detector, 2016)

SSD (Liu et al.) uses multi-scale feature maps and predefined anchor boxes to detect objects at different sizes in a single forward pass.

SSD multi-scale concept: smaller feature maps (deeper layers, larger receptive field) detect larger objects; larger feature maps (earlier layers, smaller receptive field) detect smaller objects.

Architecture:

1. Base network: VGG-16 (up to conv5_3), FC layers replaced by extra conv layers.
2. 6 prediction layers at different spatial resolutions (multi-scale).
3. At each scale, for every spatial position, two $3 \times 3$ conv layers predict class scores and box offsets for each anchor.
4. All predictions across scales go through NMS to produce final detections.

SSD architecture: VGG-16 base network followed by extra feature layers, with predictions made at 6 different spatial scales.

Performance: 74.3 mAP on VOC2007 at 59 FPS (17 ms/image).

6.2 YOLO (You Only Look Once, 2016)

YOLO (Redmon et al.) frames object detection as a single regression problem: directly predict bounding boxes and class probabilities from the full image in one evaluation.

YOLO divides the image into an $S \times S$ grid. Each cell predicts $B$ bounding boxes with confidence scores and $C$ class probabilities.

YOLO architecture: 24 conv layers (GoogLeNet-inspired, alternating 1×1 and 3×3) progressively reduce spatial resolution, followed by 2 FC layers producing the $7 \times 7 \times 30$ output tensor.

7. Anchor Boxes

Anchor boxes (also called "priors" or "default boxes") are a fundamental concept used in both two-stage (Faster R-CNN) and single-stage (SSD, RetinaNet) detectors.

Anchor boxes: predefined boxes of various aspect ratios and scales are tiled across the image at each spatial position. The detector predicts offsets to transform these templates into accurate bounding boxes.

Offset parameterization:

• $x_{\text{pred}} = x_{\text{anchor}} + \Delta x$
• $y_{\text{pred}} = y_{\text{anchor}} + \Delta y$
• $w_{\text{pred}} = w_{\text{anchor}} \cdot e^{\Delta w}$
• $h_{\text{pred}} = h_{\text{anchor}} \cdot e^{\Delta h}$

Typical configurations:

• Faster R-CNN RPN: 3 scales × 3 aspect ratios = 9 anchors per position.
• SSD: 4 or 6 anchors per position across 6 feature map scales.

8. Feature Pyramid Network (FPN)

CNNs naturally produce feature maps at multiple scales. Deeper layers have smaller spatial resolution but richer semantics; earlier layers have higher spatial resolution but weaker semantics. FPN combines the best of both.

Comparison of feature pyramid approaches: (a) image pyramid — accurate but extremely slow; (b) single feature map — fast but misses small objects; (c) pyramidal hierarchy — inconsistent semantic strength; (d) FPN — best of both worlds.

FPN Architecture

1. Bottom-up pathway: Standard CNN forward pass. Feature maps at each stage: $C_2, C_3, C_4, C_5$ (high to low resolution).
2. Top-down pathway: Starting from deepest $C_5$: 2× upsample → element-wise add with lateral $1 \times 1$ conv of corresponding $C_k$ → produces $P_2, P_3, P_4, P_5$.
3. Predictions are made independently at each pyramid level $P_k$.

FPN detail: bottom-up encoder (left) with top-down decoder using 2× upsampling and 1×1 lateral connections. Element-wise addition merges semantic richness (from deep) with spatial precision (from shallow).

FPN with Faster R-CNN: RPN is run on all pyramid levels $P_2, \ldots, P_5$. Only one anchor scale is needed per level since different levels naturally handle different object sizes.

9. RetinaNet and Focal Loss

RetinaNet (Lin et al., 2017) is a single-stage detector that matches or exceeds two-stage detector accuracy through two innovations: an FPN-based architecture and focal loss.

The Class Imbalance Problem

Single-stage detectors evaluate 10,000–100,000 candidate locations per image. The vast majority are background (easy negatives), creating severe class imbalance that causes easy examples to dominate training gradients, preventing the network from learning to detect hard examples.

Focal loss curves for different values of $\gamma$. As $\gamma$ increases, the contribution of well-classified easy examples is progressively down-weighted, allowing the network to focus on hard, misclassified examples.

RetinaNet Architecture

FPN backbone + two shared subnets applied at every pyramid level:

• Classification subnet: 4× $3 \times 3$ conv with ReLU → final $3 \times 3$ conv with depth $K \times A$ (classes × anchors). Parameters shared across pyramid levels.
• Box regression subnet: 4× $3 \times 3$ conv with ReLU → final $3 \times 3$ conv with depth $4A$. Class-agnostic. Parameters shared across pyramid levels.

10. Non-Maximum Suppression (NMS)

NMS is a post-processing algorithm used by virtually all object detectors to remove redundant overlapping detections of the same object.

Key parameters:

• IoU threshold: Typical values 0.45–0.5. Lower = more aggressive suppression.
• Confidence threshold: Detections below minimum confidence are discarded before NMS.

11. Performance Evaluation

11.1 Intersection over Union (IoU)

Example: Predicted box area = 100, GT box area = 150, intersection = 80. Union = 100 + 150 − 80 = 170. IoU = 80/170 ≈ 0.47. Since 0.47 < 0.5, this would be a False Positive at IoU threshold 0.5.

IoU is used in three contexts:

• Training: Assign ground truth labels to anchors/proposals (anchors with IoU ≥ 0.5 are positives).
• NMS: Decide which overlapping detections to suppress.
• Evaluation: Decide whether a predicted box matches a GT box for computing precision/recall.

11.2 Mean Average Precision (mAP)

mAP is the most widely used metric for evaluating object detectors, based on the Area Under the Precision-Recall Curve.

Worked Example

5 detections for class "cat" (sorted by confidence: 0.98, 0.96, 0.90, 0.30, 0.10), 4 ground truths:

AP for "cat" = area under the curve formed by these (Recall, Precision) points.

12. Detector Comparison

mAP vs. GPU time scatter plot for various detectors and backbones. The accuracy-speed trade-off is clear: two-stage detectors (upper-left) are slower but accurate; single-stage (lower-right) are faster.

COCO test-dev mAP progression 2015–2024. Continuous improvement from Fast R-CNN (~20 mAP) to modern transformer-based models (~63–65 mAP).

Two-Stage vs. Single-Stage Trade-offs

Flashcards