12. 3D Data Analysis

PointNet, permutation invariance, MVCNN, VoxNet, 3D segmentation

1. Challenges of 3D Data Analysis

3D data analysis extends the three core 2D vision tasks into 3D: classification (one label per object), object detection (locate and classify multiple objects), and semantic segmentation (label every single point or voxel).

3D classification: point cloud objects (sofa, chair, desk, monitor, table) each labeled with their class.

Semantic segmentation: an indoor room point cloud (left: raw) and per-point class labels (right: walls blue, floor brown, furniture in other colors).

Six Key Challenges

Same car point cloud at 256, 512, 768, and 1024 points. In each case point ordering is arbitrary; only spatial distribution matters.

2. Three Families of Approaches

The same 3D scene as a raw point cloud (left) and as a colored voxel grid (right).

3. Multi-View Projection: MVCNN

MVCNN (Multi-View Convolutional Neural Network) is the pioneering projection-based method. It leverages pre-trained 2D CNN models — critical because labeled 3D training data is scarce compared to massive 2D datasets like ImageNet.

MVCNN Architecture for Classification

MVCNN: a 3D shape rendered from 12 virtual viewpoints, processed by a shared CNN, aggregated via view pooling, then classified.

1. Multi-view rendering: Render the 3D shape from 12 virtual camera viewpoints, producing 12 standard 2D images.
2. Per-view feature extraction (CNN_1, shared weights): Each of the 12 images passes through the same CNN (e.g., VGG pre-trained on ImageNet). All 12 views share identical weights.
3. View pooling (element-wise max): Aggregate 12 feature vectors into a single descriptor:
   $$\mathbf{d} = \text{ViewPool}(\mathbf{f}_1, \ldots, \mathbf{f}_{12}) = \max(\mathbf{f}_1, \ldots, \mathbf{f}_{12})$$
   where max is applied element-wise across the feature dimension.
4. Classification (CNN_2 + FC): The pooled feature passes through CNN_2 and fully connected layers to produce class probabilities.

MVCNN Fusion Strategies

Three multi-view fusion strategies: early fusion (combine after initial layers), late fusion (combine after full per-view processing), score fusion (combine at prediction level).

4. SnapNet: Multi-View 3D Segmentation

SnapNet extends the multi-view idea from classification to semantic segmentation of point clouds.

SnapNet full pipeline: mesh view generation, semantic labeling with autoencoder, back-projection to 3D points.

1. View generation: Place virtual cameras around/within the scene; synthesize RGB and depth composite texture images from each camera.
2. Semantic labeling: Feed each RGB-depth image pair into an encoder-decoder segmentation network, producing per-pixel class labels.
3. Back-projection and voting: Map 2D per-pixel labels back onto 3D points. When multiple views see the same 3D point, accumulate votes and take the majority class.
4. Output: A fully semantized point cloud where every point has a class label.

SnapNet rendering: (a) RGB texture and (b) depth composite texture synthesized from virtual cameras placed around the scene.

5. Volumetric Methods: VoxNet and 3D CNNs

VoxNet is the pioneering volumetric method. It converts the raw point cloud to a regular 3D voxel grid, then applies 3D CNNs — solving the problem of irregular, unstructured point clouds by imposing a regular grid like pixels in an image.

Voxelization: The Critical Trade-off

Coarse vs. fine voxel quantization: coarse grids are fast but delete detail; fine grids preserve detail but memory grows cubically.

Octree: Memory-Efficient Voxels

An octree recursively subdivides 3D space into 8 octants, only where data exists. Empty regions are never subdivided — concentrating resolution where data actually is while saving enormous memory on empty space.

Point cloud (left) converted to an octree-like voxel representation (right): from a single bounding cube progressively subdivided to Level 6.

3D Spatial Convolution

3D spatial convolution: a 3D kernel cube sliding through a larger 3D data volume in all three spatial directions.

3D CNN classification pipeline: 3D data cube through 3D convolution + pooling stages, reshaping to a flat vector, then dense layers for classification.

6. Direct Point Cloud: PointNet

Multi-view and volumetric approaches are artificial workarounds that convert point clouds to other representations, introducing information loss and computational overhead. PointNet analyzes point clouds directly as they are.

PointNet: a unified framework accepting raw point clouds and supporting three tasks: object classification, part segmentation, and semantic scene parsing.

The Core Design Problem

How do you design a neural network whose output is invariant to the ordering of its inputs? Traditional networks expect fixed-order inputs (pixel $(0,0)$ is always top-left). A point cloud is a set — the same geometry listed in any order must produce the same output.

7. PointNet Architecture Walkthrough

Full PointNet classification architecture: input transform (T-Net), shared MLPs, feature transform (second T-Net), max pooling to global feature, and classification MLP.

T-Net (Spatial Transformer Network)

The T-Net is a sub-network that predicts a transformation matrix to align the point cloud to a standard canonical pose. Objects in real scans can appear in arbitrary orientations; the T-Net learns to undo this.

T-Net transformation: (a) the same car overlaid in random orientations before alignment; (b) all copies aligned to a single canonical orientation after applying the learned transformation.

T-Net architecture: shared MLPs (3 to 64 to 128 to 1024), max pool to global vector, FC layers (1024 to 512 to 256), then matrix multiply with trainable weights to produce a 3x3 transformation matrix.

Max Pooling: The Key to Permutation Invariance

After shared MLP Block 2 maps each of the $n$ points to 1024 dimensions, max pooling is applied across all $n$ points for each feature dimension:

This produces a single 1024-dimensional global feature vector describing the entire point cloud.

PointNet Dimension Table

PointNet for Segmentation

For per-point class labels, PointNet concatenates the global feature vector with each point's local feature (from shared MLP block 1, 64-dimensional):

Per-point MLPs then predict a class label for each point. This gives every point both local context (its own features) and global context (the entire object's feature).

PointNet part segmentation: each point in the car point cloud labeled by part class (body, wheels, windows, etc.).

Feature Transform Regularization

The second T-Net produces a $64 \times 64$ matrix. To prevent it from being arbitrary, a regularization loss encourages it to be close to an orthogonal matrix:

The total loss is $L = L_{\text{cls}} + \lambda \cdot L_{\text{reg}}$ where $L_{\text{cls}}$ is standard cross-entropy loss.

8. Advanced Point-Based Architectures

In the 4 years following PointNet, approximately ~300 point-based 3D analysis networks were published. They address PointNet's main limitation: it does not capture local geometric structure — max pooling creates a global feature but ignores relationships between nearby points.

PointNet++ Key Idea

PointNet++ applies PointNet hierarchically in local neighborhoods, mirroring how CNNs learn local then global features:

1. Farthest Point Sampling (FPS): Select well-spread subset of points.
2. Ball Query: For each selected point, find all points within radius $r$.
3. Mini-PointNet on local patches: Apply PointNet to each neighborhood to extract a local feature.
4. Repeat hierarchically: Use outputs as inputs to the next level, capturing progressively larger-scale patterns.

9. Comparison of All Three Families

Flashcards