8. 3D Sensors & Depth Sensing

Stereo vision, Time-of-Flight, structured light, LiDAR, 4D radar, sonar, and 3D data formats

1. Taxonomy of 3D Sensors

The real world is three-dimensional, yet 2D images discard all depth information — a person at 2 m and at 20 m can project to the same pixel pattern. 3D sensing restores the lost depth dimension, enabling applications impossible with 2D data alone.

All 3D sensors divide into two fundamental categories based on whether they emit their own energy signal:

Supportive Sensors

Active and passive 3D sensors are often combined with supportive sensors that do not measure depth directly but provide auxiliary position and orientation information:

• GPS — global position (latitude, longitude, altitude)
• Odometers — distance traveled from wheel rotation counts
• IMU (Inertial Measurement Unit): accelerometers (linear acceleration), gyroscopes (rotational velocity), magnetometers (heading), altimeters (elevation)

These are essential for aerial LiDAR (GPS/IMU tell you where the aircraft was oriented when each pulse fired) and mobile mapping.

2. Stereo Vision & Disparity

Stereo vision exploits the same principle as human depth perception: two slightly offset viewpoints create parallax — nearby objects shift more between the two views than distant objects.

Binocular disparity: when fixating on one object, others appear at slightly different visual angles in each eye. Larger angular difference = closer object.

Stereo Disparity Equation

Consider left and right cameras separated by baseline $b$. A 3D point projects to pixel $p_L$ in the left image and $p_R$ in the right image. The disparity is $d = p_L - p_R$.

Derivation via similar triangles: left camera projects point $P = (X, Y, Z)$ to $p_L = f(X + b/2)/Z$, right camera to $p_R = f(X - b/2)/Z$. Disparity $d = p_L - p_R = fb/Z$, so $Z = fb/d$.

Key Relationships

Disparity-to-depth: $z = (b \cdot f) / (p \cdot d)$. Brighter pixels in the disparity map = higher disparity = closer objects.

Disparity Map

By finding correspondences across the full image pair, we obtain a disparity map where each pixel value encodes the horizontal displacement between left and right views:

• Bright pixels = high disparity = close to camera
• Dark pixels = low disparity = far from camera
• Black/undefined = no match found

A raw disparity map: depth values found only for textured areas; significant noise. Brighter = closer.

Depth enhancement: raw disparity map (left) vs. enhanced map after watershed/inpainting (right).

Stereo Enhancement Techniques

ZED Mini stereo camera: RGB views (top) and corresponding depth maps (bottom). Passive stereo fails in low light and on textureless surfaces.

3. Time-of-Flight (ToF) Sensors

A ToF sensor measures depth by timing how long a light signal takes to travel to a scene point and return. It floods the entire scene with modulated infrared (IR) light and measures the phase shift independently at every pixel simultaneously — producing a dense depth map at real-time rates.

Direct nanosecond timing is impractical for consumer electronics. Instead, ToF sensors use continuous-wave modulation: the sensor emits a periodic signal, the reflected signal arrives with a phase delay, and the sensor correlates the sent and received signals to extract the phase shift (and thus the distance).

ToF correlation: close objects (left) produce short delays with closely aligned correlation peaks; far objects (right) produce longer delays with shifted peaks.

ToF Hardware — Kinect One (v2)

Kinect One (v2) teardown: RGB camera (red), IR sensor array (green), and IR illuminator (blue). The illuminator and sensor work as a pair to measure per-pixel phase shift.

Depth map from a ToF sensor: brighter = closer, darker = farther. Dense, per-pixel depth at real-time frame rates.

4. Structured Light Sensors

Structured light computes depth by active triangulation: a projector emits a known pattern (stripes, dots, grid) onto the scene; a camera at a known distance from the projector observes how the pattern deforms on 3D surfaces. The deformation encodes depth.

Structured light triangulation: a stripe projector (left) projects light stripes onto a shaped object (hand). A matrix camera (right) observes the deformed pattern. Known geometry of the triangle (baseline, projection angle, observation angle) yields depth.

Triangulation Geometry

The unknown is depth — the distance to the surface where the pattern element landed. Basic trigonometry from the projector-camera-surface triangle yields it.

Kinect v1 internal components: IR Emitter, Color Sensor, IR Depth Sensor, Tilt Motor, and Microphone Array. The Kinect v1 projects ~30,000 IR dots and observes their displacement.

5. Passive vs Active Sensors

6. LiDAR

LiDAR stands for Light Detection And Ranging (or Laser Imaging Detection And Ranging). It uses the same time-of-flight principle as ToF cameras but with a focused laser beam instead of a flood of modulated IR — enabling much longer range and higher accuracy.

Why Laser Light Is Special

LiDAR Scanning Mechanisms

LiDAR scanning mechanisms: oscillating mirror (sawtooth, most common), rotating polygon (parallel lines), nodding mirror (elliptical).

Velodyne LiDAR (Multi-beam, Real-time)

The Velodyne contains 8, 16, or 64 laser beams at different vertical angles that spin together via a rotating mechanism, producing full 360-degree scans 10–20 times per second.

Velodyne multi-beam LiDAR: 8/16/64 beams fire at different vertical angles from a rotating sensor, producing a 360-degree point cloud in real time.

Point cloud from a Velodyne LiDAR scan, colored by reflectivity/height. Road, vehicles, and surroundings are clearly resolved.

Faro LiDAR (Single-beam, High Precision)

The Faro uses a single laser beam with a rotating mirror, taking 2–20 minutes per scan but achieving millimeter precision. Used for architectural documentation and cultural heritage scanning.

Faro LiDAR scan of a cathedral interior: every architectural detail — vaulted ceilings, columns, arches — captured with millimeter precision.

Aerial LiDAR

LiDAR mounted on aircraft or drones, combined with GPS (absolute position) and IRS/IMU (aircraft orientation at each pulse), for topographic mapping, forestry, and urban planning. A GPS ground base station provides differential correction.

Airborne LiDAR: aircraft fires pulses at ground while GPS and IRS/IMU track position and orientation. A GPS base station provides differential correction.

7. 4D Imaging Radar

A 4D imaging radar uses electromagnetic (EM) pulses (millimeter-wave radio waves) and extracts four dimensions of information from each detection.

4D imaging radar: measures distance (up to 300 m), direction/azimuth, vertical height, and Doppler speed simultaneously.

The Doppler Effect (4th Dimension)

When an object moves toward the radar, reflected radio waves are compressed (shifted to higher frequency). When moving away, waves are stretched (lower frequency). The magnitude of the frequency shift is directly proportional to the object's radial velocity.

This is the critical differentiator of 4D radar: it senses not only where objects are but also how fast they are moving — invaluable for predicting whether a car ahead is braking, accelerating, or stationary.

4D radar in fog: camera image is nearly useless (blurry, low contrast), but the radar heatmap still clearly detects and localizes the car. Radio waves are largely unaffected by fog, rain, and dust.

8. Multi-beam Sonar

Multi-beam sonar uses acoustic pulses (sound waves) for underwater 3D sensing. Light and radio waves attenuate rapidly underwater and are useless beyond a few meters — only acoustic waves propagate effectively.

Multi-beam sonar: a vessel emits a fan of acoustic beams downward; beams bounce off the seabed and return to hydrophones on the hull.

Sonar transducer: piezoelectric ceramic element (converts electrical signals to acoustic pulses), aluminum resonator, elastic adhesive, connector.

Sonar provides 3 dimensions of information: distance (range), direction (beam angle), and seabed material type (inferred from acoustic return character — hard rock reflects sharply; soft mud absorbs and scatters).

Multi-beam sonar seabed mapping (British Antarctic Survey): color-coded bathymetric map revealing 3D topology of the ocean floor — blue = deep, red = shallow.

9. Sensor Comparison

Multi-objective comparison of depth sensing technologies: stereoscopic vision, structured light, and Time-of-Flight.

Radar chart: Kinect (depth sensor) vs FARO (LiDAR). No single sensor dominates all axes — the right sensor depends on your application.

Kinect vs Faro — Trade-off Summary

10. 3D Data Representation Formats

Once depth data is acquired, it must be stored and processed in a digital representation. Eight formats exist, each with fundamentally different trade-offs.

Overview of 3D data representation formats: voxel grid, octree, point cloud, mesh, RGB-D, multi-view, NeRF, and Gaussian Splatting.

Voxel Grid

A regular 3D grid of cubic cells (voxels — volumetric pixels). Each cell is empty (0) or occupied (non-zero). Analogy: a 2D image is a grid of pixels; a voxel grid is a 3D grid of volume elements.

• Pros: Easy sensor data aggregation; volumetric; good for physics simulation
• Cons: Requires lots of memory — even completely empty cells must be stored; no GPU hardware support for efficient rendering

Voxel grid: cells containing objects have non-zero values; vast majority of cells (air) are empty but still stored — memory wasteful.

Octree

A hierarchical tree that adaptively subdivides 3D space. The root is one large cube; only regions needing detail are recursively divided into 8 sub-cubes (octants). Homogeneous regions remain undivided.

• Pros: Memory-efficient (1000x less than voxel grid for typical scenes); volumetric; easy sensor aggregation
• Cons: Additional processing overhead when updating (tree restructuring); no GPU rendering hardware support

Octree of the Stanford Bunny: surface regions are finely subdivided; interior and distant exterior regions are large, undivided cells.

Point Cloud

An unordered set of 3D points $(X_i, Y_i, Z_i)$ representing object surfaces. Points may carry additional attributes: color, intensity, surface normal, classification. This is the native output of LiDAR sensors.

• Pros: Easy sensor aggregation (just append points); can accurately represent complex shapes
• Cons: No surface connectivity; high memory (12+ bytes per point, millions of points); difficult to convert to mesh

Point cloud: the Stanford Bunny as a dense cloud of colored 3D points — shape is clear but no surface connectivity exists.

Polygonal (Mesh) Model

Vertices, edges, and faces (triangles) form the standard computer graphics representation. GPUs are fundamentally designed to process triangles at extreme throughput.

• Pros: Best for GPU rendering and photorealism (native GPU format); compact memory; supports texturing, shading; good for physics simulation
• Cons: Does NOT allow direct sensor data aggregation — converting raw sensor data to a clean mesh (surface reconstruction) is expensive and error-prone

Polygonal mesh: Stanford Bunny at increasing resolution from coarse (few large triangles) to fine (thousands of small triangles).

RGB-D Image

A color (RGB) image paired with a depth (D) image, pixel-aligned. The direct output of ToF cameras and structured light sensors. Brighter depth pixels = closer.

• Pros: Direct sensor output; easy to visualize; low memory
• Cons: Captures only one viewpoint (technically "2.5D"); requires camera pose knowledge to aggregate multiple frames into a 3D model

RGB-D depth image: the Stanford Bunny as a grayscale depth map. Brighter = closer. Only the front-facing surface is captured — this is 2.5D, not full 3D.

Multi-view Sequences

RGB images from multiple camera viewpoints with known camera poses. No depth is directly measured — just photos from many angles.

• Pros: Low memory (just 2D images + pose metadata)
• Cons: No 3D model as-is; requires further processing (SfM, MVS, NeRF, Gaussian Splatting); not suitable for large complex scenes without dense coverage

Multi-view sequences: cameras positioned around a scene with known poses. These are the input to NeRF and Gaussian Splatting.

NeRF (Neural Radiance Field)

A neural network (MLP) that learns a continuous 3D scene representation. Given any 3D position and viewing direction, it outputs color and density. The entire scene is "baked" into the network weights.

NeRF: a 5D function mapping spatial position (3D) and viewing direction (2D) to color and density via a fully-connected neural network.

NeRF architecture: 8 layers of 256 channels process positional-encoded $(x,y,z)$ to output density $\sigma$. A separate 128-channel branch takes the 256-dim feature plus viewing direction and outputs color $(r,g,b)$.

NeRF pipeline: multi-view images with known camera poses are used to train the MLP, which can then render photorealistic novel views from any viewpoint.

Gaussian Splatting

Represents a 3D scene as a collection of 3D Gaussian ellipsoids, each with learnable position, shape (covariance), color (spherical harmonics), and opacity. Gaussians are projected ("splatted") directly onto the 2D image plane — much faster than NeRF's per-ray MLP queries.

Gaussian Splatting pipeline: SfM points initialize 3D Gaussians, which are projected via camera parameters, rendered with a differentiable tile rasterizer, and refined through gradient flow with adaptive density control.

Adaptive density control: cloning fixes under-reconstruction (top row); splitting fixes over-reconstruction (bottom row).

Gaussian Splatting: a scene represented as 3D Gaussian ellipsoids with learnable position, shape, color, and opacity.

3D Format Comparison Summary

Flashcards