Production-Grade 4D Auto-Annotation: Multi-Camera Rig SfM & High-Precision Geometry

Project context

My contributions

Led a 5-person algorithm team engineering the multi-camera 3D reconstruction and 4D annotation pipeline; processed 1,000+ km multi-run fleet data with sub-5 cm cross-run spatial alignment, boosting annotation efficiency by over 80%.

As autonomous driving perception models shift from single-frame monocular detection toward bird’s-eye-view (BEV) architectures, 3D occupancy networks, and end-to-end foundation models, the availability of massive, temporally consistent 4D ground truth—encompassing metric road surfaces, 3D lane topologies, and static infrastructure—has become the definitive catalyst for algorithmic advancement.

Conventional workflows relying on mobile LiDAR scanning paired with manual outsourcing suffer from long turnaround times, high labeling costs, and limited generalization across heterogeneous sensor layouts. However, naively transplanting classical Structure-from-Motion (SfM) pipelines directly onto vehicle surround-view video streams immediately encounters severe physical and mathematical degeneracies: collinear motion corridors, textureless road surfaces, moving self-vehicle shadows, and unconstrained camera extrinsics drift.

In partnership with leading electric vehicle manufacturer NETA Auto, we developed and deployed an automotive-grade 4D vision auto-annotation data factory. As Project Tech Lead, I directed the overall algorithmic architecture and led the development of the core rigid multi-camera rig SfM reconstruction engine, ground-feature decoupling, spatiotemporal-guided dense multi-view stereo, and multi-session cross-epoch aggregation. The core geometric optimization principles and engineering systems developed during this industrial deployment were subsequently published at MRASfM (ICRA 2026).

Why Driving Scenarios Break Classic SfM

In standard photogrammetry textbooks, classic SfM frameworks (such as COLMAP or VisualSFM) are built on the foundational assumption of an operator walking around an isolated, static monument with a single camera, capturing omnidirectional views with wide convergent baselines. Onboard vehicular data collected on highways and urban thoroughfares violates nearly every one of these assumptions:

1. Straight Corridor Motion & Collinear Triangulation Degeneracy: On highways and arterial avenues, vehicles travel along straight corridors for kilometers at a time. The optical centers of front and rear cameras align along a 1D straight line. In multi-view epipolar geometry, pure forward translation with collinear centers is ill-conditioned. When moving directly forward along the optical axis, the epipole coincides with the image principal center. Rays originating near the optical axis intersect at near-zero parallax angles. Sub-pixel feature matching noise is amplified along the longitudinal axis, causing reconstructed 3D points to stretch and drift along the direction of travel.

2. Extrinsics Drift in Unconstrained Multi-Camera Optimization: Production autonomous vehicles feature 6 to 8 surround cameras rigidly bolted to the vehicle body (front wide, front telephoto, front-left, front-right, rear-left, rear-right). Standard SfM treats each camera at each timestamp as an unconstrained 6-DoF pose variable. For an $N$-frame sequence, this introduces $6N$ independent pose variables. As pairwise feature matching accumulates noise, unconstrained optimization causes the physically rigid sensor extrinsics to artificially drift over time, introducing severe ghosting and discontinuous stitching artifacts in downstream BEV projections.

3. Textureless Asphalt & Moving Self-Vehicle Shadows: The lower half of onboard camera images is dominated by asphalt surfaces covering $>40\%$ of the image area. Asphalt is visually homogeneous, lacking sharp corner features under direct sunlight. Even more destructive is the vehicle’s dynamic shadow cast onto the road surface: the shadow boundary presents high-contrast, sharp edges that classical feature detectors (such as SIFT or Harris) aggressively extract as salient landmarks. However, the shadow translates synchronously with the vehicle, violating the static scene assumption. Triangulating shadow features forces the optimizer to fit non-existent points beneath the chassis, pulling vehicle trajectory estimates off course.

4. Multi-Session Cross-Epoch Aggregation Without Shared Image Matches: Autonomous driving maps require merging multiple independent drives through the same intersection collected across different days (e.g., overcast Monday morning vs. sunset Friday evening). Variable solar angles, changing shadows, and transient traffic mean disparate driving runs share zero visual feature correspondences, leaving independent reconstructions geometrically isolated.

Full Pipeline Architecture: From Raw Logs to 4D Vector Maps

To process tens of thousands of kilometers of production fleet video logs with high throughput, we established a decoupled, fail-safe processing pipeline:

• Multi-Sensor Spatiotemporal Synchronization: Aligning six surround cameras, high-rate IMUs, GNSS receivers, and wheel odometry into a common timestamp reference and vehicle rear-axle coordinate frame;
• Dynamic Semantic Masking: Deploying lightweight segmentation models to isolate moving vehicles, cyclists, pedestrians, and sky regions, shielding static 3D geometry from dynamic clutter;
• Fused Inertial-GNSS Initialization: Fusing satellite positioning and dead-reckoning to initialize metric-scale trajectory guesses;
• Rig-Centric Multi-Camera SfM (Core Engine): Treating surround cameras as a unified rigid rig to jointly solve vehicle poses and calibrate sensor extrinsics, generating a centimeter-level sparse road skeleton;
• Spatiotemporal-Guided Dense MVS: Leveraging motion geometry priors to prune unshared image pairs, accelerating PatchMatch multi-view stereo to produce dense ground meshes;
• Multi-Session Cross-Epoch Aggregation: Aligning independent driving passes coarse-to-fine into a globally seamless local high-definition map.

Core Algorithmic Design & Engineering Implementation

1. Rig Bundle Adjustment (Rig BA) with Rigid-Rig Parameterization

To eliminate extrinsics drift and reduce optimization dimensionality, we reformulated the multi-view optimization variables: rather than treating every frame as an independent pose, all surround cameras captured at timestamp $t$ are grouped into a rigid camera set. A camera’s absolute world pose is factored into a time-varying vehicle chassis pose and a static chassis-to-camera mounting extrinsic.

Under this parameterization, the number of pose variables collapses dramatically. We formulate the Rig Bundle Adjustment (Rig BA) framework to jointly optimize vehicle chassis trajectories, mounting extrinsics, and 3D static landmark coordinates.

This rigid formulation delivers two decisive advantages:

• Complementary Viewpoint Geometry: When the vehicle drives into direct sun glare or passes alongside large trucks, forward cameras may suffer severe overexposure or visual occlusion. However, lateral and rear cameras continue tracking static architecture, trees, and street furniture. Through fixed internal rig extrinsics, unobstructed cameras constrain the chassis pose, eliminating longitudinal depth stretching;
• Online Extrinsic Self-Calibration: By treating mounting extrinsics as optimization parameters, the solver refines mounting extrinsics over multi-kilometer runs to high precision, absorbing mechanical chassis flexing.

2. Spatial Overlap Constraints & Epipolar Pruning

Exhaustive pairwise matching across $N$ timestamps and $M$ cameras incurs an intractable $O(M \cdot N^2)$ computational explosion.

We introduced structured camera rig overlap priors:

• A front-wide camera and front-left camera share only $\sim 30^\circ$ of horizontal FOV overlap. The right half of the front camera image cannot geometrically intersect the front-left camera’s viewing frustum;
• Restricting feature searches strictly to overlapping spatial frustums ($u < W/2$ on the front camera) eliminates false associations between visually repetitive facades;
• Enforcing Essential matrix $E$ epipolar distance checks and Homography $H$ planar constraints removes $>95\%$ of outlier correspondences while reducing matching runtime by $80\%$.

3. Decoupled Ground-Plane Filtering Prior to Triangulation

To prevent dynamic vehicle shadows and weak-texture asphalt from corrupting 3D point clouds:

• Within the coarse sparse point cloud, we partition the road corridor into 5-meter sliding longitudinal cells;
• Locally Optimal RANSAC (LO-RANSAC) fits local tangent ground planes $n_{\text{road}}^\top X + d = 0$;
• Feature points residing within a 10 cm slice of the fitted plane are tagged and excluded from sparse bundle adjustment. The bundle adjustment solver converges purely on high-elevation rigid structures (building facades, light poles, overhead sign gantries), eliminating shadow-induced trajectory bias.

4. Spatiotemporal-Guided Dense PatchMatch Stereo (MVS)

Generating continuous 3D ground meshes and voxel grids for occupancy networks requires high-throughput dense multi-view stereo:

• Using the estimated vehicle trajectory, image pairs are screened based on optical parallax angles $\theta_{\text{parallax}} \in [5^\circ, 30^\circ]$ within a sliding temporal window of $\pm 15$ frames, avoiding global point-cloud searches;
• GPU-accelerated multi-view PatchMatch (ACMP) iteratively optimizes depth maps and surface normal fields, employing slanted plane support priors to interpolate dense, continuous depth across textureless asphalt;
• A multi-view geometric consistency filter fuses depth estimates across all six surround cameras into unified metric point clouds.

Standard COLMAP Dense

ACMP + Optimized Fusion

5. Coarse-to-Fine Multi-Session Aggregation Network

To reconcile asynchronous passes through identical intersections collected on different dates:

1. Coarse Georeferencing: Differential RTK-GNSS coordinates anchor each pass into a common metric coordinate system;
2. Fine Geometric Registration: The system isolates invariant vertical landmarks (traffic light poles, bridge pillars, structural facades). PnP and RANSAC solve the inter-session rigid transformation $T_{\text{run1} \to \text{run2}}$;
3. Global Pose Graph Relaxation: Fixing the primary reference pass, all subsequent passes are formulated as nodes connected by relative geometric constraints in a global pose graph. Non-linear least-squares optimization absorbs inter-session drift, stitching disparate runs into a seamless high-definition map.

Direct Overlay Before Aggregation

After Multi-Session Aggregation

Production Quality Assurance & Field Validation

The full-stack pipeline was validated in production data centers against ground-truth 128-beam mobile LiDAR SLAM and RTK reference trajectories:

1. Dense Surface Accuracy: Reprojecting dense visual point clouds onto forward-facing camera images and overlaying synchronized LiDAR sweeps demonstrated absolute road elevation errors under $\pm 3\text{ cm}$ within a 50-meter range, with sharp curb profiles;
2. Trajectory Consistency: Over 5-kilometer continuous city and highway loops, pure visual Rig BA suppressed cumulative yaw drift to 0.05°/100m, keeping closed-loop position errors below 0.1% of total travel distance;
3. Extrinsic Alignment Continuity: Applying optimized extrinsics to 360° BEV stitching eliminated lane marking shear across camera seams, restoring smooth physical continuity.

Pipeline Video Recordings

The four video sequences below present the end-to-end journey from raw onboard video and high-throughput dense depth generation to static 3D point cloud extraction and pixel-level reprojection checks:

1. Onboard Surround Video Stream: Synchronized six-camera footage recorded by production vehicles navigating dense city traffic;
2. Static Scene 3D Reconstruction: Dense, crisp geometric point-cloud models of overpasses, street furniture, and road structures;
3. High-Accuracy Dense Depth Estimation: Real-time dense disparity and depth maps generated across all views via spatiotemporal PatchMatch;
4. Appearance & Geometric Reprojection: Pixel-level reprojection checks confirming geometric alignment against original camera frames.

Engineering Summary & Technical Takeaways

Scaling Structure-from-Motion and multi-view stereo to automotive-grade automated annotation demonstrates key engineering principles:

1. Mechanical Rigidity as an Algorithmic Anchor: Multi-camera physical rigidity must be embedded directly into optimization formulations. Grouping surround cameras into a rigid rig halts extrinsic drift and overcomes collinear triangulation degeneracies through complementary multi-directional views.
2. Semantic Decoupling Guards Against Environmental Noise: In the presence of textureless asphalt and moving shadows, accepting all features indiscriminately destabilizes optimization. Decoupling ground features from vertical structures ensures robust convergence.
3. Bridging Isolated Trajectories into Spatial Networks: Autonomous driving requires scalable, repeatable mapping. Linking independent driving sessions through topological pose graphs establishes the foundation for large-scale digital twins and end-to-end driving models.