4D Auto-Annotation | Chang Nie

Production-Grade 4D Auto-Annotation: Multi-Camera Rig SfM & High-Precision Geometry

Thu, 20 Jun 2024 00:00:00 +0000

Project context

NETA Auto Collaboration · Automotive 3D Reconstruction & 4D Annotation

Meeting the demand for massive high-precision training ground truth in autonomous driving, this project established an industrial data pipeline spanning sensor sync, semantic perception, fused localization, rig SfM, and multi-run map aggregation, substantially improving offline labeling throughput.

Production Deliverables & Automotive Validation: Automotive 4D auto-annotation production pipeline; processed 1,000+ km fleet data (sub-5 cm cross-run registration), online extrinsic self-calibration, and 80%+ labeling efficiency gain

My contributions

Led a 5-person algorithm team engineering the multi-camera 3D reconstruction and 4D annotation pipeline; processed 1,000+ km multi-run fleet data with sub-5 cm cross-run spatial alignment, boosting annotation efficiency by over 80%.

As autonomous driving perception models shift from single-frame monocular detection toward bird’s-eye-view (BEV) architectures, 3D occupancy networks, and end-to-end foundation models, the availability of massive, temporally consistent 4D ground truth—encompassing metric road surfaces, 3D lane topologies, and static infrastructure—has become the definitive catalyst for algorithmic advancement.

Conventional workflows relying on mobile LiDAR scanning paired with manual outsourcing suffer from long turnaround times, high labeling costs, and limited generalization across heterogeneous sensor layouts. However, naively transplanting classical Structure-from-Motion (SfM) pipelines directly onto vehicle surround-view video streams immediately encounters severe physical and mathematical degeneracies: collinear motion corridors, textureless road surfaces, moving self-vehicle shadows, and unconstrained camera extrinsics drift.

In partnership with leading electric vehicle manufacturer NETA Auto, we developed and deployed an automotive-grade 4D vision auto-annotation data factory. As Project Tech Lead, I directed the overall algorithmic architecture and led the development of the core rigid multi-camera rig SfM reconstruction engine, ground-feature decoupling, spatiotemporal-guided dense multi-view stereo, and multi-session cross-epoch aggregation. The core geometric optimization principles and engineering systems developed during this industrial deployment were subsequently published at .

Why Driving Scenarios Break Classic SfM

In standard photogrammetry textbooks, classic SfM frameworks (such as COLMAP or VisualSFM) are built on the foundational assumption of an operator walking around an isolated, static monument with a single camera, capturing omnidirectional views with wide convergent baselines. Onboard vehicular data collected on highways and urban thoroughfares violates nearly every one of these assumptions:

Straight Corridor Motion & Collinear Triangulation Degeneracy: On highways and arterial avenues, vehicles travel along straight corridors for kilometers at a time. The optical centers of front and rear cameras align along a 1D straight line. In multi-view epipolar geometry, pure forward translation with collinear centers is ill-conditioned. When moving directly forward along the optical axis, the epipole coincides with the image principal center. Rays originating near the optical axis intersect at near-zero parallax angles. Sub-pixel feature matching noise is amplified along the longitudinal axis, causing reconstructed 3D points to stretch and drift along the direction of travel.
Extrinsics Drift in Unconstrained Multi-Camera Optimization: Production autonomous vehicles feature 6 to 8 surround cameras rigidly bolted to the vehicle body (front wide, front telephoto, front-left, front-right, rear-left, rear-right). Standard SfM treats each camera at each timestamp as an unconstrained 6-DoF pose variable. For an $N$-frame sequence, this introduces $6N$ independent pose variables. As pairwise feature matching accumulates noise, unconstrained optimization causes the physically rigid sensor extrinsics to artificially drift over time, introducing severe ghosting and discontinuous stitching artifacts in downstream BEV projections.
Textureless Asphalt & Moving Self-Vehicle Shadows: The lower half of onboard camera images is dominated by asphalt surfaces covering $>40\%$ of the image area. Asphalt is visually homogeneous, lacking sharp corner features under direct sunlight. Even more destructive is the vehicle’s dynamic shadow cast onto the road surface: the shadow boundary presents high-contrast, sharp edges that classical feature detectors (such as SIFT or Harris) aggressively extract as salient landmarks. However, the shadow translates synchronously with the vehicle, violating the static scene assumption. Triangulating shadow features forces the optimizer to fit non-existent points beneath the chassis, pulling vehicle trajectory estimates off course.
Multi-Session Cross-Epoch Aggregation Without Shared Image Matches: Autonomous driving maps require merging multiple independent drives through the same intersection collected across different days (e.g., overcast Monday morning vs. sunset Friday evening). Variable solar angles, changing shadows, and transient traffic mean disparate driving runs share zero visual feature correspondences, leaving independent reconstructions geometrically isolated.

Full Pipeline Architecture: From Raw Logs to 4D Vector Maps

To process tens of thousands of kilometers of production fleet video logs with high throughput, we established a decoupled, fail-safe processing pipeline:

Wider program

4D vision auto-annotation system

From synchronized vehicle data to a structured local map, with multi-camera SfM supplying geometry for annotation and mapping.

Upstream foundations

01Data preparationCameras · IMU · GNSS · chassis · calibration

02Semantic perceptionScene and lane masks

03Sensor-fusion localizationMetric vehicle trajectory

04 · My geometry workMulti-camera SfM reconstruction and system interfacesProgram coordination · poses and extrinsics · sparse/dense geometry · downstream delivery

Downstream applications

05Ground reconstructionDense road surface

06Auto-annotationLanes · signs · static elements

07Multi-run local mapAggregation and topology

Multi-Sensor Spatiotemporal Synchronization: Aligning six surround cameras, high-rate IMUs, GNSS receivers, and wheel odometry into a common timestamp reference and vehicle rear-axle coordinate frame;
Dynamic Semantic Masking: Deploying lightweight segmentation models to isolate moving vehicles, cyclists, pedestrians, and sky regions, shielding static 3D geometry from dynamic clutter;
Fused Inertial-GNSS Initialization: Fusing satellite positioning and dead-reckoning to initialize metric-scale trajectory guesses;
Rig-Centric Multi-Camera SfM (Core Engine): Treating surround cameras as a unified rigid rig to jointly solve vehicle poses and calibrate sensor extrinsics, generating a centimeter-level sparse road skeleton;
Spatiotemporal-Guided Dense MVS: Leveraging motion geometry priors to prune unshared image pairs, accelerating PatchMatch multi-view stereo to produce dense ground meshes;
Multi-Session Cross-Epoch Aggregation: Aligning independent driving passes coarse-to-fine into a globally seamless local high-definition map.

01Inputs & calibration Six-view imagery · multi-sensor pose · semantic masks · initial calibration

02Constrained association SuperPoint · temporal/overlap priors · two-view geometry

03Rig-aware sparse SfM Initialization · rig registration · triangulation · RIG BA · outlier filtering

04Densification & aggregation ACMP depth · COLMAP fusion · ground filtering · multi-run alignment

05Engineering outputs Refined poses/extrinsics · sparse/dense ground cloud · downstream priors

Core Algorithmic Design & Engineering Implementation

1. Rig Bundle Adjustment (Rig BA) with Rigid-Rig Parameterization

To eliminate extrinsics drift and reduce optimization dimensionality, we reformulated the multi-view optimization variables: rather than treating every frame as an independent pose, all surround cameras captured at timestamp $t$ are grouped into a rigid camera set. A camera’s absolute world pose is factored into a time-varying vehicle chassis pose and a static chassis-to-camera mounting extrinsic.

Under this parameterization, the number of pose variables collapses dramatically. We formulate the Rig Bundle Adjustment (Rig BA) framework to jointly optimize vehicle chassis trajectories, mounting extrinsics, and 3D static landmark coordinates.

This rigid formulation delivers two decisive advantages:

Complementary Viewpoint Geometry: When the vehicle drives into direct sun glare or passes alongside large trucks, forward cameras may suffer severe overexposure or visual occlusion. However, lateral and rear cameras continue tracking static architecture, trees, and street furniture. Through fixed internal rig extrinsics, unobstructed cameras constrain the chassis pose, eliminating longitudinal depth stretching;
Online Extrinsic Self-Calibration: By treating mounting extrinsics as optimization parameters, the solver refines mounting extrinsics over multi-kilometer runs to high precision, absorbing mechanical chassis flexing.

Exploiting multi-camera overlap priors removes numerous visually similar but physically impossible cross-camera false matches.

2. Spatial Overlap Constraints & Epipolar Pruning

Exhaustive pairwise matching across $N$ timestamps and $M$ cameras incurs an intractable $O(M \cdot N^2)$ computational explosion.

We introduced structured camera rig overlap priors:

A front-wide camera and front-left camera share only $\sim 30^\circ$ of horizontal FOV overlap. The right half of the front camera image cannot geometrically intersect the front-left camera’s viewing frustum;
Restricting feature searches strictly to overlapping spatial frustums ($u < W/2$ on the front camera) eliminates false associations between visually repetitive facades;
Enforcing Essential matrix $E$ epipolar distance checks and Homography $H$ planar constraints removes $>95\%$ of outlier correspondences while reducing matching runtime by $80\%$.

3. Decoupled Ground-Plane Filtering Prior to Triangulation

To prevent dynamic vehicle shadows and weak-texture asphalt from corrupting 3D point clouds:

Within the coarse sparse point cloud, we partition the road corridor into 5-meter sliding longitudinal cells;
Locally Optimal RANSAC (LO-RANSAC) fits local tangent ground planes $n_{\text{road}}^\top X + d = 0$;
Feature points residing within a 10 cm slice of the fitted plane are tagged and excluded from sparse bundle adjustment. The bundle adjustment solver converges purely on high-elevation rigid structures (building facades, light poles, overhead sign gantries), eliminating shadow-induced trajectory bias.

Neighborhood filtering strips isolated outliers while grid-based RANSAC suppresses thickness artifacts along the road surface.

4. Spatiotemporal-Guided Dense PatchMatch Stereo (MVS)

Generating continuous 3D ground meshes and voxel grids for occupancy networks requires high-throughput dense multi-view stereo:

Using the estimated vehicle trajectory, image pairs are screened based on optical parallax angles $\theta_{\text{parallax}} \in [5^\circ, 30^\circ]$ within a sliding temporal window of $\pm 15$ frames, avoiding global point-cloud searches;
GPU-accelerated multi-view PatchMatch (ACMP) iteratively optimizes depth maps and surface normal fields, employing slanted plane support priors to interpolate dense, continuous depth across textureless asphalt;
A multi-view geometric consistency filter fuses depth estimates across all six surround cameras into unified metric point clouds.

Standard COLMAP Dense

ACMP + Optimized Fusion

The hybrid strategy drastically accelerates fusion while improving reconstruction over weak-texture asphalt.

5. Coarse-to-Fine Multi-Session Aggregation Network

To reconcile asynchronous passes through identical intersections collected on different dates:

Coarse Georeferencing: Differential RTK-GNSS coordinates anchor each pass into a common metric coordinate system;
Fine Geometric Registration: The system isolates invariant vertical landmarks (traffic light poles, bridge pillars, structural facades). PnP and RANSAC solve the inter-session rigid transformation $T_{\text{run1} \to \text{run2}}$;
Global Pose Graph Relaxation: Fixing the primary reference pass, all subsequent passes are formulated as nodes connected by relative geometric constraints in a global pose graph. Non-linear least-squares optimization absorbs inter-session drift, stitching disparate runs into a seamless high-definition map.

Direct Overlay Before Aggregation

After Multi-Session Aggregation

Cross-session visual association and transform optimization reconcile distinct driving passes into one consistent spatial map.

Production Quality Assurance & Field Validation

The full-stack pipeline was validated in production data centers against ground-truth 128-beam mobile LiDAR SLAM and RTK reference trajectories:

Dense Surface Accuracy: Reprojecting dense visual point clouds onto forward-facing camera images and overlaying synchronized LiDAR sweeps demonstrated absolute road elevation errors under $\pm 3\text{ cm}$ within a 50-meter range, with sharp curb profiles;
Trajectory Consistency: Over 5-kilometer continuous city and highway loops, pure visual Rig BA suppressed cumulative yaw drift to 0.05°/100m, keeping closed-loop position errors below 0.1% of total travel distance;
Extrinsic Alignment Continuity: Applying optimized extrinsics to 360° BEV stitching eliminated lane marking shear across camera seams, restoring smooth physical continuity.

BEV projection comparison: lane markings across camera boundaries exhibit smooth physical continuity following optimization.

Pipeline Video Recordings

The four video sequences below present the end-to-end journey from raw onboard video and high-throughput dense depth generation to static 3D point cloud extraction and pixel-level reprojection checks:

Surround Camera StreamSynchronized surround-view video captured from production autonomous test fleets under dynamic driving conditions.

Static 3D Point Cloud ReconstructionHigh-precision static road geometry and architectural structures recovered by multi-camera rig SfM.

High-Throughput Dense Depth EstimationDense depth estimation guided by temporal and spatial overlap priors, achieving dramatic pipeline speedups.

Geometry & Appearance ReprojectionPixel-level reprojection verification demonstrating spatial consistency between recovered poses, dense geometry, and raw imagery.

Onboard Surround Video Stream: Synchronized six-camera footage recorded by production vehicles navigating dense city traffic;
Static Scene 3D Reconstruction: Dense, crisp geometric point-cloud models of overpasses, street furniture, and road structures;
High-Accuracy Dense Depth Estimation: Real-time dense disparity and depth maps generated across all views via spatiotemporal PatchMatch;
Appearance & Geometric Reprojection: Pixel-level reprojection checks confirming geometric alignment against original camera frames.

Engineering Summary & Technical Takeaways

Scaling Structure-from-Motion and multi-view stereo to automotive-grade automated annotation demonstrates key engineering principles:

Mechanical Rigidity as an Algorithmic Anchor: Multi-camera physical rigidity must be embedded directly into optimization formulations. Grouping surround cameras into a rigid rig halts extrinsic drift and overcomes collinear triangulation degeneracies through complementary multi-directional views.
Semantic Decoupling Guards Against Environmental Noise: In the presence of textureless asphalt and moving shadows, accepting all features indiscriminately destabilizes optimization. Decoupling ground features from vertical structures ensures robust convergence.
Bridging Isolated Trajectories into Spatial Networks: Autonomous driving requires scalable, repeatable mapping. Linking independent driving sessions through topological pose graphs establishes the foundation for large-scale digital twins and end-to-end driving models.

MRASfM: Multi-Camera Reconstruction and Aggregation through Structure-from-Motion in Driving Scenes

Sat, 31 Jan 2026 00:00:00 +0000

Driving is the wrong shape for photogrammetry

Image-based reconstruction for driving is usually handled by visual SLAM, and for HD mapping and novel-view synthesis the SLAM route has a structural limitation. Its estimates are incremental and locally optimized, which blocks global refinement, and the result is drift that accumulates over a long trajectory. Structure-from-motion does the opposite: it processes everything in a batch with a global bundle adjustment at the end. The offline nature that makes SfM impractical for a live robot is exactly what makes it attractive here, because it can spend computation a real-time system cannot.

So the obvious move is to use SfM for driving. Three facts about driving data make that harder than it should be.

The first is the capture geometry. Classic SfM assumes you walk around the subject and photograph it from many directions. A car drives down a corridor. Viewpoints are almost collinear, which is the worst case for the translation estimation that SfM depends on.

The second is that data arrives fragmented. Driving scenes are captured over short durations with limited overlap, and drives covering the same area may be recorded weeks apart. There are often no shared images at all between two datasets that need to end up in the same map.

The third is calibration, and this is the one that changed how we designed everything. Vehicle calibration is difficult, initial values contain errors, and those errors get worse on uneven terrain. So the extrinsics you have at the start of a drive are not the extrinsics you need, and they are also the strongest prior available. That tension runs through the whole design.

What prior methods assume

Multi-camera SLAM systems are accurate and require precise internal relative poses and intrinsics as input. They need calibration done before deployment, and they struggle to accommodate different camera configurations, which restricts flexibility in practice.

Incremental SfM is the closest baseline, and our criticism of it is specific. It ignores camera-set priors during incremental reconstruction and applies them only afterward, in rigid bundle adjustment once the reconstruction is already complete. By that point, errors that accumulated during incremental estimation cannot be corrected. Rigid BA cannot undo drift that has already been baked into the structure. Per-image bundle adjustment has a related problem: it can leave the internal relative poses inconsistent from frame to frame.

Global multi-camera SfM does fix the relative camera poses from the start, and it is sensitive to outlier matches and tends to degenerate when the estimated relative translations are collinear. In a driving scene the trajectory is close to collinear by default, so this is a structural hazard rather than an edge case.

Odometry-guided SfM takes the opposite bet and uses odometry poses and camera extrinsics directly for registration. That demands highly precise extrinsic calibration and accurate odometry, which are exactly the two things field deployment cannot reliably supply. We borrow this line of work’s idea of using homography to identify overlapping image pairs. We do not borrow the assumption that the priors can be trusted.

There is also a matching problem specific to the setting. Traditional descriptors produce limited feature points in low-texture images, and road scenes are full of low-texture surfaces.

Finally, existing aggregation modules for merging reconstructions typically assume shared imagery between the segments being merged. In practice that imagery usually does not exist.

What makes it hard

Priors have to be used without being trusted. Extrinsics and odometry are the strongest signal available and the least reliable thing on the vehicle. Trust them entirely and you get one family of failures; ignore them and you get another. The design space in between is narrow.

Collinearity is a real degeneracy, not a numerical nuisance. Fixed-rig global methods hinge on relative translation estimates, and straight-line driving makes those ill-conditioned.

The failure modes sit exactly where the useful signal is. The road surface occupies a large part of every image and has almost no texture, so it produces both few matches and bad ones. Worse, the edges of vehicle shadows are high-contrast and get detected as feature points. A shadow edge is not a scene point. When the shadow moves, that feature moves, which injects motion into the road surface precisely where the structure matters most.

Occlusion behaves differently on a rig than on a single camera. One blocked camera on a monocular system is a total failure; on a surround rig the other cameras still see the world. But that redundancy is only worth anything if registration actually shares the pose, and per-camera estimation throws the advantage away.

And robustness has a price. More cameras mean more poses to optimize in bundle adjustment, which reduces efficiency. The coverage that buys robustness is paid for in optimization size, and cancelling that cost is what a large part of our design exists to do.

The camera set is the unit

Our idea is to change the atomic element of reconstruction. Instead of treating each image as an independent camera to be solved, we take the rigid unit, every image captured at the same instant, as the thing that gets a pose. That pose is by definition the vehicle’s pose at that moment.

The benefit is that the pose of a rig factorizes into a time-varying vehicle pose and a static internal relative pose. One well-observed camera can carry the poses of the cameras that see nothing useful. The rig is estimated once rather than re-learned at every frame.

The important part is how that factorization is applied. We use it as a reparameterization inside registration and bundle adjustment, not as a constraint imposed afterward. That single decision does three things at once. It reduces the pose variables from one per camera per timestamp down to one per timestamp plus one per camera. It keeps the optimized rig consistent across frames, because there is only one rig to be consistent. And it turns calibration from an input that must be correct into a quantity that can be corrected.

Building a reconstruction

Correspondence search comes first. SuperPoint detects and SuperGlue matches, pairs whose matched points fall into different semantic categories are discarded, and geometric verification removes the rest. Pairs are not all tried: with the rough internal relative poses and vehicle poses in hand, we compute the approximate field of view of each image and only match pairs that could plausibly overlap. Filtering this way improves both efficiency and accuracy, since matching everything against everything is expensive and injects noise from pairs that were never going to share content.

Registration then runs in three steps. PnP is estimated on the images with the richest correspondences. Those per-image estimates are converted into per-unit estimates and fused by local robust rotation averaging and translation averaging into a single unit pose. Every remaining image in that unit gets its pose by composing the unit pose with the calibration.

That middle step is where the method sits between the two failure modes described earlier. It averages, like the global family, but it averages over unit hypotheses derived from reliable images rather than over raw relative translations, which is what makes it survive collinearity.

The honest accounting of this module is narrow. It has minimal effect on average pose accuracy and efficiency, and it is particularly useful for handling occluded viewpoints. In scenes with rich correspondences the impact is less noticeable. We state that tradeoff rather than presenting the module as uniformly beneficial, because the value of redundancy is concentrated in the tail.

View full-size image ↗

Occlusion on one camera no longer fails the unit.

Getting the road out of the way

After a unit is registered, we fit a plane model to the road points with LO-RANSAC and remove them. This targets the shadow-edge problem directly. A moving shadow edge produces a feature that is not attached to the scene, and it lands on the largest and least textured surface in view, so the triangulation noise concentrates there by default. Removing those points improves reconstruction accuracy and accelerates convergence.

The assumption being made is local flatness of the road, and our own real-world scenes include slope changes, so this is an approximation rather than a guarantee.

View full-size image ↗

Removing shadow-driven and texture-poor points where the noise concentrates.

Bundle adjustment on the rig

Camera set bundle adjustment optimizes vehicle poses and internal relative poses under substitution constraints that express each camera pose in terms of the vehicle pose and the rig geometry, with a Cauchy loss for the local stage. The variables include intrinsics, vehicle poses, internal relative poses and scene points.

There are two modes and the split matters. Local CSBA fixes intrinsics and internal relative poses and touches only newly registered units and their neighbours. Global CSBA optimizes everything across all registered units, and we trigger it when many units remain unoptimized.

Global CSBA re-optimizing intrinsics and relative poses is the online recalibration capability claimed in the paper. It is also the practical payoff, because it converts the field’s least reliable input into an output. Local CSBA is cheaper, and it freezes exactly those parameters, which is the tradeoff: the cheap mode cannot fix calibration, and the mode that can fix it costs more.

The variable reduction from one pose per camera per timestamp to one per timestamp plus one per camera is worth dwelling on. Six cameras over a long trajectory means the per-image formulation optimizes six times as many poses as it needs to, and every extra variable is another degree of freedom that noise can hide in. Fewer variables makes convergence faster and the result more consistent. Robustness and efficiency turn out to be the same lever here, which is not usually the case.

View full-size image ↗

Vehicle poses and internal relative poses are refined jointly, which is what allows calibration to improve in the field.

Aggregation has to work without shared imagery, so the whole strategy is coarse-to-fine. Scenes are first placed in a global frame using satellite positioning, represented by trajectory midpoints: one scene sits at the geometric centre, its nearest neighbours become candidates, and homography-guided spatial pairs pick whichever candidate has the greatest visual overlap.

The coarse placement is wrong by an unknown transform, because positioning error is larger than the alignment tolerance. So the transform itself becomes the variable. One image from the incoming scene is relocalized to get a refined pose, which yields the transform between the coarse placement and the true one, and that transform registers the entire rigid unit at once. Triangulation follows, then a transformation-based bundle adjustment optimizes the transform and the scene points while keeping the reference scene’s camera poses fixed.

Registration, triangulation and adjustment iterate until every unit of the incoming scene has been absorbed. The merged result then becomes the new reference for the next scene, so aggregation proceeds as a chain rather than against a single global anchor. The reason is simple: chaining keeps error from originating at one fallible alignment.

The whole stage is designed so aggregation never has to be exact at the coarse step, only close enough to relocalize.

View full-size image ↗

Satellite positioning places the scene; vision refines the placement.

What we take from this

A physical invariant is best used as a reparameterization rather than a constraint. Enforcing rigidity after the fact cannot prevent the error that accumulated before. Making rigidity part of the variable structure prevents it from accumulating in the first place.

Priors should prune, not conclude. Rough poses decide which pairs to match and where to start, and the geometry still comes from matched evidence. That is the line between our approach and the one we criticize.

Design so priors can be corrected rather than merely trusted. Making relative poses optimizable is what turns the least reliable input into an output.

And judge a module where its failure mode lives. Camera set registration barely moves average pose error, and it is the difference between working and failing when a camera is occluded. Aggregate metrics can hide a module’s entire value.

What it still assumes

The method takes rough trajectories, rough calibration and semantic information as inputs. The semantic source is assumed rather than specified, and we report no sensitivity to the quality of those labels.

Multi-scene aggregation is not ablated. The choice of roughly three candidate scenes and the geometric-centre reference heuristic are stated without justification or failure analysis.

The rigid unit is defined purely by simultaneity, and we do not test what happens when that assumption weakens. Camera time-synchronization error and rig flex would both violate it, and neither is discussed. Online calibration is claimed but not characterized: we do not say under what motion diversity or camera count the global stage’s refinement of intrinsics and relative poses is well posed.

The plane model presumes a locally flat road while our own deployment includes slope changes. Generality across different multi-camera configurations is asserted but exercised only on six- and seven-camera rigs. Dynamic objects beyond shadows appear in the motivation but have no dedicated treatment.

Where it leads

The contribution we would state is a change of unit, and everything else follows from it. What the change buys is worth spelling out: camera sets as atomic units for registration and refinement give robust pose estimation in complex environments and better efficiency than conventional per-camera methods. Combined with the road filtering and the aggregation stage, the pipeline produces a single map from drives that were never meant to be merged.

We validated it in two directions on purpose. Real instrumented vehicles demonstrate that it holds up outside a dataset, and public benchmarks establish the accuracy claim against a conventional baseline. The practical statement of the stake is that intelligent vehicles can perceive their environment and estimate their own pose more reliably, which is what the downstream tasks actually consume.

4D Auto-Annotation | Chang Nie

Production-Grade 4D Auto-Annotation: Multi-Camera Rig SfM & High-Precision Geometry

Project context

My contributions

Why Driving Scenarios Break Classic SfM

Full Pipeline Architecture: From Raw Logs to 4D Vector Maps

Core Algorithmic Design & Engineering Implementation

1. Rig Bundle Adjustment (Rig BA) with Rigid-Rig Parameterization

2. Spatial Overlap Constraints & Epipolar Pruning

3. Decoupled Ground-Plane Filtering Prior to Triangulation

4. Spatiotemporal-Guided Dense PatchMatch Stereo (MVS)

5. Coarse-to-Fine Multi-Session Aggregation Network

Production Quality Assurance & Field Validation

Pipeline Video Recordings

Engineering Summary & Technical Takeaways

MRASfM: Multi-Camera Reconstruction and Aggregation through Structure-from-Motion in Driving Scenes

Driving is the wrong shape for photogrammetry

What prior methods assume

What makes it hard

The camera set is the unit

Building a reconstruction

Getting the road out of the way

Bundle adjustment on the rig

Merging drives that share no images

What we take from this

What it still assumes

Where it leads