Conference Papers | Chang Nie

MRASfM: Multi-Camera Reconstruction and Aggregation through Structure-from-Motion in Driving Scenes

Sat, 31 Jan 2026 00:00:00 +0000

Driving is the wrong shape for photogrammetry

Image-based reconstruction for driving is usually handled by visual SLAM, and for HD mapping and novel-view synthesis the SLAM route has a structural limitation. Its estimates are incremental and locally optimized, which blocks global refinement, and the result is drift that accumulates over a long trajectory. Structure-from-motion does the opposite: it processes everything in a batch with a global bundle adjustment at the end. The offline nature that makes SfM impractical for a live robot is exactly what makes it attractive here, because it can spend computation a real-time system cannot.

So the obvious move is to use SfM for driving. Three facts about driving data make that harder than it should be.

The first is the capture geometry. Classic SfM assumes you walk around the subject and photograph it from many directions. A car drives down a corridor. Viewpoints are almost collinear, which is the worst case for the translation estimation that SfM depends on.

The second is that data arrives fragmented. Driving scenes are captured over short durations with limited overlap, and drives covering the same area may be recorded weeks apart. There are often no shared images at all between two datasets that need to end up in the same map.

The third is calibration, and this is the one that changed how we designed everything. Vehicle calibration is difficult, initial values contain errors, and those errors get worse on uneven terrain. So the extrinsics you have at the start of a drive are not the extrinsics you need, and they are also the strongest prior available. That tension runs through the whole design.

What prior methods assume

Multi-camera SLAM systems are accurate and require precise internal relative poses and intrinsics as input. They need calibration done before deployment, and they struggle to accommodate different camera configurations, which restricts flexibility in practice.

Incremental SfM is the closest baseline, and our criticism of it is specific. It ignores camera-set priors during incremental reconstruction and applies them only afterward, in rigid bundle adjustment once the reconstruction is already complete. By that point, errors that accumulated during incremental estimation cannot be corrected. Rigid BA cannot undo drift that has already been baked into the structure. Per-image bundle adjustment has a related problem: it can leave the internal relative poses inconsistent from frame to frame.

Global multi-camera SfM does fix the relative camera poses from the start, and it is sensitive to outlier matches and tends to degenerate when the estimated relative translations are collinear. In a driving scene the trajectory is close to collinear by default, so this is a structural hazard rather than an edge case.

Odometry-guided SfM takes the opposite bet and uses odometry poses and camera extrinsics directly for registration. That demands highly precise extrinsic calibration and accurate odometry, which are exactly the two things field deployment cannot reliably supply. We borrow this line of work’s idea of using homography to identify overlapping image pairs. We do not borrow the assumption that the priors can be trusted.

There is also a matching problem specific to the setting. Traditional descriptors produce limited feature points in low-texture images, and road scenes are full of low-texture surfaces.

Finally, existing aggregation modules for merging reconstructions typically assume shared imagery between the segments being merged. In practice that imagery usually does not exist.

What makes it hard

Priors have to be used without being trusted. Extrinsics and odometry are the strongest signal available and the least reliable thing on the vehicle. Trust them entirely and you get one family of failures; ignore them and you get another. The design space in between is narrow.

Collinearity is a real degeneracy, not a numerical nuisance. Fixed-rig global methods hinge on relative translation estimates, and straight-line driving makes those ill-conditioned.

The failure modes sit exactly where the useful signal is. The road surface occupies a large part of every image and has almost no texture, so it produces both few matches and bad ones. Worse, the edges of vehicle shadows are high-contrast and get detected as feature points. A shadow edge is not a scene point. When the shadow moves, that feature moves, which injects motion into the road surface precisely where the structure matters most.

Occlusion behaves differently on a rig than on a single camera. One blocked camera on a monocular system is a total failure; on a surround rig the other cameras still see the world. But that redundancy is only worth anything if registration actually shares the pose, and per-camera estimation throws the advantage away.

And robustness has a price. More cameras mean more poses to optimize in bundle adjustment, which reduces efficiency. The coverage that buys robustness is paid for in optimization size, and cancelling that cost is what a large part of our design exists to do.

The camera set is the unit

Our idea is to change the atomic element of reconstruction. Instead of treating each image as an independent camera to be solved, we take the rigid unit, every image captured at the same instant, as the thing that gets a pose. That pose is by definition the vehicle’s pose at that moment.

The benefit is that the pose of a rig factorizes into a time-varying vehicle pose and a static internal relative pose. One well-observed camera can carry the poses of the cameras that see nothing useful. The rig is estimated once rather than re-learned at every frame.

The important part is how that factorization is applied. We use it as a reparameterization inside registration and bundle adjustment, not as a constraint imposed afterward. That single decision does three things at once. It reduces the pose variables from one per camera per timestamp down to one per timestamp plus one per camera. It keeps the optimized rig consistent across frames, because there is only one rig to be consistent. And it turns calibration from an input that must be correct into a quantity that can be corrected.

Building a reconstruction

Correspondence search comes first. SuperPoint detects and SuperGlue matches, pairs whose matched points fall into different semantic categories are discarded, and geometric verification removes the rest. Pairs are not all tried: with the rough internal relative poses and vehicle poses in hand, we compute the approximate field of view of each image and only match pairs that could plausibly overlap. Filtering this way improves both efficiency and accuracy, since matching everything against everything is expensive and injects noise from pairs that were never going to share content.

Registration then runs in three steps. PnP is estimated on the images with the richest correspondences. Those per-image estimates are converted into per-unit estimates and fused by local robust rotation averaging and translation averaging into a single unit pose. Every remaining image in that unit gets its pose by composing the unit pose with the calibration.

That middle step is where the method sits between the two failure modes described earlier. It averages, like the global family, but it averages over unit hypotheses derived from reliable images rather than over raw relative translations, which is what makes it survive collinearity.

The honest accounting of this module is narrow. It has minimal effect on average pose accuracy and efficiency, and it is particularly useful for handling occluded viewpoints. In scenes with rich correspondences the impact is less noticeable. We state that tradeoff rather than presenting the module as uniformly beneficial, because the value of redundancy is concentrated in the tail.

View full-size image ↗

Occlusion on one camera no longer fails the unit.

Getting the road out of the way

After a unit is registered, we fit a plane model to the road points with LO-RANSAC and remove them. This targets the shadow-edge problem directly. A moving shadow edge produces a feature that is not attached to the scene, and it lands on the largest and least textured surface in view, so the triangulation noise concentrates there by default. Removing those points improves reconstruction accuracy and accelerates convergence.

The assumption being made is local flatness of the road, and our own real-world scenes include slope changes, so this is an approximation rather than a guarantee.

View full-size image ↗

Removing shadow-driven and texture-poor points where the noise concentrates.

Bundle adjustment on the rig

Camera set bundle adjustment optimizes vehicle poses and internal relative poses under substitution constraints that express each camera pose in terms of the vehicle pose and the rig geometry, with a Cauchy loss for the local stage. The variables include intrinsics, vehicle poses, internal relative poses and scene points.

There are two modes and the split matters. Local CSBA fixes intrinsics and internal relative poses and touches only newly registered units and their neighbours. Global CSBA optimizes everything across all registered units, and we trigger it when many units remain unoptimized.

Global CSBA re-optimizing intrinsics and relative poses is the online recalibration capability claimed in the paper. It is also the practical payoff, because it converts the field’s least reliable input into an output. Local CSBA is cheaper, and it freezes exactly those parameters, which is the tradeoff: the cheap mode cannot fix calibration, and the mode that can fix it costs more.

The variable reduction from one pose per camera per timestamp to one per timestamp plus one per camera is worth dwelling on. Six cameras over a long trajectory means the per-image formulation optimizes six times as many poses as it needs to, and every extra variable is another degree of freedom that noise can hide in. Fewer variables makes convergence faster and the result more consistent. Robustness and efficiency turn out to be the same lever here, which is not usually the case.

View full-size image ↗

Vehicle poses and internal relative poses are refined jointly, which is what allows calibration to improve in the field.

Aggregation has to work without shared imagery, so the whole strategy is coarse-to-fine. Scenes are first placed in a global frame using satellite positioning, represented by trajectory midpoints: one scene sits at the geometric centre, its nearest neighbours become candidates, and homography-guided spatial pairs pick whichever candidate has the greatest visual overlap.

The coarse placement is wrong by an unknown transform, because positioning error is larger than the alignment tolerance. So the transform itself becomes the variable. One image from the incoming scene is relocalized to get a refined pose, which yields the transform between the coarse placement and the true one, and that transform registers the entire rigid unit at once. Triangulation follows, then a transformation-based bundle adjustment optimizes the transform and the scene points while keeping the reference scene’s camera poses fixed.

Registration, triangulation and adjustment iterate until every unit of the incoming scene has been absorbed. The merged result then becomes the new reference for the next scene, so aggregation proceeds as a chain rather than against a single global anchor. The reason is simple: chaining keeps error from originating at one fallible alignment.

The whole stage is designed so aggregation never has to be exact at the coarse step, only close enough to relocalize.

View full-size image ↗

Satellite positioning places the scene; vision refines the placement.

What we take from this

A physical invariant is best used as a reparameterization rather than a constraint. Enforcing rigidity after the fact cannot prevent the error that accumulated before. Making rigidity part of the variable structure prevents it from accumulating in the first place.

Priors should prune, not conclude. Rough poses decide which pairs to match and where to start, and the geometry still comes from matched evidence. That is the line between our approach and the one we criticize.

Design so priors can be corrected rather than merely trusted. Making relative poses optimizable is what turns the least reliable input into an output.

And judge a module where its failure mode lives. Camera set registration barely moves average pose error, and it is the difference between working and failing when a camera is occluded. Aggregate metrics can hide a module’s entire value.

What it still assumes

The method takes rough trajectories, rough calibration and semantic information as inputs. The semantic source is assumed rather than specified, and we report no sensitivity to the quality of those labels.

Multi-scene aggregation is not ablated. The choice of roughly three candidate scenes and the geometric-centre reference heuristic are stated without justification or failure analysis.

The rigid unit is defined purely by simultaneity, and we do not test what happens when that assumption weakens. Camera time-synchronization error and rig flex would both violate it, and neither is discussed. Online calibration is claimed but not characterized: we do not say under what motion diversity or camera count the global stage’s refinement of intrinsics and relative poses is well posed.

The plane model presumes a locally flat road while our own deployment includes slope changes. Generality across different multi-camera configurations is asserted but exercised only on six- and seven-camera rigs. Dynamic objects beyond shadows appear in the motivation but have no dedicated treatment.

Where it leads

The contribution we would state is a change of unit, and everything else follows from it. What the change buys is worth spelling out: camera sets as atomic units for registration and refinement give robust pose estimation in complex environments and better efficiency than conventional per-camera methods. Combined with the road filtering and the aggregation stage, the pipeline produces a single map from drives that were never meant to be merged.

We validated it in two directions on purpose. Real instrumented vehicles demonstrate that it holds up outside a dataset, and public benchmarks establish the accuracy claim against a conventional baseline. The practical statement of the stake is that intelligent vehicles can perceive their environment and estimate their own pose more reliably, which is what the downstream tasks actually consume.

MovSAM: A Single-image Moving Object Segmentation Framework Based on Deep Thinking

Sun, 19 Oct 2025 00:00:00 +0000

A photograph contains no motion

Moving object segmentation is normally a measurement problem. You compare successive frames, or you compute optical flow, and the pixels that changed are the pixels that moved. The assumption is so standard that it stops being visible.

It breaks in specific, practical ways. A pedestrian shifting their weight at the curb before stepping off produces almost no pixel change, and the intent is invisible to a frame-difference method. A vehicle’s own motion contaminates the comparison, since apparent movement in the image is a mixture of the other object’s motion and the camera’s own, and separating the two is not something a two-frame difference does. Occlusion makes pixel changes unreliable for anything partly hidden. And sometimes there is only one frame: if a camera fails or a communication link drops, the multi-frame pipeline does not degrade, it stops.

There is also a simpler observation behind this work. A person looking at a photograph can usually point at the thing that is about to move. Not because they measured displacement, but because they read the scene: the posture, the contact, the loose fabric, the blur along an edge. The motion is inferred from semantics rather than observed from change.

We wanted a system that could do that. Which means the temporal signal has to be replaced by something else, and language is the obvious candidate.

What earlier methods assume

Optical-flow-based approaches use short-term motion cues to separate moving objects from static background. Generating accurate optical flow requires multiple frames, so it is not a substitute for frames. It consumes them.

Multi-frame and video-based methods relate appearance to motion patterns and are genuinely effective. They also require either flow computations or temporal information from more than one frame, and on a single image they have nothing to work with.

Segment Anything is the natural tool for the segmentation half, and it has two problems here. It was built for point and box prompts, and text prompts are not its native interface, so a bridge has to be constructed rather than configured. More fundamentally, SAM analyzes image texture. It has no scene comprehension, so it cannot tell you which texture is moving.

The line of work that connects a multimodal language model to SAM for instructed segmentation is closer, and the distinction that matters is between recognizing and reasoning. Those methods enhance SAM with text and focus on identifying object categories. They answer what something is, not whether it is moving or whether the mask captured all of it. Generic open-vocabulary segmentation has a related blind spot: it does not address problems specific to motion, such as motion illusions, where appearance suggests movement that is not happening.

And stacking an MLLM, SAM and a vision-language model together directly does not work either. That composition still produces inaccuracies from hallucinations or flawed reasoning, and it produces visible contradictions between what was predicted and what was expected. Building a loop that can notice and repair those contradictions is the part that had to be designed.

What makes it hard

The deepest difficulty is that a still image contains no displacement information at all. Motion has to be inferred from static correlates, and those correlates are heuristics. A flowing effect in hair or clothing, motion blur along contours, an object suspended in mid-air without support, a pose resting on minimal support such as a single toe touching the ground: all of these suggest motion, and none of them prove it. The paper is explicit that the model identifies the moving object based on these features but not limited to them.

Motion illusions follow directly. Appearance that suggests movement can be wrong, and genuine movement can look static.

Ego-motion is unobservable in a single frame by construction. Nothing in one photograph distinguishes “they moved” from “I moved,” and no amount of reasoning recovers a distinction the data does not contain.

There is also a representation mismatch at the centre of the architecture. SAM’s image embedding is spatially rich and semantically empty. A vision-language model’s features are semantic and spatially coarse. Something has to make SAM’s detail legible to the VLM without destroying it.

Then there is the feedback problem, which is subtler than it first appears. For a loop to work, a mask has to be turned back into something the reasoner can critique. Not just whether the right object was chosen, but whether the mask is complete and whether it contains anything that does not belong. A prediction has to become an arguable claim.

Finally, supervision barely exists. There is no dataset of single images labelled with the moving object, because the label is inherently a judgement. We had to construct the training data by filtering existing video datasets down to images where the moving object is clearly distinguishable.

The mask as something you can argue with

Our idea is to replace the missing temporal measurement with an explicit linguistic hypothesis, and then make that hypothesis revisable.

The loop has three parts. A reasoner looks at the scene and writes a sentence describing the object it believes is moving. That sentence is not a bare category label. It carries the category, the position and the motion, in the form of something like “A red ball is flying in the air, in the upper right corner of the frame.” A segmentation stack turns that sentence into a mask. Then the mask goes back to the reasoner as visual evidence, and the reasoner is allowed to change its mind.

The last step is what makes this work. Reasoning about a still image is unreliable, and a single pass produces confident mistakes. Showing the result back closes the gap between what the model meant and what it produced.

The prompt is generated under a chain-of-thought guide, which converts an under-specified open judgement into something closer to a checklist: observe the objects and their relations, attend to the plausible movers, apply the motion features above, then emit a prompt or conclude that nothing is moving. The guide is hand-authored, and we did not test alternatives against it. That is a limitation we accept rather than hide.

The segmentation network takes the prompt and the image together and produces the mask. SAM2 is the core, and we keep its image encoder and mask decoder because they are well suited to the job. We add BEiT-3 features because SAM’s weakness is exactly semantic understanding, and the vision-language model supplies what it lacks. The prompt encoder is modified so that the VLM’s multimodal features can serve as a prompt directly, rather than being squeezed into a spatial point or bounding box.

The feature aggregation network is our answer to the representation mismatch, and it runs in the direction people usually do not expect. Instead of trying to make SAM semantic, we give the VLM SAM’s spatial detail.

Let $F_{\text{sam}} \in \mathbb{R}^{H \times W \times C_s}$ denote SAM2’s high-resolution spatial feature map. The aggregation network $\Phi_{\text{agg}}$ applies five convolutional layers with residual bottlenecks followed by adaptive spatial pooling and a linear projection, extracting a compressed global context descriptor:

$$ \mathbf{v}_{\text{ctx}} = \Phi_{\text{agg}}(F_{\text{sam}}) \in \mathbb{R}^{C_v}. $$

This global descriptor is broadcast and concatenated across every spatial location of the visual embedding:

$$ \tilde{F}_{\text{sam}} = \left[ F_{\text{sam}}, \; \operatorname{tile}(\mathbf{v}_{\text{ctx}}, H \times W) \right] \in \mathbb{R}^{H \times W \times (C_s + C_v)}. $$

The augmented representation is fed alongside prompt tokens $T_{\text{beit}} \in \mathbb{R}^{L \times D}$ into bidirectional cross-attention layers. Removing this aggregation pathway degrades validation performance noticeably, confirming that cross-modal feature alignment is functionally active rather than merely cosmetic.

View full-size image ↗

MovSAM uses scene semantics and appearance to infer likely moving objects from one image.

The reasoning loop: Propose, segment, critique

The critique stage has its own guided procedure. Given the original image with the candidate mask overlaid, the multimodal reasoner executes a four-point verification protocol:

Object Identity & Attribution: Does the segmented boundary correspond cleanly to the entity identified in the prompt?
Physical Motion Plausibility: Does the entity exhibit unambiguous static motion heuristics (unbalanced dynamic pose, airborne suspension, motion blur, directionally trailing fabric/hair)?
Segmentation Completeness: Has any contiguous functional part of the moving entity (extended limbs, carried equipment, attached garments) been erroneously severed from the mask?
Mask Purity: Has any non-moving background structure (road surface, cast contact shadows, adjacent stationary furniture) been incorrectly absorbed into the mask?

Notice that completeness and purity are separate questions from correctness. A mask can cover the right object and still be a bad answer.

When the reasoner rejects the result, it rewrites the prompt and the mask is regenerated. When it accepts, it must provide an explicit physical rationale. The published example illustrates how much work the loop accomplishes. The first prompt asks for the runner on the right side of the image, and the reasoner rejects it: that runner is in mid-stride but grounded and secondary. The second prompt targets the primary moving subject, identifying an upside-down breakdancer in the center. That proposal is accepted, justified by the physical observation that the dancer is inverted in mid-air with minimal support contact and impending rotational displacement.

That exchange embodies the architecture in miniature: the initial output was plausible but suboptimal; the second revision emerged directly from visually critiquing the concrete artifacts of the first pass.

We cap the reasoning loop at five iterations and terminate upon acceptance. If the budget expires without convergence, the candidate with highest verification confidence is selected.

View full-size image ↗

Because each frame is processed independently, occlusion does not propagate errors across time.

What is and is not trained

The SAM image encoder remains frozen, preserving its pre-trained visual primitives without incurring massive training overhead. The vision-language model, feature aggregation module, prompt encoder adapter, and mask decoder are jointly optimized for moving object segmentation. The high-level multimodal language model runs off-the-shelf in zero-shot mode without parameter fine-tuning, preserving its general reasoning and linguistic breadth.

The training objective combines balanced soft Dice overlap with focal binary cross-entropy:

$$ \mathcal{L}_{\text{seg}} = \lambda_{\text{dice}} \mathcal{L}_{\text{dice}}(M_{\text{pred}}, M_{\text{gt}}) + \lambda_{\text{focal}} \mathcal{L}_{\text{focal}}(M_{\text{pred}}, M_{\text{gt}}). $$

The soft Dice loss penalizes boundary and spatial volume discrepancies:

$$ \mathcal{L}_{\text{dice}} = 1 - \frac{2 \sum_{i} m_i \hat{m}_i + \epsilon}{\sum_i m_i + \sum_i \hat{m}_i + \epsilon}, $$

while the focal loss counteracts extreme foreground-background class imbalance where dynamic targets occupy an extremely small fraction of the image frame:

$$ \mathcal{L}_{\text{focal}} = - \sum_{i} \alpha_t (1 - p_{t, i})^\gamma \log(p_{t, i}). $$

In practice, this composite formulation provides superior boundary sharpness and training stability compared to either loss alone.

What we take from this

Text is a usable control channel for frozen foundation models. A prompt can carry category, position and a motion verb at once, which is far richer than a class label and requires no retraining at all.

The awkward representation mismatch has a clean resolution that runs against intuition: feed spatial detail into the semantic model rather than trying to make the spatial model semantic.

Making the output criticisable is the generalizable trick. It applies to any task where the supervision signal is a judgement rather than a measurement, and where a plausible wrong answer is the main failure mode.

And it is worth noticing that the composition itself can be the contribution. We say this plainly in the paper rather than pretending otherwise. The novelty is in the loop and in the way the parts are connected, not in any single component.

One more practical note. The training set was filtered down to images where the moving object is clearly distinguishable, which means the supervision concentrates exactly where static inference is easiest. We state the filter and we do not resolve the tension it creates.

View full-size image ↗

Qualitative comparison of single-image moving object segmentation.

What this opens

Single-image moving object segmentation is worth treating as its own problem rather than as a degraded video task. Recovering temporal information from one image is a capability we think has independent value, and it changes what a robot can do when its perception stream is degraded rather than healthy.

The reasoning mechanism is also portable. Any task that needs logical inference over a scene from a large model, including cross-modal settings such as reasoning about the relationships between objects in an image and a point cloud, could use the same propose, segment, critique structure.

The application we care most about is the one we started from. A robot that receives an isolated image, because a link dropped or a sensor failed, can still form an object-level interpretation of what is moving and what to watch. The paper’s closing claim is that this makes systems more robust in exactly the situations where conventional methods stop working.

RLSAC: Reinforcement Learning Enhanced Sample Consensus for End-to-End Robust Estimation

Sun, 01 Oct 2023 00:00:00 +0000

The sampler is the part nobody trains

Robust estimation carries most of geometric vision on its back. Camera pose, motion segmentation, wide-baseline matching, plane fitting, line fitting: strip away the application and they all come down to the same question. Given observations where a large fraction are simply wrong, recover the model that the correct ones agree on. RANSAC has been the default answer to that question for four decades. It draws a minimum set, solves for a model from that set, counts how many observations agree with the result, and tries again. Two points define a line. Eight correspondences define a fundamental matrix.

The drawing step is deliberately blind. Every observation is treated as equally worth trying, and nothing about the data ever enters the decision. That bet pays off when outliers are rare. It stops paying off quickly. Raise the outlier fraction and the probability of drawing a set entirely from the inliers falls away, and the number of iterations needed to find a good model climbs with it. The evidence that would have made the search easy is sitting in the observations themselves, and the sampler never looks at it.

The second problem is quieter, and it is the reason we wrote this paper. A uniform random draw is not differentiable. You cannot attach it to a network, and you cannot train it against an objective that lives downstream of it. So the robust estimator ends up parked at the edge of a learned system: an off-the-shelf component wrapped in interfaces, never optimized together with the parts around it.

Earlier work has attacked this from several directions, and each one leaves something on the table. LO-RANSAC keeps sampling inside the inliers of the best hypothesis so far, which improves a model it has already found rather than the search that found it. NAPSAC restricts sampling to a local neighborhood and buys better-conditioned sets at the cost of global coverage; Progressive NAPSAC’s expanding neighborhood patches that assumption rather than replacing it. GC-RANSAC and MAGSAC++ change how hypotheses are scored, not how they are proposed. PROSAC and USAC reorder or schedule the sampling, but the ordering rule is designed by hand and frozen before the first hypothesis has been scored. Methods such as NG-RANSAC learn a useful preference over points, and the filtering approaches around it discard bad candidate sets early, yet neither treats the sequence of sampling decisions as something worth planning. DSAC takes the most direct route and makes selection probabilistic so gradients flow through the pipeline. That works, but it requires the sampler, the solver and the scoring step to all be differentiable, and the classical pieces we trust most are exactly the ones that are not.

The sample is an action

Our starting observation is that the two problems have the same shape. In reinforcement learning, an agent reads a state, picks an action, and the environment reports what happened. Sampling a minimum set has that structure already. The observations are the state. The set chosen from them is the action. And the environment exists and is running: it is the solver and the consensus function.

The reward is the interesting part, because the pipeline produces one for free. Once the solver turns a proposed set into a model, the consensus function measures what fraction of all observations agree with it. That number scores the model, and since the model came out of that particular set, it also scores the choice that produced it. No labels for which points are inliers are needed anywhere in the loop. Any task that already ships a solver and a residual definition comes with its own reward.

That last point matters more than it sounds. Training from the joint outcome means the policy is never taught which individual observations are good. It is taught whether the model that came out of a combination was good. Pointwise inlier classification is a different and much stronger assumption than set-level scoring, and it is the assumption we did not want to make.

Question	RLSAC’s answer
What is learned?	The sampling policy; the solver and the scoring function stay as they are
What supervision is needed?	Nothing beyond the consensus score the pipeline already computes
What enters the state?	Observation features, the chosen set, residuals against the last hypothesis, and usage counts
Where is it used?	Synthetic line fitting and real two-view fundamental-matrix estimation

What the policy gets to see

A state that contains only the observations is not enough. The same data at step one and step ten would look identical, and the policy would have no way to know what it had already tried. So the state is built to carry the consequences of the previous decision.

Four signals are stacked per observation. The first is the observation’s own features: point coordinates for line fitting, or coordinates, matching scores and local descriptors for correspondences. The second is an action indicator, +1 for the observations that formed the last minimum set and -1 for everything else. The third is the residual from the previous hypothesis, which is dense evidence about how the last model related to every observation, not only the ones that built it. The fourth is a counter of how often each observation has been selected so far.

We think of these last three as a navigation aid. The residuals say which direction looks promising. The usage counter is a breadcrumb trail, recording where the search has already been. The action indicator marks the current position. None of them changes the underlying geometry; they change what the policy can conclude from it.

View full-size image ↗

From one sample to the next: observation features, the selected set, residuals, and sampling history form a state transition.

One episode, one random start

RLSAC runs several episodes over the same input, and each episode opens with a randomly sampled minimum set. This costs almost nothing and buys two things. It guarantees a performance floor, because every episode starts from an unbiased draw and the policy can only improve on it. And it gives the policy a meaningful state to react to, since a residual field computed against a terrible hypothesis is far more informative than one computed against nothing.

The policy itself is an EdgeConv-style graph network. The choice is not cosmetic. Observations have no canonical order, and a policy that read them as an ordered vector would behave differently depending on how the input happened to be shuffled. The graph aggregates each observation’s neighbors into its own representation, which keeps the policy permutation-invariant and lets points exchange local context before any of them is proposed.

The network outputs a probability per observation, and the top-scoring set becomes the proposal. One detail worth stating plainly: that probability is not a claim about how good an individual observation is. It is a claim about the long-run return from including it. The two are not the same, and treating them as the same is a common way to misread what the policy has learned.

Duplicate sets are rejected. Every previously used minimum set is recorded, and if the top-scoring set has already been tried, the next one is drawn according to the probabilities and checked again until an unused set is found. Without this, a policy that has learned a good region will keep re-proposing the same set and burn the entire budget on a hypothesis it already has.

Training uses discrete Soft Actor-Critic with a replay buffer, sampling probabilistically so the policy keeps exploring. At test time the sampling switches to greedy selection. The split is the standard exploration-exploitation tradeoff, and we compared the other three combinations against it before settling on this one.

Episodes terminate on any of three conditions during training: the inlier count stops changing for two steps, the inlier ratio fails to beat the episode’s best for three steps, or the episode hits its step cap. Testing keeps only the cap. The distinction we wanted was between giving up on a search that has stopped making progress and stopping one that is still improving but has run out of budget.

Watching it work on a line

Set a few hundred points in a square, draw a line through some of them, let the rest be outliers. Any pair of points defines a candidate line. Every other point has a distance to it, and that distance is the feedback. The hypothesis visibly walks toward the ground truth across successive steps, which is satisfying to watch but not the interesting part.

The interesting part is what the policy does after it gets there. Once it finds a hypothesis with a high inlier ratio, it keeps working locally, refining the set rather than jumping to a different region of the data. A sampler without memory cannot do this, because it has no way to know it is already close. This is the clearest argument for putting the residual and history features in the state, and it shows up in the qualitative traces much more plainly than in any aggregate number.

Our synthetic data also places outliers inside the inlier band, which is what happens in real scenes. That rules out memorizing coordinates. What is learnable is the distribution of point groups that make a well-conditioned model, and that is what we wanted the policy to pick up.

View full-size image ↗

Illustrative sequence from the paper: hypotheses and residual evidence change across policy transitions.

From points to image correspondences

Two-view geometry changes the observation but not the interface. An observation is now a match between two image locations, described by its coordinates, its matching score and a local descriptor. A minimum set of eight goes to the eight-point solver, which returns a fundamental matrix, and consensus scoring measures how well every correspondence agrees with the resulting epipolar geometry.

In this setting the policy settles on rigid, well-localized feature points on buildings and similar structures. Nobody told it to prefer those. They are simply the matches that survive the geometric score, episode after episode, and the policy found them.

View full-size image ↗

The learned sampler receives feedback from a classical geometric solver.

What carries across the two tasks is not a trained network. It is the contract: encode observations, propose a minimum set, solve, score, feed the evidence back. Moving to a new problem means supplying task-appropriate features, a solver and a residual definition. The paper’s claim about transferability rests on that interface, together with the random restart and the duplicate rejection, not on the state transition by itself.

What we take from this

The lesson we keep coming back to is that reward can substitute for differentiation. If a downstream metric can score an output, it can train the module that produced it. There is no need to make an argmax, a solver or a threshold differentiable, which means the interpretable classical pieces get to stay. That is what allows a traditional estimator and a learned one to be optimized toward the same objective instead of sitting on opposite sides of an interface.

Memory also turns out to be worth more than prediction here. Feeding the last attempt’s residuals back into the state is cheap and dense. Training a better one-shot predictor is expensive and, in our setting, not obviously better.

And a learned sampler should never be allowed to underperform the random baseline it replaced. Seeding every episode with a random draw is what buys that.

Where it stays limited

An honest list. The inlier threshold is still a hand-set hyperparameter and it directly determines the reward, which is the same sensitivity MAGSAC++ was designed to remove; we reintroduced it and did not study it. The state only tracks the top-N correspondences by matching score, so a good match outside that set can never be selected. Our comparisons for two-view geometry are against classical methods only, even though we discuss learning-based samplers throughout; readers should not read those as head-to-head results. Transfer to other tasks is argued from the interface, not demonstrated. And we never analyzed cost: multiple episodes times up to fifteen steps, each with a graph forward pass, a solver call and a full residual computation, is a real bill, and we did not compare accuracy at matched iteration budgets.

The larger point

The discussion section of the paper makes the argument we actually believe. One-shot deep estimators are not reliable outside the distribution they were trained on, and their poor interpretability makes them hard to deploy in engineering settings. Classical estimators have clear behavior and well-defined scope, but cannot be folded into a learned framework because they are not differentiable. Combining the two gets both properties. If a component can be scored, it can be learned, and it can keep being the component you trust.

We are curious whether the same treatment extends to other non-differentiable pieces of the geometric stack. Noise covariance estimation in SLAM is the one we keep thinking about.

Conference Papers | Chang Nie

MRASfM: Multi-Camera Reconstruction and Aggregation through Structure-from-Motion in Driving Scenes

Driving is the wrong shape for photogrammetry

What prior methods assume

What makes it hard

The camera set is the unit

Building a reconstruction

Getting the road out of the way

Bundle adjustment on the rig

Merging drives that share no images

What we take from this

What it still assumes

Where it leads

MovSAM: A Single-image Moving Object Segmentation Framework Based on Deep Thinking

A photograph contains no motion

What earlier methods assume

What makes it hard

The mask as something you can argue with

Turning language into pixels: Cross-modal feature aggregation

The reasoning loop: Propose, segment, critique

What is and is not trained

What we take from this

What this opens

RLSAC: Reinforcement Learning Enhanced Sample Consensus for End-to-End Robust Estimation

The sampler is the part nobody trains

The sample is an action

What the policy gets to see

One episode, one random start

Watching it work on a line

From points to image correspondences

What we take from this

Where it stays limited

The larger point