Computer Vision | Chang Nie

Robust Multi-Model Fitting through Learning Neighbor Regions

Sat, 14 Sep 2024 10:33:19 +0000

Multi-model fitting addresses a universal challenge across robotic navigation, 3D scene reconstruction, and spatial computing: real physical environments rarely contain a single clean geometric structure. Instead, dozens of intersecting planar walls, ground surfaces, and boundaries co-exist simultaneously, deeply corrupted by sensor noise and unassociated outliers.

In classical robust estimation, instantiating a candidate geometric hypothesis requires only a minimal sample set of cardinality $m$—for example, $m=2$ points for fitting a 2D line, $m=3$ points for a 3D plane, or $m=8$ correspondences for an epipolar fundamental matrix.

When researchers apply naive random sampling (as in standard RANSAC) to multi-structure scenes, the mathematical foundation of uniform sampling breaks down catastrophically. With dozens of structures co-existing under heavy background outliers, picking two points at random almost invariably draws points from two entirely different physical objects, or samples pure noise. The probability of randomly drawing $m$ points that all belong to the exact same physical structure collapses exponentially as the number of models grows. Yet under classical pipelines, every invalid sample pair is still fed into algebraic solvers and scored across the entire dataset. Computational budgets are overwhelmingly consumed by solving and evaluating hypotheses that were doomed from the start.

Conversely, extending single-model fitters greedily—such as Sequential RANSAC, which fits one model, strips away its inliers, and repeats—introduces hard-assignment clustering distortions. Real structures often share points along intersections and boundaries; greedily stripping inliers deprives adjacent structures of necessary geometric support, causing severe model under-counting and missed detections.

Three further constraints shaped what we built. The method has to work without being told how many models are present, which rules out a family of approaches that take that count as input. It has to run at a usable speed. And it has to transfer across tasks, because a multi-model fitter that has to be re-engineered per problem is not a general tool.

View full-size image ↗

Learning sits at the two decision points in an otherwise classical pipeline.

The four things that actually go wrong

We organize the problem around four failure modes, and each component of the method answers one of them.

The first is insufficient utilization of geometric features. Existing methods either rely on hand-designed criteria that only look at part of the geometry, or they are neural networks that never see it explicitly.

The second is inefficient optimization caused by noise. Solver calls are spent on hypotheses that were hopeless from the start.

The third is model overlap, which is the fundamental difference between fitting one model and fitting several, and which most formulations cannot even represent.

The fourth is the non-differentiable pipeline. Discrete minimal-set sampling and the task-specific minimal solver both sit in the middle of the process, and neither has a gradient. Challenges arise in incorporating learning-based methods precisely because of this, and the two available escape routes are both unattractive: replace the exact solver with a differentiable relaxation and lose it, or use policy-gradient estimation and accept the variance.

What competing methods leave unresolved

Greedy single-model extensions carry the hard-assignment problem described above. MultiRANSAC fits K models per iteration instead of one, which requires K as prior knowledge and becomes computationally inefficient as K grows, because finding several good models in a single iteration is genuinely difficult.

Iterative hypothesis generators are the more sophisticated family, and their shared defect is temporal coupling. Because each new model is built on the results of previous iterations, errors accumulate through model overlap as the process proceeds. Our own ablation is honest about the tradeoff here: substituting a stronger learned sampler into LNR raises accuracy and makes the whole process considerably slower. We chose the cheaper option deliberately and we report what it costs.

Clustering-style methods, including preference-set clustering and networks that classify minimal sets as valid or invalid, work well on the problems they target and ignore the reuse of data. Information gathered while evaluating one hypothesis is not carried into the evaluation of another.

Another family replaces sampling heuristics with a global energy function over geometric errors and cluster regularity, or a convex relaxation of the assignment energy, or hypergraph mode-seeking. These formulations handle overlap by construction, and they rely on cost functions complex enough that they may not fully exploit the geometric information actually present in the data. That limits how well they carry across diverse and complex scenes.

Efficient sampling methods that exploit parallel computation are strong, and they need the number of instances as prior information, which limits where they can be applied.

Finally, end-to-end deep methods achieve good accuracy and operate as black boxes. That is a real obstacle to reliable deployment, and it is the specific thing we did not want to build.

Learn at the decisions, condition on the points

Our approach is to keep the classical pipeline exactly as it is and put learning only where a decision is being made: before solving, to decide which minimal sets deserve a solver call, and after solving, to decide how each hypothesis should move and how good it is.

The critical design choice is what the network is conditioned on. The learned quantities are functions of the geometric features of data points, not of the hypothesis parameters. That is what allows the sampler and the solver to stay in the pipeline as non-differentiable black boxes in the forward pass. No gradient has to pass through either of them.

A precise statement is worth making, because it is easy to overstate. The solver is used during training: the labels for both classifiers come from comparing a solved hypothesis against the ground-truth instance, and the coarse labels come from solving the sampled sets. What does not happen is differentiation through the solver or the sampler. They are used forward-only.

Each point is lifted by a Fourier feature mapping to improve spatial frequency encoding, passed through shared multilayer perceptrons implemented as point-wise convolutions, and max-pooled into a global descriptor. Point-wise and global features are concatenated into a 256-dimensional vector, so every point carries both its own local geometry and a summary of the scene. The coarse module needs that holistic view. The fine module, as it turns out, does not.

View full-size image ↗

Overlap is representable because a point can contribute to several hypotheses.

Screening before solving

The coarse module draws a large number of minimal sets at random, deliberately more than needed to ensure broad coverage, and predicts a confidence for each. Only the top-scoring sets reach the solver.

Two things about this are worth dwelling on. The first is why the sampling stays random rather than becoming iterative. Random sampling avoids the error accumulation that iterative generators suffer from, and it preserves coverage across the whole observation set. It is a pragmatic choice rather than an optimal one, and we say so.

The second is that the confidence is trained as a binary classification problem, which means the module learns to select promising minimal sets without any explicit degeneracy test. Classical pipelines hand-code checks to reject degenerate configurations. Here that rejection is learned from solved hypotheses compared against ground truth. We replaced a hand-written rule with a learned one, and the benefit is that it adapts to the data rather than to the person who wrote it.

Removing the coarse module hurts significantly, because the fine network cannot cope with the resulting candidate volume. Pretraining the coarse module separately instead of training the two jointly also does worse, and the reason is the most interesting thing in the training design.

Judging a hypothesis by its neighbourhood

For each surviving hypothesis we define a neighbour region: the data points within some range under a task-specific distance. Euclidean distance for lines, a directional distance for vanishing points, reprojection error for planes, the normalized epipolar constraint for motion. The points in that region are encoded, each point’s residual to the hypothesis is concatenated to its features, short regions are zero-padded so they can be batched, and two heads read the result.

One head predicts an increment rather than a model. For a line it predicts a small change to each endpoint. For a vanishing point, a homography or a fundamental matrix it predicts a corresponding adjustment, and that adjustment is added to the coarse hypothesis. The network never invents a model from scratch. Verifying and correcting a guess is easier than generating one, so the classical solver generates and the network corrects.

The other head predicts a score. This is where the paper takes a position that surprises people: consensus maximization does not favour the best model. Counting inliers is the standard proxy for hypothesis quality, and when models overlap it is the wrong proxy. Learning the score from the data around the hypothesis is what replaces it.

The region representation is also what makes overlap representable at all. Two crossing lines have a shared neighbourhood near their intersection, and the points there sit inside both regions and count as evidence for both hypotheses. No hard assignment is made anywhere.

And the fine module deliberately discards the global context that the coarse module relies on. Local details are more relevant for refining an individual hypothesis, and global information can introduce many irrelevant details that hinder the process. The same feature that helps screening hurts refinement. That asymmetry is a reusable coarse-to-fine principle, and it is the reason the two modules are not simply the same network applied twice.

View full-size image ↗

The fine module scores hypotheses from the observations around them rather than from their parameters alone.

Deciding how many models there are

The scored hypotheses are redundant. Several of them will describe the same structure, and the answer should mention it once. We adapted non-maximum suppression from object detection to this setting, replacing intersection-over-union with a geometric notion of similarity: endpoint distance and angle for lines, proximity on the Gaussian sphere for vanishing points, and similarity between the per-point distance vectors for homographies and fundamental matrices.

This is also the mechanism that decides how many models the scene contains. The count is an output of suppression rather than an input to the method, which is what we wanted from the start.

How the two modules train each other

Three losses. The coarse confidence and the fine score are both binary cross-entropy against labels derived from similarity to the ground-truth instance. The refinement head is trained with a smooth L1 objective on the parameter increments.

The coupling is the part worth explaining. The fine-level loss back-propagates into the coarse-level module, so the selector learns to prioritize hypotheses that the fine module is able to work with, rather than simply matching a predefined label. The two modules share the encoded region features, which is what makes the gradient path exist. Training jointly beats pretraining the coarse module, and the reason is that the fine signal reshapes what good means at the coarse stage.

One further observation from the ablations that we did not expect. Training only on harder scenes, with more lines present, matched or beat training on easier ones, and we attribute this to the greater data variability acting as a regularizer. The same effect appeared more weakly for the baseline sampler we compared against.

What we take from this

Decouple learning from non-differentiable components by changing what you condition on rather than by making them differentiable. That refusal is the strongest methodological statement in this work, and it is what preserves the exact solver.

Optimize a selector for downstream compatibility rather than for its own proxy label. Pre-training loses information that joint training keeps.

Consensus is not quality. When models overlap, inlier counting is the wrong measure, and assuming it is right is what causes models to be under-counted.

The value of a feature depends on the stage. Global context helps screening and hurts refinement.

And model the multiplicity structurally. If the interface forces each point into one model, no amount of training will represent two.

What it assumes

The classifiers need supervision in the form of ground-truth instances. Behaviour in a domain with no ground-truth models is not something we characterize.

The region radius and the sampling budget are task-dependent quantities that we do not specify how to set beyond that they vary, and the suppression thresholds are not stated either.

Transfer across tasks means re-deriving four task-specific pieces: the region distance metric, the solver, the refinement parameterization, and the suppression similarity. That is real per-task engineering, and we do not claim zero-shot adaptation.

Zero-padding is our only mechanism for variable-size neighbourhoods, and whether it dilutes the encoding for regions with few points is not something we examined.

And random sampling is a tradeoff rather than an optimum. Our own ablation shows a better sampler would improve accuracy, and we did not resolve whether a cheap learned sampler could get both.

Why keep the two-step pipeline

We kept the established two-step strategy for a specific reason: interpretability. Numerical methods have inherent interpretability and clearly defined application scopes, and they struggle to fully exploit the geometric features of the data. End-to-end deep methods perform well and cannot tell you how they got there.

Because LNR keeps the pipeline, its outputs are explicit geometric models. The network accepts generic unordered data points, covering image correspondences, 3D point clouds and pixel coordinates, and the loss operates on geometric features rather than on hypotheses. That is what maintains compatibility with a broad spectrum of solvers and sampling processes, differentiable or not.

The stated ambition is modest by design. We hope this introduces a new idea for approaching multi-model fitting in a learned way, and we would rather have a method whose behaviour can be interrogated than one whose accuracy we cannot explain.

DiffSAC: Diffusion-guided Sampling for Consensus-based Robust Estimation

Sat, 31 Aug 2024 11:15:11 +0000

The expensive part is not the sampling

Sample consensus is the standard answer to robust estimation, and it has one structural property worth stating before anything else: a candidate set is not just proposed, it is spent. Every minimum set has to be solved into a model and then scored against all the observations before you know whether it was any good. With a few thousand points and a budget of a few thousand iterations, almost all of that work goes into combinations that were never going to produce the right model.

In geometric estimation, a minimal sample set is the smallest subset of observations required to compute model parameters uniquely: fitting a spatial line requires $s=2$ points, fitting a 3D plane requires $s=3$ points, estimating a homography requires $s=4$ correspondence pairs, and solving for a two-view fundamental matrix using the normalized 8-point algorithm requires $s=8$ pairs (or $s=7$ for the non-linear algebraic variant). Standard RANSAC draws these samples uniformly at random. According to binomial probability, if the dataset contains an outlier ratio $\epsilon \in (0, 1)$, the theoretical number of random sampling iterations $K$ required to ensure with confidence probability $p$ (typically $p=0.99$) that at least one clean, all-inlier minimal set is drawn is given by:

$$K = \frac{\ln(1 - p)}{\ln\left(1 - (1 - \epsilon)^s\right)}$$

Under moderate outlier contamination, $K$ remains tractable. However, in challenging unconstrained robotic environments featuring severe visual repetition, reflective surfaces, or extreme dynamic clutter, outlier ratios frequently exceed $\epsilon = 0.8$ or $0.9$. Under these conditions, the combinatorial denominator collapses catastrophically. When estimating an 8-point fundamental matrix ($s=8$) with $\epsilon = 0.8$, the probability of drawing an all-inlier minimal set in a single random trial is merely $(1 - \epsilon)^s = (0.2)^8 \approx 2.56 \times 10^{-6}$. To guarantee a 99% success rate, the system theoretically mandates over 1.8 million iterations. Even when practitioners enforce an empirical cap of several thousand iterations, the overwhelming majority of computational cycles are expended solving algebraically ill-conditioned models and calculating residual distances across thousands of points for hypotheses that were physically doomed from inception.

Furthermore, uniform random sampling is entirely oblivious to geometric conditioning. Two feature correspondences situated in close spatial proximity may be individually accurate, but numerically they span a near-zero spatial baseline, yielding a degenerate or numerically unstable model. Similarly, planar coplanar correspondences in two-view geometry induce fundamental matrix degeneracies. Uniform random sampling cannot evaluate the joint compatibility of a set of points as an integrated geometric entity; it merely aggregates independent random draws.

Critical Analysis of Prior Heuristics

Over the past two decades, extensive research has attempted to inject intelligence into sample generation, broadly bifurcating into three methodologies:

Prioritized Sampling and Local Neighborhood Heuristics: PROSAC sorts correspondence pairs by local matching metrics (such as nearest-neighbor distance ratios), drawing preferentially from top-ranked candidates. NAPSAC and Progressive NAPSAC assume inliers exhibit spatial proximity in image space, drawing samples within localized spatial balls. While effective in benign settings, these heuristics exhibit distinct failure modes: high feature similarity does not prevent geometric degeneracy, and restricting samples to tight local balls sacrifices wide-baseline triangulation geometry, collapsing when structural textures (e.g., repeating window grids or tiled floors) form locally coherent yet globally false matches.
Post-Hypothesis Refinement and Local Optimization: Algorithms such as LO-RANSAC, GC-RANSAC, and MAGSAC++ focus on the opposite phase of the estimation pipeline. Rather than altering how initial minimal sets are drawn, they accept standard random sampling and optimize the model once a promising candidate emerges—via graph-cut spatial clustering, iteratively reweighted least-squares, or marginalizing across continuous noise thresholds. These post-refinement tools are highly effective, and we actively integrate them into our downstream pipeline; however, they leave the primary bottleneck unaddressed: the astronomical cost of discovering that very first high-quality hypothesis within extreme outlier regimes.
End-to-End Differentiable RANSAC (DSAC Frameworks): DSAC and its successors relax the discrete hypothesis selection step into a continuous, differentiable probabilistic formulation, enabling end-to-end backpropagation. While theoretically elegant, direct policy gradient estimators behave as opaque black boxes in mission-critical robotics. When an autonomous vehicle or SLAM frontend experiences an unexpected trajectory jump, engineers cannot trace which specific correspondences caused the degenerate estimate. Differentiable models lack the modular, verifiable audit trail that production systems demand.

These limitations prompted our foundational inquiry: Can we reformulate minimal set sampling from passive, stochastic trial-and-error into goal-directed, conditional probabilistic generation? Instead of attempting to replace proven closed-form algebraic solvers with neural networks, we task a generative model with the hardest component of the problem: directly sampling high-quality, mutually compatible minimal sets.

Generating Candidate Sets Instead of Guessing

DiffSAC reconceptualizes hypothesis generation as reverse sampling from a Geometry-Conditioned Diffusion Model. If a neural network can model the posterior probability distribution of minimal sets that reliably yield correct models given an input observation cloud, inference reduces to generating a compact batch of candidate sets (e.g., 10 to 20 candidates). Every candidate is mathematically guaranteed to exhibit high all-inlier probability and stable numerical conditioning, which are subsequently resolved by deterministic classical solvers and evaluated via consensus scoring.

This paradigm establishes a direct conceptual parallel to text-to-image diffusion models. In text-to-image synthesis, a single conditioning prompt generates multiple diverse, photorealistic images from different random noise seeds, enabling users to select the most coherent realization. DiffSAC applies identical reasoning to geometric estimation: given an identical cloud of noisy correspondences, injecting independent Gaussian noise vectors into the reverse diffusion chain yields a small portfolio of diverse, non-degenerate minimal sets in parallel.

Critically, this formulation completely bypasses the computational latency that traditionally burdens generative diffusion models in robotics. In image generation, diffusion models denoise high-dimensional tensors containing millions of continuous pixels across dozens of autoregressive steps. In DiffSAC, by contrast, the generative target is merely a one-dimensional continuous confidence scalar per observation. For a typical visual keypoint set containing 2,000 matches, the total state vector is only 2,000 dimensions. Coupled with high-order ODE solvers, the complete reverse diffusion chain executes in 10 to 15 milliseconds on modern GPUs—an overhead that is dramatically outweighed by the computational savings achieved by eliminating hundreds of thousands of full-dataset model evaluations.

The pivotal theoretical abstraction centers on the definition of confidence: we define confidence not as an isolated measure of whether an individual point is an inlier, but as a joint set-level property measuring whether an observation belongs to a well-conditioned, mutually consistent minimal set.

Conditional Diffusion Process and Mathematical Formulation

Let the input geometric observation set be denoted by $\chi = \{x_i, d_i\}_{i=1}^N$, where $x_i \in \mathbb{R}^d$ represents spatial coordinates (e.g., 2D image coordinates or 3D point positions) and $d_i \in \mathbb{R}^{d_f}$ denotes local appearance feature descriptors (e.g., SuperPoint or SIFT embeddings).

Let $\boldsymbol{c}_0 \in [0, 1]^N$ represent the ideal target confidence field over the observation cloud. In continuous-time diffusion formulations, the forward corruption process is governed by an Ito Stochastic Differential Equation (SDE) that progressively diffuses $\boldsymbol{c}_0$ into standard Gaussian white noise over continuous time $t \in [0, 1]$:

$$\mathrm{d}\boldsymbol{c}_t = \boldsymbol{f}(\boldsymbol{c}_t, t)\mathrm{d}t + g(t)\mathrm{d}\boldsymbol{w}$$

The corresponding reverse denoising trajectory is defined by the Probability Flow Ordinary Differential Equation (ODE):

$$\mathrm{d}\boldsymbol{c}_t = \left[ \boldsymbol{f}(\boldsymbol{c}_t, t) - \frac{1}{2} g(t)^2 \nabla_{\boldsymbol{c}_t} \log p_t(\boldsymbol{c}_t \mid \chi) \right] \mathrm{d}t$$

Where the conditional score function $\nabla_{\boldsymbol{c}_t} \log p_t(\boldsymbol{c}_t \mid \chi)$ is approximated by a parameterized neural denoiser $\boldsymbol{s}_\theta(\boldsymbol{c}_t, t, \chi)$. Denoising is strictly conditioned on the empirical geometric observations $\chi$, anchoring the trajectory to actual epipolar geometry constraints rather than unconstrained prior hallucinations.

During supervised training, ground-truth geometry provides the supervisory signal $\boldsymbol{c}_0$: given a known true model, the system samples a minimal set of verified inliers that yields a well-conditioned algebraic solve, setting their target confidence to 1 while assigning all remaining points 0. The denoiser is trained via an $x_0$-prediction formulation to predict the clean confidence field $\hat{\boldsymbol{c}}_\theta(\boldsymbol{c}_t, t, \chi)$ directly, minimizing the Mean Squared Error (MSE) objective:

$$\mathcal{L}_{\text{diff}}(\theta) = \mathbb{E}_{t, \boldsymbol{c}_0, \boldsymbol{\epsilon}}\left[ \left\| \hat{\boldsymbol{c}}_\theta(\boldsymbol{c}_t, t, \chi) - \boldsymbol{c}_0 \right\|_2^2 \right]$$

This objective forces the network to learn which subsets of observations cooperatively eliminate algebraic singularities under varying noise corruptions.

View full-size image ↗

Forward corruption supplies training targets; reverse diffusion generates candidate minimum sets.

Permutation-Invariant Transformer Denoiser Architecture

In geometric perception, the ordering of feature points in an input array is entirely arbitrary. Imposing fixed coordinate orderings or 1D convolutions introduces spurious sequential biases and prevents models from handling varying numbers of observations $N$.

To preserve physical validity, we construct the denoiser as a Transformer without positional encodings:

Feature Embedding: Each observation combines geometric coordinates $x_i$, appearance descriptors $d_i$, and the current noisy confidence scalar $c_{t, i}$ into a unified vector, projected through a multi-layer perceptron (MLP) into a latent representation $z_i^{(0)} \in \mathbb{R}^D$;
All-to-All Self-Attention: Latent tokens pass through stacked self-attention and cross-attention blocks. The self-attention matrix: $$\boldsymbol{A} = \text{Softmax}\left(\frac{\boldsymbol{Q}\boldsymbol{K}^T}{\sqrt{d_k}}\right)$$ enables every observation to adjust its confidence dynamically based on the geometric distances, disparity gradients, and appearance similarities of all other candidates in the cloud. This global receptive field is precisely what evaluates multi-point geometric compatibility;
Strict Permutation Equivariance: By omitting positional encodings, the network satisfies formal equivariance: for any permutation matrix $\boldsymbol{P} \in \mathbb{R}^{N \times N}$, $\hat{\boldsymbol{c}}_\theta(\boldsymbol{P}\boldsymbol{c}_t, t, \boldsymbol{P}\chi) = \boldsymbol{P}\hat{\boldsymbol{c}}_\theta(\boldsymbol{c}_t, t, \chi)$. Shuffling the input array guarantees an identical, co-permuted confidence output.

Ablation benchmarks confirm that replacing the Transformer with PointNet++ or k-NN graph convolutions (DGCNN) severely degrades wide-baseline generalization, proving that all-to-all attention is essential for resolving long-range geometric dependencies across wide camera baselines.

View full-size image ↗

No positional encoding is used, preserving permutation invariance.

Fast Inference: Batch-Parallel Diffusion and Consensus Verification

At test time, DiffSAC executes an efficient, highly parallelized pipeline:

Batch Parallel Initialization: The observation cloud $\chi$ is replicated $B$ times in GPU memory (typically $B=20$). Each batch replica is initialized with an independent standard Gaussian noise vector $\boldsymbol{c}_1^{(b)} \sim \mathcal{N}(\mathbf{0}, \boldsymbol{I})$;
High-Order Numerical ODE Solvers: Invoking DPM-Solver++ compresses the reverse trajectory from hundreds of steps down to 10 to 15 discrete evaluation steps, executing concurrently across all $B$ replicas within 12 milliseconds;
Top-k Hard Extraction: From each refined confidence field $\boldsymbol{c}_0^{(b)}$, the system selects the $s$ observations with the highest confidence values via a hard Top-k operator, yielding $B$ candidate minimal sets $\mathcal{S}^{(b)}$;
Deterministic Algebraic Solves & Consensus Scoring: Closed-form solvers (e.g., SVD for line and plane fitting, normalized 8-point algorithm for fundamental matrices) solve for hypotheses $\mathcal{M}^{(b)}$. Each model is scored against all observations using standard consensus metrics (such as MSAC or truncated M-estimator residuals), selecting the hypothesis with the highest inlier support. Finally, the winning model is refit over its entire consensus set.

This split—stochasticity at initialization, determinism at selection—is fundamental. Independent noise seeds allow the diffusion process to explore distinct modal basins of the conditional distribution. If one candidate encounters a localized singularity, alternative parallel chains provide robust, non-degenerate solutions, eliminating single-point failure modes.

View full-size image ↗

The generated confidence changes which observations enter the minimum set.

View full-size image ↗

Conditioned confidence proposals feed the classical two-view geometry pipeline.

The Decisive Ablation: Why Iterative Diffusion Is Irreplaceable

When evaluating DiffSAC, a natural skepticism arises: Why employ an iterative diffusion process? Why not train a feedforward regression network to predict the confidence vector directly in a single forward pass?

To resolve this question definitively, we conducted a direct ablation: retaining the identical Transformer backbone, loss function, and training dataset, but eliminating the noise schedule and training the network to regress confidence values $\hat{\boldsymbol{c}}$ from clean observations $\chi$ in a single step.

The empirical disparity was stark:

The single-step feedforward regressor produced diffuse, poorly separated confidence distributions. It consistently failed to establish sharp contrast between true inliers and degenerate near-inliers, resulting in candidate sets heavily contaminated by outliers;
In contrast, the iterative reverse diffusion process exhibited systematic self-correcting dynamics: early steps leveraged global attention to purge obvious outlier matches; intermediate steps pruned geometrically degenerate configurations (such as collinear points or negligible parallax); and terminal steps rapidly polarized confidence into a sharp, decisive set of high-leverage inlier landmarks.

This comparison isolates the true driver of performance: iterative diffusion succeeds not because of added stochasticity, but because it decomposes an intractable high-order combinatorial search problem into a sequence of smooth, gradient-guided trajectory refinements.

Furthermore, because DiffSAC preserves standard algebraic solvers and consensus scoring, it integrates seamlessly with existing local optimization frameworks. Feeding DiffSAC candidate hypotheses into LO-RANSAC for secondary graph-cut refinement produces compounding accuracy improvements, uniting generative global sampling with classical local convergence.

Comprehensive Benchmark Evaluation and System Boundaries

We validated DiffSAC across synthetic stress tests and extensive real-world benchmarks, including ScanNet, MegaDepth, and PhotoTourism. Tasks encompassed line fitting, 3D planar segmentation, fundamental/essential matrix estimation, and homography recovery:

Under extreme 80% synthetic outlier contamination, classical RANSAC achieved less than 15% estimation success within 5,000 iterations. DiffSAC achieved a 94% success rate with only 20 candidate evaluations, reducing average rotation and translation errors by an order of magnitude;
In real-world wide-baseline scenes with repetitive architectural textures, DiffSAC dramatically suppressed degenerate models, outperforming PROSAC, NG-RANSAC, and MAGSAC++ in pose estimation AUC metrics.

We also explicitly state the system’s operational boundaries:

Model Specialization vs. Universality: Current models are trained for specific geometric degrees of freedom (e.g., separate network weights for homographies versus essential matrices). Developing a single unified foundation model capable of conditioning on arbitrary geometric constraints remains an active frontier;
Computational Hardware Requirements: While DPM-Solver++ reduces inference to 10-15 steps, all-to-all attention across thousands of points requires GPU tensor parallelism. On resource-constrained micro-controllers or low-power embedded platforms lacking GPU acceleration, running Transformer diffusion chains requires model pruning and INT8 quantization.

Engineering Philosophy: Reclaiming Auditability in Learned Geometry

The core thesis of DiffSAC is an intentional defense of modular engineering in deep learning: assign neural networks to solve difficult, high-dimensional pattern recognition and combinatorial sampling, while entrusting deterministic classical algorithms with well-understood algebraic solving and residual consensus verification.

Discovering well-conditioned inlier subsets among tens of thousands of noisy correspondence candidates is a non-convex, combinatorial search problem where deep learning excels. Conversely, calculating model parameters from a minimal set and measuring reprojection residuals are analytical, closed-form operations that neural networks should never approximate.

This disciplined separation delivers exceptional estimation accuracy while preserving absolute engineering interpretability. Whenever an anomaly occurs, practitioners can inspect the confidence field, examine the condition number of the minimal set matrix, and verify the algebraic consensus curve, ensuring that critical robotic perception systems remain transparent, verifiable, and safe.

RLSAC: Reinforcement Learning Enhanced Sample Consensus for End-to-End Robust Estimation

Sun, 01 Oct 2023 00:00:00 +0000

The sampler is the part nobody trains

Robust estimation carries most of geometric vision on its back. Camera pose, motion segmentation, wide-baseline matching, plane fitting, line fitting: strip away the application and they all come down to the same question. Given observations where a large fraction are simply wrong, recover the model that the correct ones agree on. RANSAC has been the default answer to that question for four decades. It draws a minimum set, solves for a model from that set, counts how many observations agree with the result, and tries again. Two points define a line. Eight correspondences define a fundamental matrix.

The drawing step is deliberately blind. Every observation is treated as equally worth trying, and nothing about the data ever enters the decision. That bet pays off when outliers are rare. It stops paying off quickly. Raise the outlier fraction and the probability of drawing a set entirely from the inliers falls away, and the number of iterations needed to find a good model climbs with it. The evidence that would have made the search easy is sitting in the observations themselves, and the sampler never looks at it.

The second problem is quieter, and it is the reason we wrote this paper. A uniform random draw is not differentiable. You cannot attach it to a network, and you cannot train it against an objective that lives downstream of it. So the robust estimator ends up parked at the edge of a learned system: an off-the-shelf component wrapped in interfaces, never optimized together with the parts around it.

Earlier work has attacked this from several directions, and each one leaves something on the table. LO-RANSAC keeps sampling inside the inliers of the best hypothesis so far, which improves a model it has already found rather than the search that found it. NAPSAC restricts sampling to a local neighborhood and buys better-conditioned sets at the cost of global coverage; Progressive NAPSAC’s expanding neighborhood patches that assumption rather than replacing it. GC-RANSAC and MAGSAC++ change how hypotheses are scored, not how they are proposed. PROSAC and USAC reorder or schedule the sampling, but the ordering rule is designed by hand and frozen before the first hypothesis has been scored. Methods such as NG-RANSAC learn a useful preference over points, and the filtering approaches around it discard bad candidate sets early, yet neither treats the sequence of sampling decisions as something worth planning. DSAC takes the most direct route and makes selection probabilistic so gradients flow through the pipeline. That works, but it requires the sampler, the solver and the scoring step to all be differentiable, and the classical pieces we trust most are exactly the ones that are not.

The sample is an action

Our starting observation is that the two problems have the same shape. In reinforcement learning, an agent reads a state, picks an action, and the environment reports what happened. Sampling a minimum set has that structure already. The observations are the state. The set chosen from them is the action. And the environment exists and is running: it is the solver and the consensus function.

The reward is the interesting part, because the pipeline produces one for free. Once the solver turns a proposed set into a model, the consensus function measures what fraction of all observations agree with it. That number scores the model, and since the model came out of that particular set, it also scores the choice that produced it. No labels for which points are inliers are needed anywhere in the loop. Any task that already ships a solver and a residual definition comes with its own reward.

That last point matters more than it sounds. Training from the joint outcome means the policy is never taught which individual observations are good. It is taught whether the model that came out of a combination was good. Pointwise inlier classification is a different and much stronger assumption than set-level scoring, and it is the assumption we did not want to make.

Question	RLSAC’s answer
What is learned?	The sampling policy; the solver and the scoring function stay as they are
What supervision is needed?	Nothing beyond the consensus score the pipeline already computes
What enters the state?	Observation features, the chosen set, residuals against the last hypothesis, and usage counts
Where is it used?	Synthetic line fitting and real two-view fundamental-matrix estimation

What the policy gets to see

A state that contains only the observations is not enough. The same data at step one and step ten would look identical, and the policy would have no way to know what it had already tried. So the state is built to carry the consequences of the previous decision.

Four signals are stacked per observation. The first is the observation’s own features: point coordinates for line fitting, or coordinates, matching scores and local descriptors for correspondences. The second is an action indicator, +1 for the observations that formed the last minimum set and -1 for everything else. The third is the residual from the previous hypothesis, which is dense evidence about how the last model related to every observation, not only the ones that built it. The fourth is a counter of how often each observation has been selected so far.

We think of these last three as a navigation aid. The residuals say which direction looks promising. The usage counter is a breadcrumb trail, recording where the search has already been. The action indicator marks the current position. None of them changes the underlying geometry; they change what the policy can conclude from it.

View full-size image ↗

From one sample to the next: observation features, the selected set, residuals, and sampling history form a state transition.

One episode, one random start

RLSAC runs several episodes over the same input, and each episode opens with a randomly sampled minimum set. This costs almost nothing and buys two things. It guarantees a performance floor, because every episode starts from an unbiased draw and the policy can only improve on it. And it gives the policy a meaningful state to react to, since a residual field computed against a terrible hypothesis is far more informative than one computed against nothing.

The policy itself is an EdgeConv-style graph network. The choice is not cosmetic. Observations have no canonical order, and a policy that read them as an ordered vector would behave differently depending on how the input happened to be shuffled. The graph aggregates each observation’s neighbors into its own representation, which keeps the policy permutation-invariant and lets points exchange local context before any of them is proposed.

The network outputs a probability per observation, and the top-scoring set becomes the proposal. One detail worth stating plainly: that probability is not a claim about how good an individual observation is. It is a claim about the long-run return from including it. The two are not the same, and treating them as the same is a common way to misread what the policy has learned.

Duplicate sets are rejected. Every previously used minimum set is recorded, and if the top-scoring set has already been tried, the next one is drawn according to the probabilities and checked again until an unused set is found. Without this, a policy that has learned a good region will keep re-proposing the same set and burn the entire budget on a hypothesis it already has.

Training uses discrete Soft Actor-Critic with a replay buffer, sampling probabilistically so the policy keeps exploring. At test time the sampling switches to greedy selection. The split is the standard exploration-exploitation tradeoff, and we compared the other three combinations against it before settling on this one.

Episodes terminate on any of three conditions during training: the inlier count stops changing for two steps, the inlier ratio fails to beat the episode’s best for three steps, or the episode hits its step cap. Testing keeps only the cap. The distinction we wanted was between giving up on a search that has stopped making progress and stopping one that is still improving but has run out of budget.

Watching it work on a line

Set a few hundred points in a square, draw a line through some of them, let the rest be outliers. Any pair of points defines a candidate line. Every other point has a distance to it, and that distance is the feedback. The hypothesis visibly walks toward the ground truth across successive steps, which is satisfying to watch but not the interesting part.

The interesting part is what the policy does after it gets there. Once it finds a hypothesis with a high inlier ratio, it keeps working locally, refining the set rather than jumping to a different region of the data. A sampler without memory cannot do this, because it has no way to know it is already close. This is the clearest argument for putting the residual and history features in the state, and it shows up in the qualitative traces much more plainly than in any aggregate number.

Our synthetic data also places outliers inside the inlier band, which is what happens in real scenes. That rules out memorizing coordinates. What is learnable is the distribution of point groups that make a well-conditioned model, and that is what we wanted the policy to pick up.

View full-size image ↗

Illustrative sequence from the paper: hypotheses and residual evidence change across policy transitions.

From points to image correspondences

Two-view geometry changes the observation but not the interface. An observation is now a match between two image locations, described by its coordinates, its matching score and a local descriptor. A minimum set of eight goes to the eight-point solver, which returns a fundamental matrix, and consensus scoring measures how well every correspondence agrees with the resulting epipolar geometry.

In this setting the policy settles on rigid, well-localized feature points on buildings and similar structures. Nobody told it to prefer those. They are simply the matches that survive the geometric score, episode after episode, and the policy found them.

View full-size image ↗

The learned sampler receives feedback from a classical geometric solver.

What carries across the two tasks is not a trained network. It is the contract: encode observations, propose a minimum set, solve, score, feed the evidence back. Moving to a new problem means supplying task-appropriate features, a solver and a residual definition. The paper’s claim about transferability rests on that interface, together with the random restart and the duplicate rejection, not on the state transition by itself.

What we take from this

The lesson we keep coming back to is that reward can substitute for differentiation. If a downstream metric can score an output, it can train the module that produced it. There is no need to make an argmax, a solver or a threshold differentiable, which means the interpretable classical pieces get to stay. That is what allows a traditional estimator and a learned one to be optimized toward the same objective instead of sitting on opposite sides of an interface.

Memory also turns out to be worth more than prediction here. Feeding the last attempt’s residuals back into the state is cheap and dense. Training a better one-shot predictor is expensive and, in our setting, not obviously better.

And a learned sampler should never be allowed to underperform the random baseline it replaced. Seeding every episode with a random draw is what buys that.

Where it stays limited

An honest list. The inlier threshold is still a hand-set hyperparameter and it directly determines the reward, which is the same sensitivity MAGSAC++ was designed to remove; we reintroduced it and did not study it. The state only tracks the top-N correspondences by matching score, so a good match outside that set can never be selected. Our comparisons for two-view geometry are against classical methods only, even though we discuss learning-based samplers throughout; readers should not read those as head-to-head results. Transfer to other tasks is argued from the interface, not demonstrated. And we never analyzed cost: multiple episodes times up to fifteen steps, each with a graph forward pass, a solver call and a full residual computation, is a real bill, and we did not compare accuracy at matched iteration budgets.

The larger point

The discussion section of the paper makes the argument we actually believe. One-shot deep estimators are not reliable outside the distribution they were trained on, and their poor interpretability makes them hard to deploy in engineering settings. Classical estimators have clear behavior and well-defined scope, but cannot be folded into a learned framework because they are not differentiable. Combining the two gets both properties. If a component can be scored, it can be learned, and it can keep being the component you trust.

We are curious whether the same treatment extends to other non-differentiable pieces of the geometric stack. Noise covariance estimation in SLAM is the one we keep thinking about.

3D Object Detection and Tracking Based on Lidar-Camera Fusion and IMM-UKF Algorithm Towards Highway Driving

Tue, 04 Apr 2023 00:00:00 +0000

The design constraint behind this work is not accuracy. It is latency.

Most fusion-based 3D detection algorithms run below 15 Hz on a workstation, and a workstation is not what drives a car. An onboard computing centre is slower still, and it is not dedicated to perception: planning, control and the rest of the stack have to run concurrently, which takes a further bite out of the same budget.

Now put a number on what 15 Hz costs. At highway speed the vehicle covers about 1.94 metres between two consecutive frames. For that distance the perception system effectively does not see anything that has just happened, and we call that being time blind. The consequences are not abstract. A system that reacts to where a car was two metres ago is reacting to a situation that no longer exists.

Thirty hertz substantially reduces the blind distance and leaves room for the other modules to run. That target, on affordable hardware rather than a development machine, is what shaped every choice in this paper. It also means we made a deliberate trade: LiDAR-only methods achieve better detection accuracy than fusion methods do, and we did not set out to beat them on that axis. We set out to be competitive while running at a frequency they cannot reach.

What the fusion families cost

There are three ways to combine an image with a point cloud, and each has a price.

Detecting in 2D and lifting the boxes into the point cloud, the frustum approach, makes the whole system depend on the 2D detector. An error in the image branch propagates into 3D with no way to recover it.

Fusing at the feature or proposal level and training end to end produces strong results, and it loses original information in the process of fusing at the region level. These methods also tend to be complex and to run slowly, which is the specific problem we were trying to solve.

The third family projects the point cloud into an image feature map and augments each point with what it finds there. The architecture stays close to a LiDAR-only detector, which is what we wanted, and the pillar encoder keeps the runtime low. We chose this family for exactly that reason.

Within LiDAR-only detection the lineage matters too. Voxel-based 3D convolution was accurate and slow. Sparse convolution improved efficiency without making it fast enough. Pillars, which reduce the problem to 2D convolutions, hit the best tradeoff between accuracy and speed, and we built on that.

For tracking, the choice of a classical filter over a learned one was deliberate rather than conservative. Learning-based trackers perform poorly in real time, and neural data association is time-consuming, so the Hungarian algorithm stays. Batch tracking is ruled out for a different reason: it uses all data, and a driving system only ever has history. 2D tracking is ruled out because it cannot tell you an object’s position and size in 3D.

The closest filter-based alternative is a well-known 3D tracker that is simple and efficient and uses a single constant-velocity model. A constant-velocity model has no way to represent a car that is turning, so its state estimate degrades in exactly the situations where prediction matters most. A monocular tracker uses an extended Kalman filter under one camera. An urban JPDA-IMM-UKF system exists and uses traditional clustering for detection, which produces fragmented trajectories and hurts tracking performance downstream.

IMM itself is standard in aviation and nearly absent in vehicles, and the reason is instructive. Aircraft maneuver drastically: large speed changes, fast heading changes, so a switching model earns its complexity. Vehicle motion is comparatively mild. And with little prior work in the automotive setting, the parameters that IMM needs are hard to set.

What makes it hard

Fusing at full image resolution, per point, at thirty hertz, is a budget problem before it is an accuracy problem. The number of image channels you attach to each point is the knob you turn, and we tuned it as a tradeoff rather than maximizing it.

Data-layer fusion is also fragile in a specific and quiet way. Because each point is matched to an image location, a calibration error puts the sampled feature at the wrong pixel for every point, and the resulting input looks entirely plausible. There is no error signal. The system does not know it is being fed garbage.

Choosing a motion model is a lose-lose if you must choose one. Constant velocity is simple and strong for linear motion. Constant turn rate and velocity handles rotation. Either alone is wrong in the other regime. Switching between them is the answer, and switching introduces a set of prior parameters: the state switching matrix, the state covariance, the process and measurement noise covariances. These are conventionally set from experience, and empirical values are not guaranteed to apply to a different scenario. That is a real methodological problem, not a tuning nuisance.

There is a nonlinearity problem too. Constant-velocity transitions are linear in the state. Constant-turn transitions involve position expressed through velocity and yaw rate multiplied by sine and cosine of the heading, which is not linear, so the filter needs sigma points rather than a plain Kalman update.

Tracking also has to survive gaps and resist noise at the same time. Occlusion causes detections to drop, and being too permissive about starting new tracks admits detector false positives. The two failure modes pull in opposite directions.

And the physics caps detection regardless of architecture. In one of our examples, three very distant cars return only three to five LiDAR points each. Too few points and the detection simply does not fire.

Attach a neighbourhood, not a pixel

The first half of the idea addresses the fragility of data-layer fusion. Instead of sampling a single projected pixel for each point, we sample a 5×5 window of image features around the projection and attach the whole neighbourhood to the point.

The consequence is that the network learns the matching relationship between the two modalities rather than trusting the calibration exactly. A point whose projection lands a pixel or two off still finds relevant appearance features in its window. This was the single most useful conceptual move in the detection half, and it is what makes the difference between fusion that survives calibration error and fusion that quietly degrades.

The features come from a full-resolution extractor, which matters for a related reason: the point has to map to a precise image location for the window to mean anything, and a downsampled feature map destroys that precision. The encoder is VGG-based with two modifications. We reduced the channel count to keep optimization manageable, and we cut the network off in the middle so that the feature map does not shrink too far, which would make it useless for generating fusion data. A top-down decoder upsamples and merges with same-resolution encoder maps, with 1×1 convolutions controlling the output dimension.

View full-size image ↗

A local window learns the correspondence between the two modalities instead of assuming the calibration is exact.

Collapsing the height axis to get the frame rate

The augmented points then go through a pillar encoder. Points are binned onto a bird’s-eye-view grid with cell size and a cap on how many points a cell keeps, with random subsampling beyond the cap. The height axis is collapsed to a single cell, which is the trick that turns 3D convolution into 2D convolution and is where the frame rate comes from.

Each point carries its geometry and appearance plus five offset features that describe where it sits relative to its cell’s mean and its voxel centre. A PointNet layer expands these, max-pooling reduces each cell to one representative, and the cells are scattered back into a pseudo-image. A 2D convolutional backbone and a region proposal head then predict oriented 3D boxes.

The cost of the pillar representation is that height structure inside a cell is discarded and the detection range is bounded by the camera’s field of view, since fusion needs pixels. We accept both as the price of the frequency target.

One loss term deserves a note because it looks redundant and is not. The yaw residual is computed as the sine of the difference between predicted and anchor angle, and sine is symmetric. A car facing forward and a car facing backward produce the same residual. Regression alone cannot break the tie, which is why a separate direction classification term exists in the loss.

View full-size image ↗

The fusion-augmented point cloud is encoded as a bird’s-eye-view pseudo-image.

Two motion models, and a search for the priors

The tracker predicts every existing trajectory, associates predictions to detections by the Hungarian algorithm on 3D overlap, updates matched tracks, and manages births and deaths for everything unmatched.

The interesting part is the update. Two models run in parallel, one with a constant-velocity state and one with a constant turn rate and velocity state. Mixing probabilities are computed from the previous model weights and the switching matrix, mixed estimates are formed for each model, each model’s sigma-point filter predicts forward, and the two are combined into the IMM prediction. After association, each model is updated for the matched detection and the model probabilities are recomputed from the measurement likelihoods. The pairing is complementary by construction: the constant-velocity model estimates linear motion cheaply and well, and the constant-turn model covers the rotational case it cannot.

Life-cycle management handles the gap-versus-noise tension with two guardrails. An unmatched trajectory keeps predicting for a bounded number of frames before it is terminated, because an unmatched track usually means occlusion rather than disappearance, and restarting the trajectory later would fragment it. An unmatched detection has to persist for several consecutive frames before a track is born, which filters out occasional false positives. The two rules guard opposite failure modes.

The priors are where we departed from convention. Rather than setting the switching matrix and the covariances from experience, we treat them as an optimization problem and fit them with particle swarm optimization. The name invites a misreading worth clearing up: particle swarm optimization is a derivative-free search over parameters, it has nothing to do with particle filtering, and no particle filter appears anywhere in this work.

Two caveats on that fitting, and we would rather state them than have them discovered. The optimization runs offline and per dataset, so it is not online adaptation; the parameters are fitted to the data they were fitted on. And a symmetric switching matrix does not by itself tell you that vehicle motion in a dataset favors the constant-velocity model, even though it is tempting to read it that way.

View full-size image ↗

Bridging a missed detection is often cheaper than detecting it.

What we take from this

Latency budgets are a design input, not a measurement you take afterward. Once the real-time threshold is crossed, the remaining budget belongs to accuracy, and the architecture should be organized around that ordering.

A local receptive field is a cheap substitute for perfect calibration. When two sensors are aligned imperfectly, learning the correspondence in a neighbourhood beats assuming a pixel.

Reduce the dimensionality of the problem before optimizing the network. Pillarization is the enabling trick, not an afterthought.

Hybrid representations outperform pure ones here: learned appearance features, an explicit geometric projection, and an explicit state estimator, each doing what it is good at. The modularity also helps when something goes wrong, because each component has a distinct role and a distinct failure signature.

Complementarity beats tuning. Two models that fail in different regimes are better than one model that is mediocre in both.

And when the priors genuinely cannot be guessed, search for them. The admission that empirical IMM values may not transfer is the most useful methodological statement in the paper.

What it does not do

The honest summary is that the system is competitive in accuracy rather than leading. It improves substantially on detection speed and remains behind the best LiDAR-only methods on accuracy, which is the trade we chose.

The IMM-UKF has trouble with objects whose motion direction changes little. A filter that switches between hypotheses needs observable maneuver to be worth switching for.

Detection range is bounded by the camera field of view, which is inherent to this fusion design. Sparse distant objects fail, as the three-to-five-point example shows. The tracker is a separate stage from the detector rather than jointly optimized with it, which is the direction we point to for future work, along with improving accuracy without giving up the frame rate.

Closing

What we built integrates 3D detection and 3D multi-object tracking into a single pipeline measured against the real-time requirements of highway driving: a detector that produces oriented boxes efficiently, and a tracker that follows objects through uncertain motion using association, an interacting multiple model filter, and tuned parameters. The claim is not that it is the most accurate system available. It is that the accuracy is real at a frequency the fusion alternatives cannot reach, and that on a highway, being two metres behind the world is its own kind of inaccuracy.

Computer Vision | Chang Nie

Robust Multi-Model Fitting through Learning Neighbor Regions

Combinatorial explosion in multi-model fitting: Why blind random sampling fails

The four things that actually go wrong

What competing methods leave unresolved

Learn at the decisions, condition on the points

Screening before solving

Judging a hypothesis by its neighbourhood

Deciding how many models there are

How the two modules train each other

What we take from this

What it assumes

Why keep the two-step pipeline

DiffSAC: Diffusion-guided Sampling for Consensus-based Robust Estimation

The expensive part is not the sampling

Critical Analysis of Prior Heuristics

Generating Candidate Sets Instead of Guessing

Conditional Diffusion Process and Mathematical Formulation

Permutation-Invariant Transformer Denoiser Architecture

Fast Inference: Batch-Parallel Diffusion and Consensus Verification

The Decisive Ablation: Why Iterative Diffusion Is Irreplaceable

Comprehensive Benchmark Evaluation and System Boundaries

Engineering Philosophy: Reclaiming Auditability in Learned Geometry

RLSAC: Reinforcement Learning Enhanced Sample Consensus for End-to-End Robust Estimation

The sampler is the part nobody trains

The sample is an action

What the policy gets to see

One episode, one random start

Watching it work on a line

From points to image correspondences

What we take from this

Where it stays limited

The larger point

3D Object Detection and Tracking Based on Lidar-Camera Fusion and IMM-UKF Algorithm Towards Highway Driving

Perception that is blind for two metres at a time

What the fusion families cost

What makes it hard

Attach a neighbourhood, not a pixel

Collapsing the height axis to get the frame rate

Two motion models, and a search for the priors

What we take from this

What it does not do

Closing