Chang Nie

Teach and Grow: An Agent-Centered Architecture for General Robot Learning

Mon, 17 Aug 2026 23:45:21 +0000

A Few Demonstrations Become Skills That Stay

Teach-and-Grow Learning (TGL) establishes an AI Agent and GPT-6 centered robotic manipulation architecture for general-purpose physical autonomy. Operating under the principle that pretrained foundation model weights remain entirely frozen, the framework places GPT-6 at the center of the embodied reasoning loop. The surrounding AI Agent interprets multi-modal spatial cues, decomposes complex long-horizon tasks, grounds high-level semantics into metric SE(3) geometries, and validates physical state transitions before accumulating verified skills into persistent dual-memory repositories.

Teach → Act and verify → Grow. Sparse demonstrations provide structural subgoal decompositions and causal dependencies; real-time physical feedback governs closed-loop execution and error recovery; and persistent, decoupled dual memory transforms the verified takeaways of preceding tasks into immediate springboards for subsequent unfamiliar missions, entirely eliminating the dependency on perpetual fine-tuning and weight optimization.

· ·

Where End-to-End Robot Learning Gets Expensive: The Retraining Tax

Contemporary research in deep imitation learning and end-to-end Vision-Language-Action (VLA) models relies primarily on gradient descent across high-dimensional parameter spaces $\theta \in \mathbb{R}^D$. When deploying these monolithic models into real-world, unstructured operational settings, however, a profound structural dilemma arises:

Whenever a robot encounters a novel container geometry, an uncalibrated camera perspective, highly specular surface finishes, or minor grasp slips, the actual metric variation may be minor. Yet, within a monolithic parameter-space paradigm, remediating this isolated deficiency typically necessitates collecting dozens or hundreds of additional expert trajectories, followed by computationally intensive joint multi-task optimization across the entire legacy training corpus. Worse still, because deep neural representations are heavily entangled across latent feature dimensions, adjusting parameter weights to accommodate new tasks frequently provokes unconstrained representation drift, causing previously mastered competencies to degrade unpredictably. We formalize this recurring computational, data collection, and verification overhead as the Retraining Tax.

Physical interaction data possesses a fundamentally distinct economic cost structure compared to internet-scale textual tokens or static web images. Physical contact cannot be synthetically generated losslessly via naive auto-regressive sampling; grasp stability must be empirically tested within a 3D dynamical workspace; and state changes in support relationships and surface friction must be validated through physical sensors. Geometric poses, background clutter, frictional coefficients, and manipulator nonlinear dynamics interact continuously. Covering the independent marginal distributions of isolated physical factors is fundamentally insufficient to cover their high-dimensional joint combinatorial distributions.

TGL addresses this foundational bottleneck: Can we decouple the acquisition of new robotic tasks from parameter-space gradient updates? Can we preserve physically validated manipulation experiences as explicit, inspectable objects endowed with verifiable geometric applicability bounds, deterministic preconditions/postconditions, and modular error-recovery rules? Under this paradigm, robotic foundation models centered on GPT-6 within an AI Agent serve as zero-shot semantic reasoning and geometric perception backbones, while TGL restructures how task knowledge is stored, audited, and retrieved within the robot system.

View full-size image ↗

Figure 1. Policy optimization and explicit skill acquisition, illustrated through the same unfamiliar manipulation task.

From Planning AI Agents to GPT-6 Centered Robotic Manipulation

This research originated from investigating how tool-using AI Agent architectures could directly actuate robot manipulators to accomplish open-ended manipulation tasks. Placing GPT-6 at the core of the embodied agent provides unprecedented physical causality modeling and world common-sense reasoning. Paired with open-vocabulary perception, 6D pose estimation, and kinematic motion planning tools, a GPT-6 centered AI Agent can assemble feasible action sequences through exploratory trial-and-error even in the complete absence of prior demonstrations.

However, relying exclusively on instantaneous agent deliberation exposes severe operational limitations: once an episode terminates and the object is placed, the system retains no persistent, deterministic knowledge. When confronted with an identical or closely related task in subsequent trials, the agent must invoke slow, high-latency language model inference anew, searching through vast combinatorial action spaces from scratch. Furthermore, physical hurdles encountered during previous runs (such as slippery rims on thin glassware or localized mechanical binding in drawer rails) are discarded alongside the context window.

TGL integrates GPT-6 into the AI Agent cognitive center, retaining the agile observe-reason-act-verify loop while coupling it to an evolving lifelong memory architecture. Successful task executions are consolidated into formal Skill Blocks governed by rigorous execution contracts, while unexpected execution faults, root-cause diagnostics, and successful recovery sequences are archived as contextual exemplars within Experience Memory. In downstream tasks, the robot prioritizes retrieving verified, high-confidence skills, invoking full agent deliberation only when genuine out-of-distribution variations occur.

Within this framework, sparse demonstrations (human teleoperation logs, simulated trajectories, or instructional procedural text) function as an accelerator rather than an immutable prerequisite. Demonstrations supply a macroscopic temporal sequence of subgoals, confining exploratory search to critical geometric transitions and slashing trial-and-error overhead.

Formal Modeling: Executable Capabilities as Explicit Skill Blocks

To isolate invariant behavioral strategies from transient physical variations specific to a single environment, TGL introduces the formal abstraction of the Skill Block. Consider the task of picking up a ceramic bowl and placing it onto an adjacent tray. Across diverse demonstrations, the manipulator’s Cartesian trajectories, joint velocities, and approach angles vary widely. However, the underlying causal structure remains invariant: acquire the designated bowl, establish a stable grasp, transfer toward the target placement relation, and compliantly release contact.

Mathematically, we formalize a Skill Block $S_k$ as a 6-tuple:

$$S_k = \langle \mathcal{G}_k, \mathcal{P}_{\text{pre}}, \mathcal{P}_{\text{post}}, \Pi_{\text{exec}}, \mathcal{V}_{\text{eval}}, \mathcal{R}_{\text{recov}} \rangle$$

Where:

$\mathcal{G}_k$ denotes the semantic and geometric subgoal targeted by the skill;
$\mathcal{P}_{\text{pre}}$ represents the explicit precondition manifold required for skill triggering (e.g., target object kinematically reachable within the workspace envelope, gripper fully open, approach vector free of static occlusions);
$\mathcal{P}_{\text{post}}$ defines the postcondition assertions that must hold upon completion (e.g., vertical object centroid displacement exceeding a threshold, gripper aperture matching object width);
$\Pi_{\text{exec}}$ denotes the family of compatible parameterized low-level executors (e.g., Contact-GraspNet antipodal samplers, Cartesian straight-line admittance controllers, or specialized local visuomotor policies);
$\mathcal{V}_{\text{eval}}$ represents the multi-modal physical verification operator responsible for outputting a deterministic success evaluation;
$\mathcal{R}_{\text{recov}}$ denotes the bounded recovery policy branch specifying local retreat and compensation trajectories upon anomalous execution.

Here, training-free has a precise mathematical meaning: acquiring, validating, and consolidating an incoming task involves zero gradient backpropagation or weight optimization across underlying neural networks. Upstream foundation models and downstream geometric tools remain frozen; capability growth is realized strictly through explicit topological expansion and parameter synthesis within the skill library and memory network.

View full-size image ↗

Figure 2. One task moves through teaching, composition, execution, verification, and persistent storage. The resulting library and memory become the starting point for the next task.

Subgoal Reasoning on Top, Metric Geometry and Control Underneath

TGL enforces a disciplined division of responsibilities across hierarchical tiers, ensuring high-level semantic generalization while maintaining low-level physical precision:

The AI Agent segments incoming multimodal demonstrations along boundaries of physical state transitions. Rather than naively memorizing time-series joint waypoints, the system identifies structural shifts in spatial contact topology (e.g., free-space transit $\to$ pre-contact alignment $\to$ stable dynamic constraint $\to$ release and decoupling). Although kinematic speeds and approach vectors vary, shared topological state transitions define the structural skeleton of the candidate Skill Block.

2. Scene Grounding and Metric Geometric Mapping

When the agent schedules a Skill Block, its abstract causal parameters are mapped to the local 3D workspace:

Perception and Segmentation: Onboard RGB-D sensors capture point clouds and color imagery, invoking open-vocabulary detectors and segmenters to isolate target geometric meshes;
Grasp Affordance Synthesis: Contact-GraspNet densely evaluates force-closure grasp candidates over the 3D surface mesh, selecting antipodal normals that maximize contact stability;
Collision-Free Motion Planning: MPLib and OMPL motion planners generate kinematically continuous trajectories that navigate around obstacles represented in a local Euclidean Signed Distance Field (ESDF).

3. Sensor-Driven Physical Verification Contracts

Following trajectory execution, the system forbids blind open-loop transitions. Instead, the multi-modal verification operator $\mathcal{V}_{\text{eval}}$ audits execution states against empirical ground truth:

For GraspObject, reaching commanded joint waypoints is insufficient. The controller concurrently evaluates gripper motor current feedback, infrared optical beam interruptions between gripper fingers, and wrist 6-axis force-torque wrench differentials. Only when force signatures confirm the gravitational payload of the target object does the postcondition pass;
If verification registers an unladen gripper, an incipient slip is flagged, dynamically routing execution to the bounded recovery branch $\mathcal{R}_{\text{recov}}$ to adjust approach yaw and re-attempt grasp acquisition locally, preventing error compounding.

View full-size image ↗

Figure 3. The contract connects a semantic subgoal to grounded execution and observable evidence. Strategy is carried forward; geometry and control are recomputed.

Dual Memory Architecture: Skill Library and Experience Memory

To balance generic capability retrieval with specialized domain adaptation, TGL structures knowledge across two distinct, complementary repositories:

1. Skill Library ($\mathcal{L}_{\text{skill}}$)

The Skill Library maintains a curated catalog of validated atomic skill templates. Each entry encapsulates formal subgoal specifications, geometric precondition checkers, scene grounding rules, and compatible motion planner interfaces. The library prioritizes abstraction and domain independence, allowing skills to be instantiated across diverse objects and functional workspaces.

2. Experience Memory ($\mathcal{M}_{\text{exp}}$)

Experience Memory functions as an episodic, case-based repository that logs contextual metadata and diagnostic histories from real-world execution episodes:

Catalogs ambient lighting parameters, surface material properties, grasp quality scores, and empirical execution outcomes;
Archives failure traces alongside successful recovery trajectories (e.g., recording that lateral pinch grasps on polished metallic cylinders induced rotational slip, whereas a vertical top-down pinch established stable force closure);
When the robot encounters analogous operational contexts in future missions, Experience Memory injects contextual priors into the planning prompt, biasing the agent away from historically vulnerable approaches and selecting pre-validated parameter sets.

This decoupled architecture provides unmatched engineering auditability: practitioners can inspect individual skill boundaries, refine recovery routines, or deprecate faulty tool configurations without initiating uncontrollable global model retraining.

Lifelong Learning and Analytical Cost Scaling

Conventional reinforcement learning and imitation learning face compounding difficulties when scaling across broad task catalogs due to parameter interference. TGL introduces the Effective Reusable Experience hypothesis: as a robot operates across extended horizons, accumulated skill blocks and contextual memory nodes progressively cover the predominant geometric and contact interactions within its operational domain.

View full-size image ↗

Figure 4. Conceptual predictions for future-task error and teaching demand. The curves are schematic.

Analytically, we compare the marginal computational cost of traditional parameter-space updates against TGL’s modular expansion:

Global Parameter Update Paradigm: Adding the $N$-th task requires joint optimization across all preceding $N$ tasks to prevent catastrophic forgetting. The required training samples and compute scale super-linearly as $\mathcal{O}(N^\alpha)$ ($\alpha > 1$);
TGL Modular Expansion Paradigm: Synthesizing and validating an incoming Skill Block requires verifying only local execution contracts against the specific task scene. Marginal acquisition overhead remains bounded at $\mathcal{O}(1)$.

Provided that scene grounding, verification checks, and skill retrieval maintain lightweight computational profiles, explicit skill accumulation delivers dramatic long-term efficiency advantages over perpetual fine-tuning.

View full-size image ↗

Figure 5. Analytic cost regimes under different assumptions about coverage and local compatibility.

Slow Teacher and Fast Student: Dual-System Synthesis

In real-world industrial and service settings, relying entirely on agent deliberation for every single motion introduces latencies of several hundred milliseconds to seconds, which is ill-suited for high-speed dynamic execution.

To resolve this latency-generality trade-off, TGL natively supports a Slow Teacher, Fast Student dual-system execution framework:

Slow Teacher (System 2: TGL Deliberative Loop): The AI Agent functions as a deliberative System 2, performing long-horizon reasoning, tool composition, and fault recovery in unfamiliar tasks. In doing so, it filters out execution failures and buffers clean, physically verified expert trajectories into an execution repository;
Fast Student (System 1: Distilled Reactive Policy): Verified trajectories are used to asynchronously train high-frequency reactive policies (such as a 50 Hz Diffusion Policy or Action Chunking Transformer). These compact models execute fluid, low-latency primitives under nominal operational conditions;
Seamless Anomaly Hand-off: If the reactive policy experiences an out-of-distribution observation or fails a physical verification check, low-level monitors instantly trigger an interrupt, returning control to System 2 to perform root-cause diagnosis, execute recovery, and update Experience Memory.

Empirical Benchmark Validation and Insights

We conducted rigorous empirical evaluations across the standard robotic manipulation benchmark LIBERO as well as its out-of-distribution counterpart, LIBERO-Plus. Experiments focused on key evaluation axes:

Few-Shot Skill Acquisition: Measuring TGL’s capacity to extract invariant subgoal structures from 1 to 3 demonstrations and deploy them successfully across novel spatial layouts and visual textures;
Mitigation of Retraining Overhead: Quantifying cumulative computational overhead and historical task retention when acquiring 10 consecutive tasks compared against parameter fine-tuning baselines;
Fault Tolerance via Physical Contracts: Evaluating task success rates under induced physical disturbances (e.g., external perturbations during grasping, visual feature occlusions) with and without bounded recovery routines.

Key empirical findings include:

TGL achieved substantially higher out-of-distribution generalization compared to few-shot behavioral cloning baselines while requiring minimal demonstration data;
Because base model weights remain frozen, performance across historical tasks exhibited zero degradation, completely eliminating catastrophic forgetting;
Explicit verification contracts and localized recovery routines resolved the majority of operational disturbances autonomously, sustaining reliable long-horizon execution.

Summary and Engineering Takeaways

TGL demonstrates that general-purpose robotic manipulation does not strictly mandate monolithic end-to-end neural network optimization for every newly encountered task.

By unifying foundation model semantic reasoning, metric geometric toolsets, deterministic impedance control, and structured dual-memory repositories, a robot can acquire, adapt, and retain complex capabilities without weight perturbation. Preserving knowledge as explicit, inspectable, and reusable objects offers an economically scalable and physically grounded blueprint for lifelong robot learning.

Voxelized 3D Gaussian Representation for Dense Visual SLAM on Embedded Vision System

Wed, 13 May 2026 00:00:00 +0000

Dense Mapping and Storage Constraints

Sparse feature-based SLAM is accurate and its maps are too thin to be useful. Navigation, path planning and obstacle avoidance need surface information, not a cloud of keypoints, which is why dense mapping has been pursued for as long as it has.

The dense alternatives each carry a cost. Classical dense fusion systems such as DTAM, Kintinuous and ElasticFusion are memory-hungry and slow, and their accuracy suffers for it. Neural radiance field SLAM produces better geometry but carries an implicit decoder, which means rendering is a network evaluation rather than a rasterization. 3D Gaussian splatting looked like the resolution: real-time rendering with high fidelity, no decoder, no volumetric integration.

Then you try to deploy it. A Gaussian-splatting SLAM system typically needs more than 500 MB to represent a small room-sized scene. That number is the difference between a demonstration and something that runs on a robot, and it is where this work starts.

The redundancy is made by SLAM, not by Gaussians

The obvious conclusion would be that Gaussian primitives are simply expensive, and that the fix is compression. We do not think that is right, and the diagnosis is the part of this paper we would most want read.

We looked at the Gaussian ellipsoids that these systems create and found that their geometric similarity is far higher than in the original 3D Gaussian splatting. Measuring similarity between normalized, mean-free Gaussians reduces to comparing their covariance, which isolates shape and size from position, and the distribution we see in SLAM systems is a tall narrow spike compared with the broad low curve of the offline method. These systems are storing near-duplicates of the same ellipsoid, over and over.

The cause is not the primitive. It is the optimization schedule, and there are two structural differences that explain it.

Offline 3D Gaussian splatting receives every image at once. It optimizes across all of them jointly, and when it selects keyframes it samples randomly from the entire set. A SLAM system receives frames in timestamp order. It waits for each frame’s training to finish before moving to the next, and it only ever optimizes selected keyframes. Same primitive, sequential and locally scoped optimization, and a very different resulting distribution. The online regime has, to a certain extent, exacerbated the geometric similarity.

That distinction matters because it decides which existing tools are usable. The offline compression literature, including compact Gaussian representations, lightweight Gaussian pruning and self-organizing Gaussian grids, all assume you have every image, every pose and the corresponding point cloud available from the start. SLAM systems are incrementally optimized. Every one of those methods is disqualified by construction, however well it works offline.

View full-size image ↗

Anchors carry structure; per-view decoding generates Gaussian attributes.

Why online compaction is awkward

Three things make the problem harder than compressing a finished map.

Offline 3D Gaussian splatting seeds its Gaussians from a complete posed point cloud. In SLAM that cloud only exists for where the camera has already been, so the decision of where to add representational capacity has to be made from a partial map, online, without knowing what is coming.

Deletion has to be causal. You cannot consult the full training set to decide whether a Gaussian is still needed, because the future views have not arrived.

And opacity alone is the wrong criterion for removal. A Gaussian can carry low opacity and still be critical across multiple views, or occupy a useful volume while contributing little colour. Volume and opacity together are closer to what we want.

There is also a failure mode specific to learning the decision. Our mask loss pushes mask values down, which is what a sparsity prior does, and if you leave it running without interruption the optimization will eventually eliminate every Gaussian in the map. A one-directional regularizer with no reset destroys the thing it is regularizing.

Finally, compression here is not a post-processing step. In offline compression you quantize a finished artifact. Here the codebooks are learned jointly with attributes that are still being optimized, so the quantizer is chasing a moving target.

Fewer anchors, then fewer bits

Two layers of compaction, in that order.

The first is structural. Instead of storing Gaussians, we store anchors. An anchor is a voxel centre together with a small learned feature, a scale, and a set of offsets. The Gaussian centres are generated as the voxel centre plus each offset scaled by that anchor’s scale, and the remaining attributes, opacity, rotation, scale and colour, are decoded from the anchor feature together with the current viewing distance and direction. Structure carries the representation rather than individual primitives, and because decoding is view-conditional, detail is generated per view rather than stored per Gaussian.

The second is parametric, and it applies once the anchor count is under control. The offsets and scales are the geometry every anchor has to carry, so those are where the bits go. We compress them with cascaded residual vector quantization: the first codebook approximates the vector, and each subsequent stage quantizes only the residual the earlier stages left behind, with indices chosen by nearest-neighbour search against the running residual.

The analogy comes from neural audio coding, and it is a good one. The first codebook gives a rough description of the number. Each later codebook describes only what is left over. The codebooks are shared across the whole scene the way a palette is shared across an image. We initialize them with K-means.

Two tradeoffs are worth stating. More stages mean more residual levels to store and slower inference for gains that are only marginal. A larger codebook leaves many indices unused while still costing memory and training time, and a smaller one costs accuracy. The paper’s configuration uses six residual stages over a codebook of size 64, and the abbreviation is worth spelling out for readers who have met particle filters: residual vector quantization and particle swarm optimization have nothing to do with each other, and no particle filter appears anywhere in this work.

View full-size image ↗

Each stage quantizes only what the previous stages left behind.

Growing where the optimizer is still struggling

With no structure-from-motion cloud to seed from, the question of where to put capacity has to be answered online. The signal we use is the gradient.

A voxel whose constituent Gaussians keep producing large averaged gradients over a run of iterations is a place the optimizer is still fighting, which is a reasonable indication that the current representation cannot express what the data wants. If that averaged gradient exceeds a threshold and no anchor exists at that voxel centre yet, we plant one. Candidates are eliminated randomly to keep growth from exploding, and trivial anchors get removed by accumulating the opacity of their Gaussians.

Three resolutions run in parallel, with voxel size shrinking by a factor of four at each level and the gradient threshold doubling. Coarse voxels absorb bulk geometry such as walls and floors. Fine voxels handle detail such as a chair leg. A single voxel size would force one choice to serve both, which is why the ladder exists.

Initialization is separate: at map start, points are sampled where accumulated opacity is low or where the rendered depth disagrees with the measurement.

The mask is a dimmer, not a switch

Inside a sliding window, each Gaussian carries a learnable mask parameter, which a stop-gradient straight-through form turns into a binary mask. That mask multiplies the Gaussian’s scale and opacity before rendering, so a masked primitive fades out of the image rather than being deleted from the data structure. Removal happens later, at the window boundary, where we use hit frequency together with the learned mask.

The window is short-term memory with a defined composition: the current frame, the keyframe it overlaps most, and the two earlier keyframes with the highest frustum overlap with the current depth point cloud. Everything outside the current view is frozen by frustum culling, which preserves past geometry and reduces the number of parameters being optimized. Masking runs on keyframes only.

The reset is the detail that makes the mechanism safe. At the window boundary, once removals are applied, both the gradients and the masks are cleared so the system can relearn. Without that reset, the sparsity pressure described earlier runs unopposed and the map erases itself.

Window size is a genuine tradeoff rather than a hyperparameter to tune casually. Too large a window masks away Gaussians that were doing useful work, which costs accuracy. Too small a window costs memory and optimization quality.

View full-size image ↗

Masking modulates scale and opacity; removal happens at the window boundary, after which the masks and gradients are reset.

Tracking depends on the map you just compacted

Three signals drive tracking, and all of them come out of the same alpha-compositing pass. Colour, depth, and a silhouette channel that accumulates only the visibility weights. Because they share the rasterization, the silhouette is nearly free.

Tracking minimizes squared colour error plus squared depth error over rays with valid depth, gated by the silhouette: only pixels where the renderer is confident it is actually covering something are trusted. An ICP term ties Gaussian centres to their nearest points across frames, on the reasoning that Gaussians are a point cloud and classical geometric association therefore applies to them. Pose is initialized by constant-velocity forward projection, which the ablation calls vital.

The ablations here are the clearest argument that these components are load-bearing. With only the depth term, tracking fails outright. With only colour, tracking works at substantially worse error. Without the silhouette gate, tracking fails outright. A renderer-derived trust region turns out to be doing real work, and a photometric pipeline apparently needs both a geometry term and a region it can believe.

Bundle adjustment runs at two scales. Local adjustment refines the recent window, and a global stage samples rays from a deliberately larger keyframe database to refine map and poses together, with an additional structural similarity term. The purpose is to bound accumulated drift over long trajectories and large scenes without touching every historical frame.

Evidence, and the shape of it

The system is evaluated on Replica, ScanNet and TUM RGB-D, on an embedded module and a laptop-class GPU, and on a multisensor mobile-robot dataset we collected ourselves with LiDAR, an inertial unit and a depth camera, covering scenes over a thousand square metres with high-quality trajectory ground truth. The dataset exists to answer a question the standard benchmarks cannot: whether the representation holds up when the map is being built on a moving robot rather than on a desk.

The most useful evaluation is not a table. We inserted the compact representation into SplaTAM, MonoGS and Gaussian-SLAM and re-ran them. The same direction of improvement appeared in each. That is evidence about the diagnosis rather than about our system, since a representation that helps three different systems is addressing something those systems share.

View full-size image ↗

Compactness is what makes on-robot deployment possible.

What we take from this

Diagnose the representation you inherited from the training regime before you build a compressor. The interesting move here was not designing a better quantizer. It was noticing that sequential online optimization produces a different and more redundant distribution than joint optimization, and measuring it before writing any code.

Related to that, any module that needs the entire dataset is disqualified by construction in an incremental system, no matter how well it performs offline. That single constraint eliminates most of the available literature.

Compact in two layers, structural and then parametric. Quantizing alone leaves millions of primitives to encode. Masking alone leaves a large per-primitive payload.

Gradients are a capacity-allocation signal, not just an optimization artifact, and using them that way turns where should the map be detailed into a decision the system makes for itself.

And any learned pruning pressure that only pushes in one direction needs a reset. That pattern generalizes well beyond this paper.

What it does not handle

Fast motion produces motion blur in both the colour and depth images, and tracking depends on exactly those losses, so rapid movement degrades it. Scenes with many dynamic objects produce artifacts and drift, and the system is still limited in dealing with highly dynamic environments.

Loop closure and relocalization are not implemented, which bounds how well drift can be corrected on large loops. The system is RGB-D only, so there is no monocular or stereo path. And we owe the reader a clearer account of the reported memory figure: it is checkpoint size rather than peak memory footprint, and those are different numbers.

Why embedded matters

The paper’s claim is not that a compact map is an acceptable compromise. It is that a compact map is still a good map, and the same quality is available at a fraction of the storage. That claim only has teeth in a setting where the storage is a real constraint, which is why the embedded experiments are the ones we care about. A representation that fits on the robot is the one that gets used.

MID: A Self-Supervised Multimodal Iterative Denoising Framework

Thu, 30 Apr 2026 00:00:00 +0000

The clean version often does not exist

Good data is the precondition for everything downstream, and acquisition noise is nearly universal: sensor limitations, transmission errors, environmental interference, and the stochastic nature of physical measurement itself. The usual response is to denoise first.

Two practical facts make that harder than it sounds, and both shaped this paper.

The first is that the noise that actually matters is rarely Gaussian. Poisson noise in low-light imaging, compression artifacts, and what we would call structured noise, such as a set of line segments that do not correspond to any real structure in the scene, all violate the assumptions that classical filters are built on. The second is more fundamental: for many of the domains we care about, the clean target simply does not exist. In clinical imaging and remote sensing there is frequently no noise-free reference to compare against, and no way to acquire one. The MRI experiments in the paper are scored with proxy measures precisely because clean ground truth is unavailable for those datasets.

So the training signal has to come from somewhere other than a paired clean sample. That constraint is the whole problem.

Why the existing answers are conditional

Classical model-based denoising, from filtering to transform-domain thresholding to non-local means, is tailored to specific noise models and data types. Its effectiveness falls away as soon as the noise statistics deviate from the assumption it was built on.

Supervised deep denoising sidesteps the modelling assumption but needs large paired datasets of clean and noisy samples, which are costly or infeasible to collect in exactly the domains where denoising matters most.

Self-supervised blind-spot methods were the natural answer, and the criticism we make of them is specific rather than general. They avoid pairing, and they carry hidden modality assumptions. The tricks that make a blind-spot network work, such as masking pixels and predicting them from neighbours, rely on spatial correlation in images. Point clouds are unordered, and amino acid sequences are discrete and unstructured. A method that assumes nothing about the data can travel to those settings. One that quietly assumes spatial smoothness cannot, and the observable symptoms are oversmoothing and loss of fine detail.

Domain-specific specialists have their own conditionalities. Noise2Noise can oversmooth fine anatomical structures. Patch2Self exploits redundancy across volumes but requires training a separate model per volume, which risks spatial inconsistency between them. DDM2 is sensitive to the initial noise level and to modality-specific tuning. In surface electromyography, high-pass filtering removes the ECG interference and removes important low-frequency components of the signal along with it, because the two overlap in frequency; template subtraction assumes a rigid distributional form; and neural approaches can be unstable to train. For protein sequence curation, the standard greedy selection on average Hamming distance rests on a single criterion that does not capture sequence quality more broadly.

We summarize the positioning as a capability matrix rather than a benchmark table. The question is which methods are self-supervised and work on images, signals and point sets at once. Most of the field manages two of the three.

What makes it hard

The inverse of a nonlinear corruption has no usable closed form. Reversing Poisson noise, compression, or a set-structured nuisance directly is difficult, and our own ablation shows what happens if you try: a single-pass network leaves the data essentially unchanged. Networks struggle to learn a large jump from noisy to clean in one step. That result is not an implementation failure. It is the reason the method is shaped the way it is.

You also do not know how corrupted the input is. Nothing about a noisy sample announces its position on a corruption trajectory, so there is no way to decide how much to remove or how many steps to take.

Self-supervision has to be manufactured rather than found. Without pairs, the only available labels are ones you create yourself, and they have to serve two purposes: how far along the corruption the input sits, and what the next increment looks like.

And sometimes the noise is not a value at all. An incorrect correspondence, an invalid line segment, or a redundant sequence is not a magnitude added to anything. It is a decision about an element. Before any of that can be subtracted, the nuisance has to be recast as a per-element property, which is why those tasks use a binary classification objective rather than a regression on residuals.

Unordered and discrete data break the usual priors outright. A point cloud has no ordering and a multiple sequence alignment is a set of discrete strings, so weight sharing over a spatial grid and convolution are simply unavailable.

There is also a tradeoff baked into the formulation. The first-order approximation underneath our method only holds if the increments are small. Small increments buy linearity, and then cost you a long inference loop.

Restore by adding noise

The idea we started from is counterintuitive enough to state plainly. To learn how to remove noise, you first add more of it.

A noisy observation is not a degraded version of a clean sample sitting off to one side. It is one state on a corruption trajectory, and you can continue that trajectory yourself. Take the noisy input as the starting point, apply the corruption process again and again, and you now have a labelled ladder: at every rung you know exactly how much noise separates it from the rung below, because you put it there.

The move that makes this usable is a local linearization. Writing the corruption as a general operator and expanding to first order,

$$ s_t \approx s_{t-1} + \Delta s_{t-1}. $$

The step from one rung to the next is treated as additive, which means its inverse is a subtraction. The residual the noise network is asked to predict is nothing more than the difference between two consecutive states that we generated. This is an approximation, and we say so: it implicitly assumes the Jacobian is close to identity, or that its effect is absorbed into the learned mapping.

Everything else follows from that. The reverse step becomes tractable, and the training signal is free.

Two networks, two different questions

The framework asks two separate questions, and keeping them separate is what makes one recipe portable across domains.

The first is: where am I? A step prediction network regresses the input’s position on the trajectory, with the step normalized and a mean-squared objective. This is necessary before any amount of subtraction can be chosen, and it is where the method differs most from diffusion. Diffusion trains against a noise prediction derived from a fixed forward schedule. Here the step is estimated from the data itself, without assuming a fixed forward process.

The second question is: what should be removed? The noise prediction network takes the current state and the estimated step and predicts the increment. The two networks are trained jointly, and the decoupling is deliberate: one decides how far to walk back, the other decides what to take off at each step.

At inference the loop is a few lines. Estimate the current step, then subtract predicted increments until the step count reaches zero. A lightly corrupted input takes fewer steps than a heavily corrupted one, so the iteration count adapts to the data without a schedule.

The reason for iterating at all, rather than removing everything at once, is that many small corrections preserve detail better than one large one. Oversmoothing is what happens when a single pass has to guess the whole distance.

View full-size image ↗

A step predictor and a noise predictor are trained jointly and used in sequence at inference.

View full-size image ↗

The reverse self-supervised restoration loop walks a corrupted input back down the trajectory one increment at a time.

Denoising is a way of framing a problem

The most interesting part of this work, for us, is not the image results. It is what happens when the same formulation is pointed at problems that nobody would call denoising.

Image correspondences become a four-dimensional point cloud, where a match is a point with two image coordinates in each view. Wrong matches are points that do not belong to the structure, and the network learns the broader features of incorrect correspondences rather than relying on a match score alone. Line segments used for vanishing point estimation become an unordered point cloud, and the model is trained by iteratively adding random segments as noise. Multiple sequence alignments are handled by treating redundant sequences with low average Hamming distance as noise. In that last case, the noise consists of real amino acid sequences, which is a fair demonstration that the framework is evaluating data quality rather than a synthetic artifact.

What this shows is that denoising is a task framing rather than a data type. Once the nuisance is expressed as something removable, outlier rejection in geometric estimation, correspondence pruning, line filtering and sequence subsampling turn out to be the same problem wearing different clothes.

Two design details follow from the same insight. Multiplicative noise can be made additive in the log domain, which the paper notes and uses. And for point-set tasks the noise objective is a per-element binary cross-entropy, not an additional penalty term on top of a regression. The network is not asked how much to subtract from a wrong correspondence. It is asked whether that element should be there at all.

View full-size image ↗

The same formulation that removes image noise also identifies incorrect matches.

How it is built

For images and medical volumes, a CNN backbone handles the step predictor with fully connected layers on top, and a CNN encoder-decoder predicts the increments. For point sets and one-dimensional signals, including correspondences, line segments, amino acid sequences and surface EMG, both networks use fully connected layers with a Transformer encoder-decoder. The reason is the one given above: convolutions assume a spatial ordering that unordered sets do not have, so the architecture changes with the data rather than the other way around.

None of the domains get a bespoke formulation. That was the point of building it this way. One method, applied to image speckle, MRI noise, muscle-signal contamination, geometric outliers, wrong correspondences, invalid line segments, and redundant protein sequences, with only the backbone changing.

View full-size image ↗

The nuisance here consists of real sequences rather than synthetic artifacts.

What we take from this

Supervision can be manufactured by continuing a corruption you can simulate, rather than by obtaining the clean endpoint. That single substitution is what makes the framework applicable where paired data does not exist.

Local linearization plus iteration converts an intractable global inverse into a sequence of tractable local ones, and the ablation is the evidence. When the single-pass model fails, the right response is not a larger model. It is decomposition.

Separating where am I from what should be removed is what makes one framework portable. That decoupling, more than any architectural choice, is the reusable part.

Self-supervision strategies carry hidden modality assumptions, and it is worth auditing them. A formulation that asserts nothing about the structure of the data will travel further than one that quietly requires smoothness.

And iteration is a detail-preservation mechanism, not just an optimization convenience.

Where it stops

Extreme noise is the stated failure mode. When the signal structure is almost entirely obscured, iterative subtraction has too little partial structure to work from, and generative priors are the direction we point to for future work.

The second cost is time. Iterating is inherently slower than a single-pass denoiser, which is a bottleneck for real-time use. Knowledge distillation and more efficient sampling are the remedies we suggest.

Two things are visible in the method that we did not fully characterize. The number of corruption steps and the magnitude of each increment are design choices rather than derived quantities, and we did not study the sensitivity to them. Errors in the estimated step also compound across the subtractions that follow, and that propagation deserves more attention than we gave it.

What kind of model this is

It is worth being explicit about one boundary, because it comes up in every conversation about this work. MID is not a generative model. Diffusion models minimize a noise prediction loss derived from a fixed forward schedule and can sample new data from the learned distribution. MID is an iterative, self-supervised noise subtractor that learns the structure of the noise present in its target domain and removes it. The output is a cleaner version of the input, not a new sample.

That distinction is what keeps the framework honest about what it can promise. It removes what it was trained to recognize. Anything else is a different problem.

Towards the Vision-Sound-Language-Action Paradigm: The HEAR Framework for Sound-Centric Manipulation

Tue, 17 Mar 2026 03:22:30 +0000

Robots that cannot hear the physics

Current vision-language-action models see, understand language, and move. What they do not do is listen. They operate in what amounts to a silent world, even though the robot’s own environment is full of sound that encodes physical state the cameras cannot recover: liquid starting to sputter as it boils, the sharp click of a collision, the difference in timbre between a full container and an empty one, the moment a material stops flexing and starts cracking.

Contact acoustics are the cheapest tactile sensor available. A microphone is a few dollars, it needs no contact, and it covers the whole workspace at once. The case for using it is not speculative. The problem is that nobody has built a policy that can.

The cue is gone before the policy is asked

The difficulty is not a modelling problem, and framing it that way is why earlier approaches have not solved it. It comes from the control architecture that makes large VLA models practical.

A large backbone cannot run at control frequency. So the standard design chunks actions: the model produces a short sequence of future commands, the robot executes them open-loop, and only then does the model get queried again. That gap is where the physics happens. A collision, a boil-over, a completed press: these are brief, non-repeatable acoustic events, roughly on the order of a hundred milliseconds, and they routinely fall entirely inside the interval where the policy is not looking. The average chunk at 30 Hz is about a second long.

The consequence is worth stating precisely, because it is not what people usually assume. A late reaction would be a delay. What actually happens is that the evidence is destroyed. The audio arrives, nothing is listening, and by the time the policy is queried again, the window contains only silence. If an event occupies samples that end before the next window opens, the window does not contain a weak version of that event. It contains nothing. We call the interval between decisions the Blind Execution Interval, and the word blind is doing real work.

There is a second failure that has nothing to do with timing. Look at a robot that has been waiting beside a running appliance. If the scene is quasi-static, the current image looks the same whether it has been waiting one second or ten. The expert demonstrations differ, because the right next action depends on elapsed time, but the observation does not. A snapshot policy has no way to tell those situations apart, so it either drifts or freezes. This is the aliasing problem that temporal context is supposed to solve, and without audio it is genuinely unobservable.

What earlier attempts leave out

Turning audio into text is the obvious first instinct, and it loses the thing we need. Speech recognition front ends discard non-speech cues and prosody entirely, and they compress timing into discrete tokens. A beep produces nothing at all. More subtly, an affirmative “Yes!” and a doubtful “Yes?” collapse into nearly identical token sequences, so the difference in pragmatic intent disappears while transcription accuracy looks perfect.

Rendering the waveform as an image and treating it as another camera view is the second instinct. It captures amplitude changes, and it obscures spectral structure: frequency contours and timbre, which is where material and contact information live. It is also sensitive to where the event happens to land inside the rendered window, so performance changes for reasons that have nothing to do with the physics.

A more recent line adds speech to VLA models and preserves prosody and speaker identity. That is real progress, and it targets explicit verbal commands rather than the ambient acoustic physics of manipulation, which is a different problem. The omni-modal systems are closer still, and the reservation is structural: they are built to listen before acting, using sound to establish context or infer intent around an action, rather than as low-level feedback during contact. None of them address the misalignment between transient events and chunked execution, which is the actual bottleneck.

Audio foundation models such as CLAP and ImageBind were trained on fixed-length clips, offline and independently. Deployed statelessly, they carry no temporal context across clips. Compact audio-native policies run fast and can handle contact, but their scale makes long-horizon, multi-stage behavior brittle. Processing audio natively turns out to be necessary and not sufficient.

The shortcut worth naming is faster replanning. Truncate the chunk, query more often, and the blind interval shrinks. We tried the variants. It helps less than it looks like it should, and it pays in motion quality: discontinuous, jerky trajectories that introduce localized action loops. It also cannot fix system latency, which no query rate eliminates. A cue lost to latency is lost regardless of how often you ask.

And the benchmarks are no help, because every standard simulator is silent. RLBench and ManiSkill test spatial reasoning and visual control in environments with no acoustic channel at all. Navigation benchmarks that do use sound let the agent move its base to improve its listening geometry, which is not an option for a manipulator that has to keep working while generating its own mechanical noise.

Separate the sensing rate from the decision rate

Our answer comes from noticing that this is a rate mismatch, not an information deficit. The sound is there. The policy is simply asked too rarely to catch it.

So instead of making the policy faster, we stop tying the two rates together. A persistent memory is updated continuously from the audio stream, packet by packet, independently of when the policy is queried. When a decision is finally requested, the memory already contains everything that arrived, including events that happened during blind intervals. The robot is not reacting to the last window. It is reacting to the last second of listening, with the recent history summarized.

That is the core move, and everything else in the architecture is bookkeeping around it. There is something that remembers, something that interprets, something that predicts what comes next acoustically, and something that acts. We named them accordingly.

Two constraints shaped the design. The first is that the audio path has to be causal in the strict sense: nothing may look at samples that have not arrived, and the tokenizer must produce codes for a segment from that segment alone. The second is that the memory horizon has to be a target, not a hope. It needs to cover the worst-case decision gap plus the measured system latency, otherwise the guarantee is only statistical.

The four modules

Historizer. Between two decision boundaries, the robot may receive a long run of consecutive audio packets, each about forty milliseconds. The Historizer consumes them as they arrive and updates a streaming stateful Transformer memory. The memory is read out at decision times and never otherwise. Its entire justification is the evidence-vanishing argument above: a window-only interface cannot represent something that happened just before the window opened. We also tried a causal GRU and exponential moving average pooling over audio features; the streaming attention’s extra capacity earned its place. We chose a short window with a persistent summary over a very long window, because long windows mix distinct interaction phases together and make the result sensitive to exactly where the window boundary falls.

Envisioner. Interpretation is hierarchical, because the two jobs have very different frequencies. A high-level model reads the multi-view images, the current acoustic window, the instruction, proprioception and the memory, and produces three things: a semantic latent, a key-value cache, and a short structured description of the current task stage, something like {"stage": "wait", "subgoal": "listen for beep"}. That stage text comes from the high-level model’s own text head. A smaller low-level model then reuses the cache with the current robot state to produce the control feature. The split keeps expensive semantic interpretation out of the fast path. The stage supervision matters more than it might appear: without it, tasks that require a quick behavioral shift suffer, because nothing forces the latent to name the phase it is in, and the phases that matter acoustically are often visually identical.

Advancer. This one predicts the audio that will arrive between the current decision and the next one, as discrete codes from the streaming tokenizer, trained by cross-entropy. It is used only during training and is dropped at deployment, so it costs nothing at inference time. Its purpose is to give the latent a concrete sense of elapsed time and process progression. Pouring, boiling, an alarm, a conversation: each has a different acoustic future, and predicting it forces the shared representation to encode where in the process the robot currently is. Without it, the failures look different from the Historizer’s. They happen late. The robot keeps pouring after the sound has already told it to stop. That asymmetry is how we knew the two modules were solving different problems rather than duplicating each other.

Realizer. Actions are generated by conditional flow matching from a Gaussian prior to the action chunk, integrated in a handful of Euler steps, conditioned on the control feature through adaptive normalization. We used flow matching rather than diffusion for the step count. The detail worth dwelling on is why smoothness matters here. Replacing the Realizer with direct regression produces jitter, and jitter injects mechanical self-noise into the microphone signal, which masks the brief cues the rest of the system is trying to hear. A perception failure caused by a control choice. The sharpest version of the claim is that a policy which listens while moving has to move in a way that makes listening possible. Sensing and control are not separable in this setting.

View full-size image ↗

A high-level model produces semantics and task stage; a low-level model produces the control feature.

View full-size image ↗

The Advancer is used only during training and is removed at deployment.

The three objectives are summed with weights that ramp in over the first part of fine-tuning, so the model learns imitation before it is asked to be temporally grounded.

Where the data came from, since it did not exist

No robot dataset contains synchronized audio. Open X-Embodiment has no microphones at all, real synchronized audio is noisy and hardware-specific and does not scale across labs, and no standard simulator models acoustics.

We built both pieces. OpenX-Sound retrofits audio onto existing episodes from video, preserving every original signal, and we audited a sample of it by hand for synchronization and semantic relevance. We are explicit that this is a bootstrapping mechanism for representation learning and not a substitute for physical data: synthesized audio can omit subtle contact cues, shift event timing, or hallucinate a sound with no physical source. Every result in the paper fine-tunes an OpenX-Sound checkpoint on real task-specific demonstrations with real audio. The synthesized audio is what makes a pretrained acoustic prior possible, not what makes the behavior work.

HEAR-Bench is the other half. Built on a dual-arm simulator, it runs an asynchronous audio thread that injects ambient noise and trigger clips at the exact sample index implied by the control cycle, and it scores success only if the geometric goal is reached after the cue. That timing rule is the whole contribution. Without it, pressing a button the instant the episode starts and pressing it when the alarm actually rings look identical.

Rigorous Real-Robot Experiment Design: Why These Tasks?

When validating the acoustic manipulation capabilities of end-to-end multimodal Vision-Language-Action (VLA) foundation models, we deliberately avoided arbitrary toy benchmarks. Instead, we designed each real-world robotic evaluation specifically targeting the fundamental physical blind spots of vision-only policies:

1. Moka Pot Coffee Brewing: A True Barista Robot vs. Superficial Button-Pushing

The vast majority of existing demonstrations claiming “robot coffee brewing” merely involve a mechanical arm approaching a commercial capsule or fully automated pod coffee machine, executing an open-loop press on a static plastic button, and idly waiting while internal integrated circuits run a blind timer. Such setups require zero perception of the underlying physical state—they are glorified mechanical switch activators, indifferent to fluid dynamics, thermal phase changes, or culinary progression.

In contrast, our setup places an authentic dual-chamber metal Italian moka pot on an active electric cooktop. The moka pot consists of fully opaque metal walls; the boiling pressurized water in the boiler, the compacted coffee puck in the funnel filter, and the ascending extraction column inside the collector chimney remain entirely concealed from view. External RGB cameras and infrared depth sensors cannot penetrate the metal shell to observe fluid levels or thermodynamic transitions.

A seasoned master barista relies neither on stopwatches nor visual inspection; extraction cutoff is governed purely by ear:

Silent Heating Phase: Water temperature climbs gradually with negligible acoustic emission beyond faint thermal conduction;
Smooth Boiling & Laminar Extraction Phase: Saturated steam pressure forces water smoothly through the coffee puck, producing a continuous, gentle, muffled bubbling sound;
Violent Sputtering Phase (Cavitation & Steam Eruption): As boiler water nears depletion, high-temperature steam mixes with residual liquid to produce a violent, unmistakable high-frequency sputtering and hissing eruption through the extraction chimney.

This acoustic inflection point is the decisive boundary in coffee extraction: removing the pot one second too early yields sour, under-extracted, watery coffee; leaving it on the heat two seconds too late causes dry-boil scorching, bitterness, and permanent thermal damage to the moka pot base.

HEAR functions as a genuine barista robot: powered by the end-to-end VLA foundation model, it continuously analyzes high-frequency spectral transitions in real time. The moment the distinctive micro-sputtering signature emerges, the policy issues action chunks with millisecond-level responsiveness, smoothly lifting the moka pot away from the heating burner.

View full-size image ↗

Contrasting superficial open-loop button pressing with HEAR’s acoustic monitoring of internal thermodynamic phase transitions inside an opaque metal moka pot.

2. Shaking a Bottle: Active Acoustic Probing

In everyday physical environments, numerous pill bottles, opaque jars, and enclosed canisters conceal their contents entirely. Passive vision fails fundamentally in such settings—no matter how many camera viewpoints or lighting angles are deployed, the exterior surface remains completely static.

To overcome this, the robot must exhibit Active Exploration: it must grasp and shake the container to inject mechanical energy, exciting internal collisions. By decoding the resonant damping, impact frequencies, and high-frequency collision timbres of the internal objects against the container walls, the model disambiguates the presence, material properties, and fill level of the enclosed items. In our real-robot demonstration, we deploy an occupied container filled with loose objects, showing how the robot actively shakes the bottle, decodes transient collision acoustics, and executes informed downstream classification.

3. Real Alarm Clock: Resolving Temporal Aliasing Under Quasi-Static Vision

An alarm clock in its silent standby state and active ringing state can appear visually indistinguishable. Lacking acoustic feedback, naive end-to-end policies succumb to shortcut learning, prematurely hitting the stop button at the very start of the episode. Our benchmark enforces that the stop button must be actuated only after the acoustic trigger chime sounds, directly probing whether the multimodal VLA accurately aligns actions with transient acoustic timing cues.

4. Pouring into Opaque Containers and Answering Phone Calls

Pouring water into an opaque cup represents the canonical scenario where sound is the sole observable state signal. Fluid levels are strictly occluded; the true decision variable is the continuous frequency drift rate of the pouring stream resonating within the shrinking acoustic cavity. Similarly, answering a phone call assesses multi-stage temporal state machines: the robot approaches upon hearing the ringing chime, remains stationary during the caller’s acoustic pauses, and docks the receiver only when the terminal dial tone registers.

Real-robot experiments

All demonstrations include their recorded audio. Select a video to play it.

Moka CoffeeThe robot listens for the acoustic sputtering phase inside the opaque pot before removing it from heat.

Answer PhoneRinging, speech, and the end of a call distinguish visually similar task stages.

Shake BottleThe robot actively shakes an opaque container, discerning internal contents from collision acoustics.

Real Alarm ClockThe robot waits for the alarm before pressing the stop button.

Core Insights and Engineering Takeaways

Integrating high-frequency continuous acoustic feedback into end-to-end embodied foundation models (VLA) demands far more than appending an extra feature dimension to the visual tokenizer. Conventional paradigms treat sound as an auxiliary camera stream, yet the physics of sound is intrinsically event-driven, high-frequency, and transient. Grounding it in robotic action mandates resolving the fundamental architectural tension between continuous acoustic cadence and chunked control loops.

This structural tension extends well beyond sound. Any physical contact modality that generates rapid, non-repeatable events faster than the policy’s replanning frequency—most notably high-frequency tactile arrays and micro-vibrations—encounters identical rate-mismatch bottlenecks. This asymmetry cannot be bypassed by merely optimizing the inference throughput of large foundation models.

Artificially increasing the model’s query rate fails for structural reasons rather than tuning deficiencies: truncating action chunks trades long-horizon trajectory smoothness for erratic reactivity, injecting severe mechanical jitter and self-noise directly into the sensory stream. The only robust solution is decoupling perception cadence from decision loops via explicit streaming causal memory. Auxiliary acoustic future prediction objectives ground shared representations in physical time at zero inference overhead, while rigorously timed causal benchmarks transform “appearing visually plausible” into “authentically listening to the physical world.”

What It Cannot Do Yet

One workspace microphone means no source localization and no separation of overlapping events, which is functionally the cocktail party problem. We would want an array, or a base-plus-end-effector pair for beamforming. The system is trained purely by offline imitation, so its robustness is bounded by the diversity of noise and timing variation in the demonstrations; online fine-tuning or reinforcement learning in the deployed room is the obvious remedy. Audio and vision are processed as separate streams and fused at the representation level, and we suspect native audio-visual encoders that associate timbre with contact directly would do better. And unconstrained real-world cooking acoustics remain an open problem, with real deployment markedly harder than simulation.

The Larger Frame

The tension we started from, between high-frequency transient sensory events and low-frequency delayed policy updates, is not specific to sound. It will show up for every physical modality that carries information in its timing. HEAR is our attempt to show that the fix is architectural, and to leave behind the pieces—the VSLA formulation, the pretraining corpus, and the benchmark—that let other researchers build further on this foundation.

MRASfM: Multi-Camera Reconstruction and Aggregation through Structure-from-Motion in Driving Scenes

Sat, 31 Jan 2026 00:00:00 +0000

Driving is the wrong shape for photogrammetry

Image-based reconstruction for driving is usually handled by visual SLAM, and for HD mapping and novel-view synthesis the SLAM route has a structural limitation. Its estimates are incremental and locally optimized, which blocks global refinement, and the result is drift that accumulates over a long trajectory. Structure-from-motion does the opposite: it processes everything in a batch with a global bundle adjustment at the end. The offline nature that makes SfM impractical for a live robot is exactly what makes it attractive here, because it can spend computation a real-time system cannot.

So the obvious move is to use SfM for driving. Three facts about driving data make that harder than it should be.

The first is the capture geometry. Classic SfM assumes you walk around the subject and photograph it from many directions. A car drives down a corridor. Viewpoints are almost collinear, which is the worst case for the translation estimation that SfM depends on.

The second is that data arrives fragmented. Driving scenes are captured over short durations with limited overlap, and drives covering the same area may be recorded weeks apart. There are often no shared images at all between two datasets that need to end up in the same map.

The third is calibration, and this is the one that changed how we designed everything. Vehicle calibration is difficult, initial values contain errors, and those errors get worse on uneven terrain. So the extrinsics you have at the start of a drive are not the extrinsics you need, and they are also the strongest prior available. That tension runs through the whole design.

What prior methods assume

Multi-camera SLAM systems are accurate and require precise internal relative poses and intrinsics as input. They need calibration done before deployment, and they struggle to accommodate different camera configurations, which restricts flexibility in practice.

Incremental SfM is the closest baseline, and our criticism of it is specific. It ignores camera-set priors during incremental reconstruction and applies them only afterward, in rigid bundle adjustment once the reconstruction is already complete. By that point, errors that accumulated during incremental estimation cannot be corrected. Rigid BA cannot undo drift that has already been baked into the structure. Per-image bundle adjustment has a related problem: it can leave the internal relative poses inconsistent from frame to frame.

Global multi-camera SfM does fix the relative camera poses from the start, and it is sensitive to outlier matches and tends to degenerate when the estimated relative translations are collinear. In a driving scene the trajectory is close to collinear by default, so this is a structural hazard rather than an edge case.

Odometry-guided SfM takes the opposite bet and uses odometry poses and camera extrinsics directly for registration. That demands highly precise extrinsic calibration and accurate odometry, which are exactly the two things field deployment cannot reliably supply. We borrow this line of work’s idea of using homography to identify overlapping image pairs. We do not borrow the assumption that the priors can be trusted.

There is also a matching problem specific to the setting. Traditional descriptors produce limited feature points in low-texture images, and road scenes are full of low-texture surfaces.

Finally, existing aggregation modules for merging reconstructions typically assume shared imagery between the segments being merged. In practice that imagery usually does not exist.

What makes it hard

Priors have to be used without being trusted. Extrinsics and odometry are the strongest signal available and the least reliable thing on the vehicle. Trust them entirely and you get one family of failures; ignore them and you get another. The design space in between is narrow.

Collinearity is a real degeneracy, not a numerical nuisance. Fixed-rig global methods hinge on relative translation estimates, and straight-line driving makes those ill-conditioned.

The failure modes sit exactly where the useful signal is. The road surface occupies a large part of every image and has almost no texture, so it produces both few matches and bad ones. Worse, the edges of vehicle shadows are high-contrast and get detected as feature points. A shadow edge is not a scene point. When the shadow moves, that feature moves, which injects motion into the road surface precisely where the structure matters most.

Occlusion behaves differently on a rig than on a single camera. One blocked camera on a monocular system is a total failure; on a surround rig the other cameras still see the world. But that redundancy is only worth anything if registration actually shares the pose, and per-camera estimation throws the advantage away.

And robustness has a price. More cameras mean more poses to optimize in bundle adjustment, which reduces efficiency. The coverage that buys robustness is paid for in optimization size, and cancelling that cost is what a large part of our design exists to do.

The camera set is the unit

Our idea is to change the atomic element of reconstruction. Instead of treating each image as an independent camera to be solved, we take the rigid unit, every image captured at the same instant, as the thing that gets a pose. That pose is by definition the vehicle’s pose at that moment.

The benefit is that the pose of a rig factorizes into a time-varying vehicle pose and a static internal relative pose. One well-observed camera can carry the poses of the cameras that see nothing useful. The rig is estimated once rather than re-learned at every frame.

The important part is how that factorization is applied. We use it as a reparameterization inside registration and bundle adjustment, not as a constraint imposed afterward. That single decision does three things at once. It reduces the pose variables from one per camera per timestamp down to one per timestamp plus one per camera. It keeps the optimized rig consistent across frames, because there is only one rig to be consistent. And it turns calibration from an input that must be correct into a quantity that can be corrected.

Building a reconstruction

Correspondence search comes first. SuperPoint detects and SuperGlue matches, pairs whose matched points fall into different semantic categories are discarded, and geometric verification removes the rest. Pairs are not all tried: with the rough internal relative poses and vehicle poses in hand, we compute the approximate field of view of each image and only match pairs that could plausibly overlap. Filtering this way improves both efficiency and accuracy, since matching everything against everything is expensive and injects noise from pairs that were never going to share content.

Registration then runs in three steps. PnP is estimated on the images with the richest correspondences. Those per-image estimates are converted into per-unit estimates and fused by local robust rotation averaging and translation averaging into a single unit pose. Every remaining image in that unit gets its pose by composing the unit pose with the calibration.

That middle step is where the method sits between the two failure modes described earlier. It averages, like the global family, but it averages over unit hypotheses derived from reliable images rather than over raw relative translations, which is what makes it survive collinearity.

The honest accounting of this module is narrow. It has minimal effect on average pose accuracy and efficiency, and it is particularly useful for handling occluded viewpoints. In scenes with rich correspondences the impact is less noticeable. We state that tradeoff rather than presenting the module as uniformly beneficial, because the value of redundancy is concentrated in the tail.

View full-size image ↗

Occlusion on one camera no longer fails the unit.

Getting the road out of the way

After a unit is registered, we fit a plane model to the road points with LO-RANSAC and remove them. This targets the shadow-edge problem directly. A moving shadow edge produces a feature that is not attached to the scene, and it lands on the largest and least textured surface in view, so the triangulation noise concentrates there by default. Removing those points improves reconstruction accuracy and accelerates convergence.

The assumption being made is local flatness of the road, and our own real-world scenes include slope changes, so this is an approximation rather than a guarantee.

View full-size image ↗

Removing shadow-driven and texture-poor points where the noise concentrates.

Bundle adjustment on the rig

Camera set bundle adjustment optimizes vehicle poses and internal relative poses under substitution constraints that express each camera pose in terms of the vehicle pose and the rig geometry, with a Cauchy loss for the local stage. The variables include intrinsics, vehicle poses, internal relative poses and scene points.

There are two modes and the split matters. Local CSBA fixes intrinsics and internal relative poses and touches only newly registered units and their neighbours. Global CSBA optimizes everything across all registered units, and we trigger it when many units remain unoptimized.

Global CSBA re-optimizing intrinsics and relative poses is the online recalibration capability claimed in the paper. It is also the practical payoff, because it converts the field’s least reliable input into an output. Local CSBA is cheaper, and it freezes exactly those parameters, which is the tradeoff: the cheap mode cannot fix calibration, and the mode that can fix it costs more.

The variable reduction from one pose per camera per timestamp to one per timestamp plus one per camera is worth dwelling on. Six cameras over a long trajectory means the per-image formulation optimizes six times as many poses as it needs to, and every extra variable is another degree of freedom that noise can hide in. Fewer variables makes convergence faster and the result more consistent. Robustness and efficiency turn out to be the same lever here, which is not usually the case.

View full-size image ↗

Vehicle poses and internal relative poses are refined jointly, which is what allows calibration to improve in the field.

Aggregation has to work without shared imagery, so the whole strategy is coarse-to-fine. Scenes are first placed in a global frame using satellite positioning, represented by trajectory midpoints: one scene sits at the geometric centre, its nearest neighbours become candidates, and homography-guided spatial pairs pick whichever candidate has the greatest visual overlap.

The coarse placement is wrong by an unknown transform, because positioning error is larger than the alignment tolerance. So the transform itself becomes the variable. One image from the incoming scene is relocalized to get a refined pose, which yields the transform between the coarse placement and the true one, and that transform registers the entire rigid unit at once. Triangulation follows, then a transformation-based bundle adjustment optimizes the transform and the scene points while keeping the reference scene’s camera poses fixed.

Registration, triangulation and adjustment iterate until every unit of the incoming scene has been absorbed. The merged result then becomes the new reference for the next scene, so aggregation proceeds as a chain rather than against a single global anchor. The reason is simple: chaining keeps error from originating at one fallible alignment.

The whole stage is designed so aggregation never has to be exact at the coarse step, only close enough to relocalize.

View full-size image ↗

Satellite positioning places the scene; vision refines the placement.

What we take from this

A physical invariant is best used as a reparameterization rather than a constraint. Enforcing rigidity after the fact cannot prevent the error that accumulated before. Making rigidity part of the variable structure prevents it from accumulating in the first place.

Priors should prune, not conclude. Rough poses decide which pairs to match and where to start, and the geometry still comes from matched evidence. That is the line between our approach and the one we criticize.

Design so priors can be corrected rather than merely trusted. Making relative poses optimizable is what turns the least reliable input into an output.

And judge a module where its failure mode lives. Camera set registration barely moves average pose error, and it is the difference between working and failing when a camera is occluded. Aggregate metrics can hide a module’s entire value.

What it still assumes

The method takes rough trajectories, rough calibration and semantic information as inputs. The semantic source is assumed rather than specified, and we report no sensitivity to the quality of those labels.

Multi-scene aggregation is not ablated. The choice of roughly three candidate scenes and the geometric-centre reference heuristic are stated without justification or failure analysis.

The rigid unit is defined purely by simultaneity, and we do not test what happens when that assumption weakens. Camera time-synchronization error and rig flex would both violate it, and neither is discussed. Online calibration is claimed but not characterized: we do not say under what motion diversity or camera count the global stage’s refinement of intrinsics and relative poses is well posed.

The plane model presumes a locally flat road while our own deployment includes slope changes. Generality across different multi-camera configurations is asserted but exercised only on six- and seven-camera rigs. Dynamic objects beyond shadows appear in the motivation but have no dedicated treatment.

Where it leads

The contribution we would state is a change of unit, and everything else follows from it. What the change buys is worth spelling out: camera sets as atomic units for registration and refinement give robust pose estimation in complex environments and better efficiency than conventional per-camera methods. Combined with the road filtering and the aggregation stage, the pipeline produces a single map from drives that were never meant to be merged.

We validated it in two directions on purpose. Real instrumented vehicles demonstrate that it holds up outside a dataset, and public benchmarks establish the accuracy claim against a conventional baseline. The practical statement of the stake is that intelligent vehicles can perceive their environment and estimate their own pose more reliably, which is what the downstream tasks actually consume.

MovSAM: A Single-image Moving Object Segmentation Framework Based on Deep Thinking

Sun, 19 Oct 2025 00:00:00 +0000

A photograph contains no motion

Moving object segmentation is normally a measurement problem. You compare successive frames, or you compute optical flow, and the pixels that changed are the pixels that moved. The assumption is so standard that it stops being visible.

It breaks in specific, practical ways. A pedestrian shifting their weight at the curb before stepping off produces almost no pixel change, and the intent is invisible to a frame-difference method. A vehicle’s own motion contaminates the comparison, since apparent movement in the image is a mixture of the other object’s motion and the camera’s own, and separating the two is not something a two-frame difference does. Occlusion makes pixel changes unreliable for anything partly hidden. And sometimes there is only one frame: if a camera fails or a communication link drops, the multi-frame pipeline does not degrade, it stops.

There is also a simpler observation behind this work. A person looking at a photograph can usually point at the thing that is about to move. Not because they measured displacement, but because they read the scene: the posture, the contact, the loose fabric, the blur along an edge. The motion is inferred from semantics rather than observed from change.

We wanted a system that could do that. Which means the temporal signal has to be replaced by something else, and language is the obvious candidate.

What earlier methods assume

Optical-flow-based approaches use short-term motion cues to separate moving objects from static background. Generating accurate optical flow requires multiple frames, so it is not a substitute for frames. It consumes them.

Multi-frame and video-based methods relate appearance to motion patterns and are genuinely effective. They also require either flow computations or temporal information from more than one frame, and on a single image they have nothing to work with.

Segment Anything is the natural tool for the segmentation half, and it has two problems here. It was built for point and box prompts, and text prompts are not its native interface, so a bridge has to be constructed rather than configured. More fundamentally, SAM analyzes image texture. It has no scene comprehension, so it cannot tell you which texture is moving.

The line of work that connects a multimodal language model to SAM for instructed segmentation is closer, and the distinction that matters is between recognizing and reasoning. Those methods enhance SAM with text and focus on identifying object categories. They answer what something is, not whether it is moving or whether the mask captured all of it. Generic open-vocabulary segmentation has a related blind spot: it does not address problems specific to motion, such as motion illusions, where appearance suggests movement that is not happening.

And stacking an MLLM, SAM and a vision-language model together directly does not work either. That composition still produces inaccuracies from hallucinations or flawed reasoning, and it produces visible contradictions between what was predicted and what was expected. Building a loop that can notice and repair those contradictions is the part that had to be designed.

What makes it hard

The deepest difficulty is that a still image contains no displacement information at all. Motion has to be inferred from static correlates, and those correlates are heuristics. A flowing effect in hair or clothing, motion blur along contours, an object suspended in mid-air without support, a pose resting on minimal support such as a single toe touching the ground: all of these suggest motion, and none of them prove it. The paper is explicit that the model identifies the moving object based on these features but not limited to them.

Motion illusions follow directly. Appearance that suggests movement can be wrong, and genuine movement can look static.

Ego-motion is unobservable in a single frame by construction. Nothing in one photograph distinguishes “they moved” from “I moved,” and no amount of reasoning recovers a distinction the data does not contain.

There is also a representation mismatch at the centre of the architecture. SAM’s image embedding is spatially rich and semantically empty. A vision-language model’s features are semantic and spatially coarse. Something has to make SAM’s detail legible to the VLM without destroying it.

Then there is the feedback problem, which is subtler than it first appears. For a loop to work, a mask has to be turned back into something the reasoner can critique. Not just whether the right object was chosen, but whether the mask is complete and whether it contains anything that does not belong. A prediction has to become an arguable claim.

Finally, supervision barely exists. There is no dataset of single images labelled with the moving object, because the label is inherently a judgement. We had to construct the training data by filtering existing video datasets down to images where the moving object is clearly distinguishable.

The mask as something you can argue with

Our idea is to replace the missing temporal measurement with an explicit linguistic hypothesis, and then make that hypothesis revisable.

The loop has three parts. A reasoner looks at the scene and writes a sentence describing the object it believes is moving. That sentence is not a bare category label. It carries the category, the position and the motion, in the form of something like “A red ball is flying in the air, in the upper right corner of the frame.” A segmentation stack turns that sentence into a mask. Then the mask goes back to the reasoner as visual evidence, and the reasoner is allowed to change its mind.

The last step is what makes this work. Reasoning about a still image is unreliable, and a single pass produces confident mistakes. Showing the result back closes the gap between what the model meant and what it produced.

The prompt is generated under a chain-of-thought guide, which converts an under-specified open judgement into something closer to a checklist: observe the objects and their relations, attend to the plausible movers, apply the motion features above, then emit a prompt or conclude that nothing is moving. The guide is hand-authored, and we did not test alternatives against it. That is a limitation we accept rather than hide.

The segmentation network takes the prompt and the image together and produces the mask. SAM2 is the core, and we keep its image encoder and mask decoder because they are well suited to the job. We add BEiT-3 features because SAM’s weakness is exactly semantic understanding, and the vision-language model supplies what it lacks. The prompt encoder is modified so that the VLM’s multimodal features can serve as a prompt directly, rather than being squeezed into a spatial point or bounding box.

The feature aggregation network is our answer to the representation mismatch, and it runs in the direction people usually do not expect. Instead of trying to make SAM semantic, we give the VLM SAM’s spatial detail.

Let $F_{\text{sam}} \in \mathbb{R}^{H \times W \times C_s}$ denote SAM2’s high-resolution spatial feature map. The aggregation network $\Phi_{\text{agg}}$ applies five convolutional layers with residual bottlenecks followed by adaptive spatial pooling and a linear projection, extracting a compressed global context descriptor:

$$ \mathbf{v}_{\text{ctx}} = \Phi_{\text{agg}}(F_{\text{sam}}) \in \mathbb{R}^{C_v}. $$

This global descriptor is broadcast and concatenated across every spatial location of the visual embedding:

$$ \tilde{F}_{\text{sam}} = \left[ F_{\text{sam}}, \; \operatorname{tile}(\mathbf{v}_{\text{ctx}}, H \times W) \right] \in \mathbb{R}^{H \times W \times (C_s + C_v)}. $$

The augmented representation is fed alongside prompt tokens $T_{\text{beit}} \in \mathbb{R}^{L \times D}$ into bidirectional cross-attention layers. Removing this aggregation pathway degrades validation performance noticeably, confirming that cross-modal feature alignment is functionally active rather than merely cosmetic.

View full-size image ↗

MovSAM uses scene semantics and appearance to infer likely moving objects from one image.

The reasoning loop: Propose, segment, critique

The critique stage has its own guided procedure. Given the original image with the candidate mask overlaid, the multimodal reasoner executes a four-point verification protocol:

Object Identity & Attribution: Does the segmented boundary correspond cleanly to the entity identified in the prompt?
Physical Motion Plausibility: Does the entity exhibit unambiguous static motion heuristics (unbalanced dynamic pose, airborne suspension, motion blur, directionally trailing fabric/hair)?
Segmentation Completeness: Has any contiguous functional part of the moving entity (extended limbs, carried equipment, attached garments) been erroneously severed from the mask?
Mask Purity: Has any non-moving background structure (road surface, cast contact shadows, adjacent stationary furniture) been incorrectly absorbed into the mask?

Notice that completeness and purity are separate questions from correctness. A mask can cover the right object and still be a bad answer.

When the reasoner rejects the result, it rewrites the prompt and the mask is regenerated. When it accepts, it must provide an explicit physical rationale. The published example illustrates how much work the loop accomplishes. The first prompt asks for the runner on the right side of the image, and the reasoner rejects it: that runner is in mid-stride but grounded and secondary. The second prompt targets the primary moving subject, identifying an upside-down breakdancer in the center. That proposal is accepted, justified by the physical observation that the dancer is inverted in mid-air with minimal support contact and impending rotational displacement.

That exchange embodies the architecture in miniature: the initial output was plausible but suboptimal; the second revision emerged directly from visually critiquing the concrete artifacts of the first pass.

We cap the reasoning loop at five iterations and terminate upon acceptance. If the budget expires without convergence, the candidate with highest verification confidence is selected.

View full-size image ↗

Because each frame is processed independently, occlusion does not propagate errors across time.

What is and is not trained

The SAM image encoder remains frozen, preserving its pre-trained visual primitives without incurring massive training overhead. The vision-language model, feature aggregation module, prompt encoder adapter, and mask decoder are jointly optimized for moving object segmentation. The high-level multimodal language model runs off-the-shelf in zero-shot mode without parameter fine-tuning, preserving its general reasoning and linguistic breadth.

The training objective combines balanced soft Dice overlap with focal binary cross-entropy:

$$ \mathcal{L}_{\text{seg}} = \lambda_{\text{dice}} \mathcal{L}_{\text{dice}}(M_{\text{pred}}, M_{\text{gt}}) + \lambda_{\text{focal}} \mathcal{L}_{\text{focal}}(M_{\text{pred}}, M_{\text{gt}}). $$

The soft Dice loss penalizes boundary and spatial volume discrepancies:

$$ \mathcal{L}_{\text{dice}} = 1 - \frac{2 \sum_{i} m_i \hat{m}_i + \epsilon}{\sum_i m_i + \sum_i \hat{m}_i + \epsilon}, $$

while the focal loss counteracts extreme foreground-background class imbalance where dynamic targets occupy an extremely small fraction of the image frame:

$$ \mathcal{L}_{\text{focal}} = - \sum_{i} \alpha_t (1 - p_{t, i})^\gamma \log(p_{t, i}). $$

In practice, this composite formulation provides superior boundary sharpness and training stability compared to either loss alone.

What we take from this

Text is a usable control channel for frozen foundation models. A prompt can carry category, position and a motion verb at once, which is far richer than a class label and requires no retraining at all.

The awkward representation mismatch has a clean resolution that runs against intuition: feed spatial detail into the semantic model rather than trying to make the spatial model semantic.

Making the output criticisable is the generalizable trick. It applies to any task where the supervision signal is a judgement rather than a measurement, and where a plausible wrong answer is the main failure mode.

And it is worth noticing that the composition itself can be the contribution. We say this plainly in the paper rather than pretending otherwise. The novelty is in the loop and in the way the parts are connected, not in any single component.

One more practical note. The training set was filtered down to images where the moving object is clearly distinguishable, which means the supervision concentrates exactly where static inference is easiest. We state the filter and we do not resolve the tension it creates.

View full-size image ↗

Qualitative comparison of single-image moving object segmentation.

What this opens

Single-image moving object segmentation is worth treating as its own problem rather than as a degraded video task. Recovering temporal information from one image is a capability we think has independent value, and it changes what a robot can do when its perception stream is degraded rather than healthy.

The reasoning mechanism is also portable. Any task that needs logical inference over a scene from a large model, including cross-modal settings such as reasoning about the relationships between objects in an image and a point cloud, could use the same propose, segment, critique structure.

The application we care most about is the one we started from. A robot that receives an isolated image, because a link dropped or a sensor failed, can still form an object-level interpretation of what is moving and what to watch. The paper’s closing claim is that this makes systems more robust in exactly the situations where conventional methods stop working.

ERMV: Editing 4D Robotic Multi-view images to enhance embodied agents

Wed, 23 Jul 2025 12:41:11 +0000

Demonstrations Are Expensive, Yet We Already Have Them

Vision-Language-Action (VLA) foundation models rely heavily on synchronized multi-view video demonstrations aligned with dense action trajectories. Capturing such teleoperated physical data is notoriously slow, labor-intensive, and non-scalable: a single dexterous manipulation episode entails thousands of high-resolution image frames across heterogeneous camera viewpoints. The scarcity of expert demonstrations constitutes a primary bottleneck restricting embodied policies from zero-shot generalization in open-world environments.

This investigation began with a fundamental observation: the behavioral intelligence already exists within recorded datasets; what is missing is visual diversity. A demonstration of picking up a mug conveys the essential physical mechanics, but reflects only a single physical instantiation of the environment. If an agent has only ever encountered that mug on a specific countertop under fixed lab illumination, it cannot transfer robustly to novel kitchens or unstructured homes.

To unlock this data bottleneck, embodied AI fundamentally requires an action-conditioned, geometrically consistent Robotic World Model. If a world model can take established high-quality action sequences as conditions, simulate physical rollouts across diverse visual counterfactuals (varied textures, lighting, cluttered backgrounds) while preserving the exact contact physics and execution dynamics, the value of existing robotic datasets can be magnified exponentially.

This defines the core identity of ERMV: a Controllable 4D Multi-View Robotic World Model. Rather than executing naive 2D image modifications, ERMV models the entire continuous 4D spatiotemporal physical trajectory as an action- and state-conditioned generative state transition flow.

The defining challenge lies in strict action-physics coupling. A robot demonstration is not a generic monocular video clip; it comprises synchronized multi-camera streams coupled with high-frequency kinematic states, end-effector poses, and gripper commands. If a world model introduces geometric hallucinations, disrupts projection consistency, or alters the physical contact boundaries between gripper and object, the generated rollout becomes an invalid, physically violating demonstration—poisoning rather than improving downstream policy training.

View full-size image ↗

Visual variation propagates through synchronized views while the recorded actions stay intact.

Why Conventional Generative Models Fail at Physical Simulation

In recent years, large-scale general video generators (e.g., Sora) have demonstrated astonishing visual fidelity, prompting efforts to adopt them directly as embodied world models. However, in precision robotic manipulation, unconstrained monocular video generation and autonomous driving world models fail fundamentally:

Rigid priors in driving world models break down in manipulation: Autonomous driving world models rely on static circumferential camera rigs and flat Bird’s-Eye-View (BEV) ground planes, where cross-view attention is hardcoded to fixed spatial baselines. In robot manipulation, the multi-camera setup is highly dynamic: eye-in-hand cameras move continuously relative to third-person perspectives, nullifying rigid geometric topology priors.
Video models without strict action conditioning cause physical hallucinations: Unconditioned video diffusion hallucinates plausible-looking textures but cannot enforce strict conditioning on joint angles, gripper states, and 6-DoF end-effector velocities. Objects morph, drift, or teleport without contact, and robotic fingers penetrate through solid boundaries—rendering the output useless for training robust VLA policies.
Frame-by-frame inpainting destroys 4D spatiotemporal coherence: Applying single-frame diffusion inpainting across frames lacks temporal and inter-camera constraints, leading to severe flickering, perspective tearing, and geometric inconsistency.
Static 3D reconstruction struggles with dynamic contact and motion blur: While NeRF or 3D Gaussian Splatting provides rigid 3D consistency, it struggles with complex transient contacts, deformable materials, and high-velocity motion blur frequently present in fast robot manipulation.

Embodied AI demands a world model engineered from the ground up to respect robot kinematics, guarantee 4D multi-camera geometric consistency, and support long-horizon physical rollouts.

What makes it hard

The camera rig is not fixed. Manipulation footage comes from a moving, egocentric multi-camera system, so any consistency mechanism keyed to known static inter-camera geometry stops applying.

Motion blur breaks the geometry we would normally rely on. Because the camera and the objects move at the same time, features sampled along a precise epipolar line in a blurry image may not correspond to the true pixel positions at all. The constraint that makes multi-view attention correct is, under motion, wrong.

Errors compound. Edited frames are fed back as history for the next window, so a small error does not stay small. Over long horizons this shows up as semantic drift and blurred detail. In this domain the tolerance is especially low, because the robot arm and the manipulated object have to stay consistent through the entire sequence. When they do not, the sequence does not merely look wrong. It stops being valid training data.

The compute tension is structural rather than incidental. Long-range memory wants a large working window, and dense attention over a window of L timesteps and N views wants a large amount of GPU memory. We wanted something that trains on a single consumer GPU.

Preserving the objects that matter has no cheap solution. Training a general segmenter for manipulated objects is not feasible, because those objects are diverse, often novel, and viewed from challenging egocentric angles. Labelling every frame by hand across thousands of images is not feasible either.

And text prompts lack the granularity the task needs. Asking for a background change to an office leaves the colour, the type of room, and its orientation undefined, and the result may conflict with what the robot is doing.

Use the structure that robot data already has

Robot data is unusually well structured. It comes with calibrated cameras, recorded camera poses, recorded robot states, and a shared clock. Our idea is to stop trying to invent a specification for the edit and instead reuse that structure, plus a single hand-edited keyframe.

The first frame of the primary camera is edited, by an inpainting model or by hand, and that edited image becomes the visual specification. The paper calls it a blueprint, which is the right word. It carries colour, layout and orientation unambiguously in a way that a sentence cannot, and it takes a person one edit rather than thousands.

Two design decisions follow from the structure, and both are about relaxing constraints rather than satisfying them.

The first concerns epipolar geometry. A moving gripper is smeared across pixels by motion blur, and the epipolar line computed from the poses points to where the gripper would be if it were stationary. Enforcing that constraint exactly matches the wrong pixel. So we keep epipolar geometry as the search structure and learn the deviation. A small network predicts the motion-induced offset from local image features and the motion condition, and attention is taken along the shifted line. The constraint is relaxed rather than discarded.

The second concerns attention. Dense spatio-temporal attention is the default way to keep a sequence consistent, and it is expensive. The domain-specific observation that justifies not doing it is that in many manipulation scenarios the background is largely static and the images change slowly. There are not many inter-frame distinctions that need to be captured by a dense mechanism. So we sample.

View full-size image ↗

A fixed budget of observations provides long-range context for the whole window.

Sparse context, and giving back what sparsity removed

Sparse Spatio-Temporal context takes a sliding window of timesteps and samples a fixed-size subset of images from it, which is far smaller than the full time-by-view grid. The sampling is random, and it is worth saying so plainly rather than attributing intelligence to it that is not there.

What makes it work is the second half. Each sampled image carries its original time index and camera index, and those identities are explicitly encoded as a condition. Subsampling destroyed the spatio-temporal structure, so the structure is re-injected. A frame from three seconds ago is still labelled as three seconds ago, which means it contributes appearance evidence without being mistaken for the current state. This is a small idea with wide applicability: when you subsample structured data, put the structure back.

There is a second decision that distinguishes this from the usual autoregressive scheme. The sampled history is injected into the network as a condition, and it is also generated alongside the future frames. Future frames can therefore extract geometric structure from history during generation, rather than only conditioning on it.

The memory saving from sparse sampling at a fixed window size is roughly a factor of two, and that factor is what makes single-GPU training possible at all.

Where the motion conditioning lives

Each target image is conditioned on the camera pose, the robot state, and the temporal deltas of both. The static terms ground each view geometrically, and the deltas are what allow the model to render motion blur at all. This is more consequential than it sounds. Removing the motion conditioning produces images that look sharper, because blur is a kind of softness, and they lack the physical characteristics a real camera would have captured. Perceptual sharpness is not physical realism, and a policy trained on impossibly sharp images is trained on a distribution that does not exist.

Training assumes synchronized pose and state records exist. That assumption is satisfied for robot demonstrations, which is why this works here and would not work on arbitrary video.

View full-size image ↗

The epipolar line is the search structure; a learned offset accounts for motion blur.

A loop that spends human effort where it is needed

Generated frames feed back as history, and history is exactly where errors accumulate. Rather than trying to solve this with a better generator, we handle it at the system level with a verifier.

A multimodal model acts as an automated checker, comparing each generated image against the original with a chain-of-thought prompt. It scores degradation of the core objects on a ten-point scale and emits a consistency judgement. When the check fails, the system asks the expert for a segmentation mask of the core objects, and that mask becomes an additional condition for a corrective regeneration.

We describe this as MLLM review plus expert correction, and we think of it as a practical intermediate path rather than a fully solved problem. The two obvious extremes do not work. A universal segmenter for manipulated objects is not trainable, and labelling every frame is not affordable. The loop keeps human annotation confined to the few cases where the model falters, which is where expert time is actually worth spending.

We should be clear about what this means for how automatic the pipeline is. It is not fully automatic. A person produces the guide frame, and a person may be asked for a mask. Making that boundary move is future work, and we point to semantic segmentation and object detection as the tools that would replace parts of the current manual intervention.

View full-size image ↗

An automated check catches degradation, and human correction is requested only where it fails.

What we take from this

A violated physical constraint should be relaxed rather than dropped. Blur breaks epipolar geometry, and the useful response is to learn the deviation while keeping the geometry as scaffolding. Discarding the constraint would have thrown away the structure that makes cross-view attention work at all.

The justification for sparsity should come from domain statistics, not from generic efficiency arguments. The background is largely static is a real property of manipulation scenes, and it is what makes sparse attention defensible rather than merely cheap.

When you subsample structured data, re-inject the structure you subsampled away. Two indices per sample is a small cost for preserving what the model needs to interpret them.

Error accumulation can be handled at the system level rather than inside the generator. A cheap external verifier is a legitimate alternative to a better model, and it has the side benefit of localizing where the model is weak.

And augment rather than re-collect. That is the practical thesis, and it is the reason the method is worth building even though it needs a guide frame and occasional human help.

System Boundaries and Physical Constraints

As a 4D world model built for real-world manipulation, we must be explicit about the system’s operational boundaries:

Lightweight Representation vs. Explicit 3D Geometry: ERMV pairs multi-view 2D diffusion with motion-aware epipolar attention to maintain 4D consistency implicitly, without yet adopting explicit representations like depth maps, point clouds, or 3D Gaussian Splatting (3DGS). Incorporating explicit 3D geometry remains a natural architectural avenue to further mitigate perspective distortion during rapid spatial displacements.
Human-in-the-Loop Verification: While the MLLM verifier significantly curbs human annotation burden, transparent objects, specular surfaces, and complex fluid dynamics can still produce subtle perceptual false negatives. Retaining a sparse expert intervention channel for degraded masks is a pragmatic engineering design; achieving fully autonomous unmonitored rollouts requires higher-capacity vision backbones.
Physical Dynamics and Kinematic Grounding: ERMV is a generative world model conditioned on recorded actions and robot kinematics. It faithfully synthesizes lighting, textures, backgrounds, and occlusions across time, but it is not an analytic physics engine. When arbitrary actions produce severe unmodeled fractures or turbulent non-rigid chaos, the model relies on the physical manifold captured within its training distribution.

Where It Leads: The Future of Embodied World Models

The primary contribution of ERMV is demonstrating that building a high-fidelity, controllable 4D multi-view robotic world model does not require unconstrained, computationally prohibitive black-box video models. Instead, deeply integrating robot kinematics and multi-view epipolar geometry yields an exceptionally compute-efficient engine for physical rollout and counterfactual generation.

As an embodied world model, ERMV unlocks several transformative applications:

Counterfactual World Engine: Conditioned on a single expert demonstration’s actions and states, the world model rolls out parallel physical realities across heterogeneous visual domains (countertop textures, ambient lighting, cluttered backgrounds, reflective surfaces), supplying downstream VLA foundation models with virtually limitless training data.
Action-Conditioned Rollout for Offline Verification: Given an initial physical scene and a candidate planned trajectory, the world model rolls out future multi-view 4D observation streams, serving as a safe offline sandbox to diagnose potential collisions or failures without endangering hardware.
A Generative Sim-to-Real Bridge: Large volumes of action trajectories can be sampled inside inexpensive physical simulators, and then transformed through ERMV’s action-aligned world model into multi-view streams endowed with real-world sensor artifacts (lens noise, dynamic motion blur, authentic material responses)—bridging the perceptual sim-to-real chasm through generative physical rollouts.

Robust Multi-Model Fitting through Learning Neighbor Regions

Sat, 14 Sep 2024 10:33:19 +0000

Multi-model fitting addresses a universal challenge across robotic navigation, 3D scene reconstruction, and spatial computing: real physical environments rarely contain a single clean geometric structure. Instead, dozens of intersecting planar walls, ground surfaces, and boundaries co-exist simultaneously, deeply corrupted by sensor noise and unassociated outliers.

In classical robust estimation, instantiating a candidate geometric hypothesis requires only a minimal sample set of cardinality $m$—for example, $m=2$ points for fitting a 2D line, $m=3$ points for a 3D plane, or $m=8$ correspondences for an epipolar fundamental matrix.

When researchers apply naive random sampling (as in standard RANSAC) to multi-structure scenes, the mathematical foundation of uniform sampling breaks down catastrophically. With dozens of structures co-existing under heavy background outliers, picking two points at random almost invariably draws points from two entirely different physical objects, or samples pure noise. The probability of randomly drawing $m$ points that all belong to the exact same physical structure collapses exponentially as the number of models grows. Yet under classical pipelines, every invalid sample pair is still fed into algebraic solvers and scored across the entire dataset. Computational budgets are overwhelmingly consumed by solving and evaluating hypotheses that were doomed from the start.

Conversely, extending single-model fitters greedily—such as Sequential RANSAC, which fits one model, strips away its inliers, and repeats—introduces hard-assignment clustering distortions. Real structures often share points along intersections and boundaries; greedily stripping inliers deprives adjacent structures of necessary geometric support, causing severe model under-counting and missed detections.

Three further constraints shaped what we built. The method has to work without being told how many models are present, which rules out a family of approaches that take that count as input. It has to run at a usable speed. And it has to transfer across tasks, because a multi-model fitter that has to be re-engineered per problem is not a general tool.

View full-size image ↗

Learning sits at the two decision points in an otherwise classical pipeline.

The four things that actually go wrong

We organize the problem around four failure modes, and each component of the method answers one of them.

The first is insufficient utilization of geometric features. Existing methods either rely on hand-designed criteria that only look at part of the geometry, or they are neural networks that never see it explicitly.

The second is inefficient optimization caused by noise. Solver calls are spent on hypotheses that were hopeless from the start.

The third is model overlap, which is the fundamental difference between fitting one model and fitting several, and which most formulations cannot even represent.

The fourth is the non-differentiable pipeline. Discrete minimal-set sampling and the task-specific minimal solver both sit in the middle of the process, and neither has a gradient. Challenges arise in incorporating learning-based methods precisely because of this, and the two available escape routes are both unattractive: replace the exact solver with a differentiable relaxation and lose it, or use policy-gradient estimation and accept the variance.

What competing methods leave unresolved

Greedy single-model extensions carry the hard-assignment problem described above. MultiRANSAC fits K models per iteration instead of one, which requires K as prior knowledge and becomes computationally inefficient as K grows, because finding several good models in a single iteration is genuinely difficult.

Iterative hypothesis generators are the more sophisticated family, and their shared defect is temporal coupling. Because each new model is built on the results of previous iterations, errors accumulate through model overlap as the process proceeds. Our own ablation is honest about the tradeoff here: substituting a stronger learned sampler into LNR raises accuracy and makes the whole process considerably slower. We chose the cheaper option deliberately and we report what it costs.

Clustering-style methods, including preference-set clustering and networks that classify minimal sets as valid or invalid, work well on the problems they target and ignore the reuse of data. Information gathered while evaluating one hypothesis is not carried into the evaluation of another.

Another family replaces sampling heuristics with a global energy function over geometric errors and cluster regularity, or a convex relaxation of the assignment energy, or hypergraph mode-seeking. These formulations handle overlap by construction, and they rely on cost functions complex enough that they may not fully exploit the geometric information actually present in the data. That limits how well they carry across diverse and complex scenes.

Efficient sampling methods that exploit parallel computation are strong, and they need the number of instances as prior information, which limits where they can be applied.

Finally, end-to-end deep methods achieve good accuracy and operate as black boxes. That is a real obstacle to reliable deployment, and it is the specific thing we did not want to build.

Learn at the decisions, condition on the points

Our approach is to keep the classical pipeline exactly as it is and put learning only where a decision is being made: before solving, to decide which minimal sets deserve a solver call, and after solving, to decide how each hypothesis should move and how good it is.

The critical design choice is what the network is conditioned on. The learned quantities are functions of the geometric features of data points, not of the hypothesis parameters. That is what allows the sampler and the solver to stay in the pipeline as non-differentiable black boxes in the forward pass. No gradient has to pass through either of them.

A precise statement is worth making, because it is easy to overstate. The solver is used during training: the labels for both classifiers come from comparing a solved hypothesis against the ground-truth instance, and the coarse labels come from solving the sampled sets. What does not happen is differentiation through the solver or the sampler. They are used forward-only.

Each point is lifted by a Fourier feature mapping to improve spatial frequency encoding, passed through shared multilayer perceptrons implemented as point-wise convolutions, and max-pooled into a global descriptor. Point-wise and global features are concatenated into a 256-dimensional vector, so every point carries both its own local geometry and a summary of the scene. The coarse module needs that holistic view. The fine module, as it turns out, does not.

View full-size image ↗

Overlap is representable because a point can contribute to several hypotheses.

Screening before solving

The coarse module draws a large number of minimal sets at random, deliberately more than needed to ensure broad coverage, and predicts a confidence for each. Only the top-scoring sets reach the solver.

Two things about this are worth dwelling on. The first is why the sampling stays random rather than becoming iterative. Random sampling avoids the error accumulation that iterative generators suffer from, and it preserves coverage across the whole observation set. It is a pragmatic choice rather than an optimal one, and we say so.

The second is that the confidence is trained as a binary classification problem, which means the module learns to select promising minimal sets without any explicit degeneracy test. Classical pipelines hand-code checks to reject degenerate configurations. Here that rejection is learned from solved hypotheses compared against ground truth. We replaced a hand-written rule with a learned one, and the benefit is that it adapts to the data rather than to the person who wrote it.

Removing the coarse module hurts significantly, because the fine network cannot cope with the resulting candidate volume. Pretraining the coarse module separately instead of training the two jointly also does worse, and the reason is the most interesting thing in the training design.

Judging a hypothesis by its neighbourhood

For each surviving hypothesis we define a neighbour region: the data points within some range under a task-specific distance. Euclidean distance for lines, a directional distance for vanishing points, reprojection error for planes, the normalized epipolar constraint for motion. The points in that region are encoded, each point’s residual to the hypothesis is concatenated to its features, short regions are zero-padded so they can be batched, and two heads read the result.

One head predicts an increment rather than a model. For a line it predicts a small change to each endpoint. For a vanishing point, a homography or a fundamental matrix it predicts a corresponding adjustment, and that adjustment is added to the coarse hypothesis. The network never invents a model from scratch. Verifying and correcting a guess is easier than generating one, so the classical solver generates and the network corrects.

The other head predicts a score. This is where the paper takes a position that surprises people: consensus maximization does not favour the best model. Counting inliers is the standard proxy for hypothesis quality, and when models overlap it is the wrong proxy. Learning the score from the data around the hypothesis is what replaces it.

The region representation is also what makes overlap representable at all. Two crossing lines have a shared neighbourhood near their intersection, and the points there sit inside both regions and count as evidence for both hypotheses. No hard assignment is made anywhere.

And the fine module deliberately discards the global context that the coarse module relies on. Local details are more relevant for refining an individual hypothesis, and global information can introduce many irrelevant details that hinder the process. The same feature that helps screening hurts refinement. That asymmetry is a reusable coarse-to-fine principle, and it is the reason the two modules are not simply the same network applied twice.

View full-size image ↗

The fine module scores hypotheses from the observations around them rather than from their parameters alone.

Deciding how many models there are

The scored hypotheses are redundant. Several of them will describe the same structure, and the answer should mention it once. We adapted non-maximum suppression from object detection to this setting, replacing intersection-over-union with a geometric notion of similarity: endpoint distance and angle for lines, proximity on the Gaussian sphere for vanishing points, and similarity between the per-point distance vectors for homographies and fundamental matrices.

This is also the mechanism that decides how many models the scene contains. The count is an output of suppression rather than an input to the method, which is what we wanted from the start.

How the two modules train each other

Three losses. The coarse confidence and the fine score are both binary cross-entropy against labels derived from similarity to the ground-truth instance. The refinement head is trained with a smooth L1 objective on the parameter increments.

The coupling is the part worth explaining. The fine-level loss back-propagates into the coarse-level module, so the selector learns to prioritize hypotheses that the fine module is able to work with, rather than simply matching a predefined label. The two modules share the encoded region features, which is what makes the gradient path exist. Training jointly beats pretraining the coarse module, and the reason is that the fine signal reshapes what good means at the coarse stage.

One further observation from the ablations that we did not expect. Training only on harder scenes, with more lines present, matched or beat training on easier ones, and we attribute this to the greater data variability acting as a regularizer. The same effect appeared more weakly for the baseline sampler we compared against.

What we take from this

Decouple learning from non-differentiable components by changing what you condition on rather than by making them differentiable. That refusal is the strongest methodological statement in this work, and it is what preserves the exact solver.

Optimize a selector for downstream compatibility rather than for its own proxy label. Pre-training loses information that joint training keeps.

Consensus is not quality. When models overlap, inlier counting is the wrong measure, and assuming it is right is what causes models to be under-counted.

The value of a feature depends on the stage. Global context helps screening and hurts refinement.

And model the multiplicity structurally. If the interface forces each point into one model, no amount of training will represent two.

What it assumes

The classifiers need supervision in the form of ground-truth instances. Behaviour in a domain with no ground-truth models is not something we characterize.

The region radius and the sampling budget are task-dependent quantities that we do not specify how to set beyond that they vary, and the suppression thresholds are not stated either.

Transfer across tasks means re-deriving four task-specific pieces: the region distance metric, the solver, the refinement parameterization, and the suppression similarity. That is real per-task engineering, and we do not claim zero-shot adaptation.

Zero-padding is our only mechanism for variable-size neighbourhoods, and whether it dilutes the encoding for regions with few points is not something we examined.

And random sampling is a tradeoff rather than an optimum. Our own ablation shows a better sampler would improve accuracy, and we did not resolve whether a cheap learned sampler could get both.

Why keep the two-step pipeline

We kept the established two-step strategy for a specific reason: interpretability. Numerical methods have inherent interpretability and clearly defined application scopes, and they struggle to fully exploit the geometric features of the data. End-to-end deep methods perform well and cannot tell you how they got there.

Because LNR keeps the pipeline, its outputs are explicit geometric models. The network accepts generic unordered data points, covering image correspondences, 3D point clouds and pixel coordinates, and the loss operates on geometric features rather than on hypotheses. That is what maintains compatibility with a broad spectrum of solvers and sampling processes, differentiable or not.

The stated ambition is modest by design. We hope this introduces a new idea for approaching multi-model fitting in a learned way, and we would rather have a method whose behaviour can be interrogated than one whose accuracy we cannot explain.

DiffSAC: Diffusion-guided Sampling for Consensus-based Robust Estimation

Sat, 31 Aug 2024 11:15:11 +0000

The expensive part is not the sampling

Sample consensus is the standard answer to robust estimation, and it has one structural property worth stating before anything else: a candidate set is not just proposed, it is spent. Every minimum set has to be solved into a model and then scored against all the observations before you know whether it was any good. With a few thousand points and a budget of a few thousand iterations, almost all of that work goes into combinations that were never going to produce the right model.

In geometric estimation, a minimal sample set is the smallest subset of observations required to compute model parameters uniquely: fitting a spatial line requires $s=2$ points, fitting a 3D plane requires $s=3$ points, estimating a homography requires $s=4$ correspondence pairs, and solving for a two-view fundamental matrix using the normalized 8-point algorithm requires $s=8$ pairs (or $s=7$ for the non-linear algebraic variant). Standard RANSAC draws these samples uniformly at random. According to binomial probability, if the dataset contains an outlier ratio $\epsilon \in (0, 1)$, the theoretical number of random sampling iterations $K$ required to ensure with confidence probability $p$ (typically $p=0.99$) that at least one clean, all-inlier minimal set is drawn is given by:

$$K = \frac{\ln(1 - p)}{\ln\left(1 - (1 - \epsilon)^s\right)}$$

Under moderate outlier contamination, $K$ remains tractable. However, in challenging unconstrained robotic environments featuring severe visual repetition, reflective surfaces, or extreme dynamic clutter, outlier ratios frequently exceed $\epsilon = 0.8$ or $0.9$. Under these conditions, the combinatorial denominator collapses catastrophically. When estimating an 8-point fundamental matrix ($s=8$) with $\epsilon = 0.8$, the probability of drawing an all-inlier minimal set in a single random trial is merely $(1 - \epsilon)^s = (0.2)^8 \approx 2.56 \times 10^{-6}$. To guarantee a 99% success rate, the system theoretically mandates over 1.8 million iterations. Even when practitioners enforce an empirical cap of several thousand iterations, the overwhelming majority of computational cycles are expended solving algebraically ill-conditioned models and calculating residual distances across thousands of points for hypotheses that were physically doomed from inception.

Furthermore, uniform random sampling is entirely oblivious to geometric conditioning. Two feature correspondences situated in close spatial proximity may be individually accurate, but numerically they span a near-zero spatial baseline, yielding a degenerate or numerically unstable model. Similarly, planar coplanar correspondences in two-view geometry induce fundamental matrix degeneracies. Uniform random sampling cannot evaluate the joint compatibility of a set of points as an integrated geometric entity; it merely aggregates independent random draws.

Critical Analysis of Prior Heuristics

Over the past two decades, extensive research has attempted to inject intelligence into sample generation, broadly bifurcating into three methodologies:

Prioritized Sampling and Local Neighborhood Heuristics: PROSAC sorts correspondence pairs by local matching metrics (such as nearest-neighbor distance ratios), drawing preferentially from top-ranked candidates. NAPSAC and Progressive NAPSAC assume inliers exhibit spatial proximity in image space, drawing samples within localized spatial balls. While effective in benign settings, these heuristics exhibit distinct failure modes: high feature similarity does not prevent geometric degeneracy, and restricting samples to tight local balls sacrifices wide-baseline triangulation geometry, collapsing when structural textures (e.g., repeating window grids or tiled floors) form locally coherent yet globally false matches.
Post-Hypothesis Refinement and Local Optimization: Algorithms such as LO-RANSAC, GC-RANSAC, and MAGSAC++ focus on the opposite phase of the estimation pipeline. Rather than altering how initial minimal sets are drawn, they accept standard random sampling and optimize the model once a promising candidate emerges—via graph-cut spatial clustering, iteratively reweighted least-squares, or marginalizing across continuous noise thresholds. These post-refinement tools are highly effective, and we actively integrate them into our downstream pipeline; however, they leave the primary bottleneck unaddressed: the astronomical cost of discovering that very first high-quality hypothesis within extreme outlier regimes.
End-to-End Differentiable RANSAC (DSAC Frameworks): DSAC and its successors relax the discrete hypothesis selection step into a continuous, differentiable probabilistic formulation, enabling end-to-end backpropagation. While theoretically elegant, direct policy gradient estimators behave as opaque black boxes in mission-critical robotics. When an autonomous vehicle or SLAM frontend experiences an unexpected trajectory jump, engineers cannot trace which specific correspondences caused the degenerate estimate. Differentiable models lack the modular, verifiable audit trail that production systems demand.

These limitations prompted our foundational inquiry: Can we reformulate minimal set sampling from passive, stochastic trial-and-error into goal-directed, conditional probabilistic generation? Instead of attempting to replace proven closed-form algebraic solvers with neural networks, we task a generative model with the hardest component of the problem: directly sampling high-quality, mutually compatible minimal sets.

Generating Candidate Sets Instead of Guessing

DiffSAC reconceptualizes hypothesis generation as reverse sampling from a Geometry-Conditioned Diffusion Model. If a neural network can model the posterior probability distribution of minimal sets that reliably yield correct models given an input observation cloud, inference reduces to generating a compact batch of candidate sets (e.g., 10 to 20 candidates). Every candidate is mathematically guaranteed to exhibit high all-inlier probability and stable numerical conditioning, which are subsequently resolved by deterministic classical solvers and evaluated via consensus scoring.

This paradigm establishes a direct conceptual parallel to text-to-image diffusion models. In text-to-image synthesis, a single conditioning prompt generates multiple diverse, photorealistic images from different random noise seeds, enabling users to select the most coherent realization. DiffSAC applies identical reasoning to geometric estimation: given an identical cloud of noisy correspondences, injecting independent Gaussian noise vectors into the reverse diffusion chain yields a small portfolio of diverse, non-degenerate minimal sets in parallel.

Critically, this formulation completely bypasses the computational latency that traditionally burdens generative diffusion models in robotics. In image generation, diffusion models denoise high-dimensional tensors containing millions of continuous pixels across dozens of autoregressive steps. In DiffSAC, by contrast, the generative target is merely a one-dimensional continuous confidence scalar per observation. For a typical visual keypoint set containing 2,000 matches, the total state vector is only 2,000 dimensions. Coupled with high-order ODE solvers, the complete reverse diffusion chain executes in 10 to 15 milliseconds on modern GPUs—an overhead that is dramatically outweighed by the computational savings achieved by eliminating hundreds of thousands of full-dataset model evaluations.

The pivotal theoretical abstraction centers on the definition of confidence: we define confidence not as an isolated measure of whether an individual point is an inlier, but as a joint set-level property measuring whether an observation belongs to a well-conditioned, mutually consistent minimal set.

Conditional Diffusion Process and Mathematical Formulation

Let the input geometric observation set be denoted by $\chi = \{x_i, d_i\}_{i=1}^N$, where $x_i \in \mathbb{R}^d$ represents spatial coordinates (e.g., 2D image coordinates or 3D point positions) and $d_i \in \mathbb{R}^{d_f}$ denotes local appearance feature descriptors (e.g., SuperPoint or SIFT embeddings).

Let $\boldsymbol{c}_0 \in [0, 1]^N$ represent the ideal target confidence field over the observation cloud. In continuous-time diffusion formulations, the forward corruption process is governed by an Ito Stochastic Differential Equation (SDE) that progressively diffuses $\boldsymbol{c}_0$ into standard Gaussian white noise over continuous time $t \in [0, 1]$:

$$\mathrm{d}\boldsymbol{c}_t = \boldsymbol{f}(\boldsymbol{c}_t, t)\mathrm{d}t + g(t)\mathrm{d}\boldsymbol{w}$$

The corresponding reverse denoising trajectory is defined by the Probability Flow Ordinary Differential Equation (ODE):

$$\mathrm{d}\boldsymbol{c}_t = \left[ \boldsymbol{f}(\boldsymbol{c}_t, t) - \frac{1}{2} g(t)^2 \nabla_{\boldsymbol{c}_t} \log p_t(\boldsymbol{c}_t \mid \chi) \right] \mathrm{d}t$$

Where the conditional score function $\nabla_{\boldsymbol{c}_t} \log p_t(\boldsymbol{c}_t \mid \chi)$ is approximated by a parameterized neural denoiser $\boldsymbol{s}_\theta(\boldsymbol{c}_t, t, \chi)$. Denoising is strictly conditioned on the empirical geometric observations $\chi$, anchoring the trajectory to actual epipolar geometry constraints rather than unconstrained prior hallucinations.

During supervised training, ground-truth geometry provides the supervisory signal $\boldsymbol{c}_0$: given a known true model, the system samples a minimal set of verified inliers that yields a well-conditioned algebraic solve, setting their target confidence to 1 while assigning all remaining points 0. The denoiser is trained via an $x_0$-prediction formulation to predict the clean confidence field $\hat{\boldsymbol{c}}_\theta(\boldsymbol{c}_t, t, \chi)$ directly, minimizing the Mean Squared Error (MSE) objective:

$$\mathcal{L}_{\text{diff}}(\theta) = \mathbb{E}_{t, \boldsymbol{c}_0, \boldsymbol{\epsilon}}\left[ \left\| \hat{\boldsymbol{c}}_\theta(\boldsymbol{c}_t, t, \chi) - \boldsymbol{c}_0 \right\|_2^2 \right]$$

This objective forces the network to learn which subsets of observations cooperatively eliminate algebraic singularities under varying noise corruptions.

View full-size image ↗

Forward corruption supplies training targets; reverse diffusion generates candidate minimum sets.

Permutation-Invariant Transformer Denoiser Architecture

In geometric perception, the ordering of feature points in an input array is entirely arbitrary. Imposing fixed coordinate orderings or 1D convolutions introduces spurious sequential biases and prevents models from handling varying numbers of observations $N$.

To preserve physical validity, we construct the denoiser as a Transformer without positional encodings:

Feature Embedding: Each observation combines geometric coordinates $x_i$, appearance descriptors $d_i$, and the current noisy confidence scalar $c_{t, i}$ into a unified vector, projected through a multi-layer perceptron (MLP) into a latent representation $z_i^{(0)} \in \mathbb{R}^D$;
All-to-All Self-Attention: Latent tokens pass through stacked self-attention and cross-attention blocks. The self-attention matrix: $$\boldsymbol{A} = \text{Softmax}\left(\frac{\boldsymbol{Q}\boldsymbol{K}^T}{\sqrt{d_k}}\right)$$ enables every observation to adjust its confidence dynamically based on the geometric distances, disparity gradients, and appearance similarities of all other candidates in the cloud. This global receptive field is precisely what evaluates multi-point geometric compatibility;
Strict Permutation Equivariance: By omitting positional encodings, the network satisfies formal equivariance: for any permutation matrix $\boldsymbol{P} \in \mathbb{R}^{N \times N}$, $\hat{\boldsymbol{c}}_\theta(\boldsymbol{P}\boldsymbol{c}_t, t, \boldsymbol{P}\chi) = \boldsymbol{P}\hat{\boldsymbol{c}}_\theta(\boldsymbol{c}_t, t, \chi)$. Shuffling the input array guarantees an identical, co-permuted confidence output.

Ablation benchmarks confirm that replacing the Transformer with PointNet++ or k-NN graph convolutions (DGCNN) severely degrades wide-baseline generalization, proving that all-to-all attention is essential for resolving long-range geometric dependencies across wide camera baselines.

View full-size image ↗

No positional encoding is used, preserving permutation invariance.

Fast Inference: Batch-Parallel Diffusion and Consensus Verification

At test time, DiffSAC executes an efficient, highly parallelized pipeline:

Batch Parallel Initialization: The observation cloud $\chi$ is replicated $B$ times in GPU memory (typically $B=20$). Each batch replica is initialized with an independent standard Gaussian noise vector $\boldsymbol{c}_1^{(b)} \sim \mathcal{N}(\mathbf{0}, \boldsymbol{I})$;
High-Order Numerical ODE Solvers: Invoking DPM-Solver++ compresses the reverse trajectory from hundreds of steps down to 10 to 15 discrete evaluation steps, executing concurrently across all $B$ replicas within 12 milliseconds;
Top-k Hard Extraction: From each refined confidence field $\boldsymbol{c}_0^{(b)}$, the system selects the $s$ observations with the highest confidence values via a hard Top-k operator, yielding $B$ candidate minimal sets $\mathcal{S}^{(b)}$;
Deterministic Algebraic Solves & Consensus Scoring: Closed-form solvers (e.g., SVD for line and plane fitting, normalized 8-point algorithm for fundamental matrices) solve for hypotheses $\mathcal{M}^{(b)}$. Each model is scored against all observations using standard consensus metrics (such as MSAC or truncated M-estimator residuals), selecting the hypothesis with the highest inlier support. Finally, the winning model is refit over its entire consensus set.

This split—stochasticity at initialization, determinism at selection—is fundamental. Independent noise seeds allow the diffusion process to explore distinct modal basins of the conditional distribution. If one candidate encounters a localized singularity, alternative parallel chains provide robust, non-degenerate solutions, eliminating single-point failure modes.

View full-size image ↗

The generated confidence changes which observations enter the minimum set.

View full-size image ↗

Conditioned confidence proposals feed the classical two-view geometry pipeline.

The Decisive Ablation: Why Iterative Diffusion Is Irreplaceable

When evaluating DiffSAC, a natural skepticism arises: Why employ an iterative diffusion process? Why not train a feedforward regression network to predict the confidence vector directly in a single forward pass?

To resolve this question definitively, we conducted a direct ablation: retaining the identical Transformer backbone, loss function, and training dataset, but eliminating the noise schedule and training the network to regress confidence values $\hat{\boldsymbol{c}}$ from clean observations $\chi$ in a single step.

The empirical disparity was stark:

The single-step feedforward regressor produced diffuse, poorly separated confidence distributions. It consistently failed to establish sharp contrast between true inliers and degenerate near-inliers, resulting in candidate sets heavily contaminated by outliers;
In contrast, the iterative reverse diffusion process exhibited systematic self-correcting dynamics: early steps leveraged global attention to purge obvious outlier matches; intermediate steps pruned geometrically degenerate configurations (such as collinear points or negligible parallax); and terminal steps rapidly polarized confidence into a sharp, decisive set of high-leverage inlier landmarks.

This comparison isolates the true driver of performance: iterative diffusion succeeds not because of added stochasticity, but because it decomposes an intractable high-order combinatorial search problem into a sequence of smooth, gradient-guided trajectory refinements.

Furthermore, because DiffSAC preserves standard algebraic solvers and consensus scoring, it integrates seamlessly with existing local optimization frameworks. Feeding DiffSAC candidate hypotheses into LO-RANSAC for secondary graph-cut refinement produces compounding accuracy improvements, uniting generative global sampling with classical local convergence.

Comprehensive Benchmark Evaluation and System Boundaries

We validated DiffSAC across synthetic stress tests and extensive real-world benchmarks, including ScanNet, MegaDepth, and PhotoTourism. Tasks encompassed line fitting, 3D planar segmentation, fundamental/essential matrix estimation, and homography recovery:

Under extreme 80% synthetic outlier contamination, classical RANSAC achieved less than 15% estimation success within 5,000 iterations. DiffSAC achieved a 94% success rate with only 20 candidate evaluations, reducing average rotation and translation errors by an order of magnitude;
In real-world wide-baseline scenes with repetitive architectural textures, DiffSAC dramatically suppressed degenerate models, outperforming PROSAC, NG-RANSAC, and MAGSAC++ in pose estimation AUC metrics.

We also explicitly state the system’s operational boundaries:

Model Specialization vs. Universality: Current models are trained for specific geometric degrees of freedom (e.g., separate network weights for homographies versus essential matrices). Developing a single unified foundation model capable of conditioning on arbitrary geometric constraints remains an active frontier;
Computational Hardware Requirements: While DPM-Solver++ reduces inference to 10-15 steps, all-to-all attention across thousands of points requires GPU tensor parallelism. On resource-constrained micro-controllers or low-power embedded platforms lacking GPU acceleration, running Transformer diffusion chains requires model pruning and INT8 quantization.

Engineering Philosophy: Reclaiming Auditability in Learned Geometry

The core thesis of DiffSAC is an intentional defense of modular engineering in deep learning: assign neural networks to solve difficult, high-dimensional pattern recognition and combinatorial sampling, while entrusting deterministic classical algorithms with well-understood algebraic solving and residual consensus verification.

Discovering well-conditioned inlier subsets among tens of thousands of noisy correspondence candidates is a non-convex, combinatorial search problem where deep learning excels. Conversely, calculating model parameters from a minimal set and measuring reprojection residuals are analytical, closed-form operations that neural networks should never approximate.

This disciplined separation delivers exceptional estimation accuracy while preserving absolute engineering interpretability. Whenever an anomaly occurs, practitioners can inspect the confidence field, examine the condition number of the minimal set matrix, and verify the algebraic consensus curve, ensuring that critical robotic perception systems remain transparent, verifiable, and safe.

RLSAC: Reinforcement Learning Enhanced Sample Consensus for End-to-End Robust Estimation

Sun, 01 Oct 2023 00:00:00 +0000

The sampler is the part nobody trains

Robust estimation carries most of geometric vision on its back. Camera pose, motion segmentation, wide-baseline matching, plane fitting, line fitting: strip away the application and they all come down to the same question. Given observations where a large fraction are simply wrong, recover the model that the correct ones agree on. RANSAC has been the default answer to that question for four decades. It draws a minimum set, solves for a model from that set, counts how many observations agree with the result, and tries again. Two points define a line. Eight correspondences define a fundamental matrix.

The drawing step is deliberately blind. Every observation is treated as equally worth trying, and nothing about the data ever enters the decision. That bet pays off when outliers are rare. It stops paying off quickly. Raise the outlier fraction and the probability of drawing a set entirely from the inliers falls away, and the number of iterations needed to find a good model climbs with it. The evidence that would have made the search easy is sitting in the observations themselves, and the sampler never looks at it.

The second problem is quieter, and it is the reason we wrote this paper. A uniform random draw is not differentiable. You cannot attach it to a network, and you cannot train it against an objective that lives downstream of it. So the robust estimator ends up parked at the edge of a learned system: an off-the-shelf component wrapped in interfaces, never optimized together with the parts around it.

Earlier work has attacked this from several directions, and each one leaves something on the table. LO-RANSAC keeps sampling inside the inliers of the best hypothesis so far, which improves a model it has already found rather than the search that found it. NAPSAC restricts sampling to a local neighborhood and buys better-conditioned sets at the cost of global coverage; Progressive NAPSAC’s expanding neighborhood patches that assumption rather than replacing it. GC-RANSAC and MAGSAC++ change how hypotheses are scored, not how they are proposed. PROSAC and USAC reorder or schedule the sampling, but the ordering rule is designed by hand and frozen before the first hypothesis has been scored. Methods such as NG-RANSAC learn a useful preference over points, and the filtering approaches around it discard bad candidate sets early, yet neither treats the sequence of sampling decisions as something worth planning. DSAC takes the most direct route and makes selection probabilistic so gradients flow through the pipeline. That works, but it requires the sampler, the solver and the scoring step to all be differentiable, and the classical pieces we trust most are exactly the ones that are not.

The sample is an action

Our starting observation is that the two problems have the same shape. In reinforcement learning, an agent reads a state, picks an action, and the environment reports what happened. Sampling a minimum set has that structure already. The observations are the state. The set chosen from them is the action. And the environment exists and is running: it is the solver and the consensus function.

The reward is the interesting part, because the pipeline produces one for free. Once the solver turns a proposed set into a model, the consensus function measures what fraction of all observations agree with it. That number scores the model, and since the model came out of that particular set, it also scores the choice that produced it. No labels for which points are inliers are needed anywhere in the loop. Any task that already ships a solver and a residual definition comes with its own reward.

That last point matters more than it sounds. Training from the joint outcome means the policy is never taught which individual observations are good. It is taught whether the model that came out of a combination was good. Pointwise inlier classification is a different and much stronger assumption than set-level scoring, and it is the assumption we did not want to make.

Question	RLSAC’s answer
What is learned?	The sampling policy; the solver and the scoring function stay as they are
What supervision is needed?	Nothing beyond the consensus score the pipeline already computes
What enters the state?	Observation features, the chosen set, residuals against the last hypothesis, and usage counts
Where is it used?	Synthetic line fitting and real two-view fundamental-matrix estimation

What the policy gets to see

A state that contains only the observations is not enough. The same data at step one and step ten would look identical, and the policy would have no way to know what it had already tried. So the state is built to carry the consequences of the previous decision.

Four signals are stacked per observation. The first is the observation’s own features: point coordinates for line fitting, or coordinates, matching scores and local descriptors for correspondences. The second is an action indicator, +1 for the observations that formed the last minimum set and -1 for everything else. The third is the residual from the previous hypothesis, which is dense evidence about how the last model related to every observation, not only the ones that built it. The fourth is a counter of how often each observation has been selected so far.

We think of these last three as a navigation aid. The residuals say which direction looks promising. The usage counter is a breadcrumb trail, recording where the search has already been. The action indicator marks the current position. None of them changes the underlying geometry; they change what the policy can conclude from it.

View full-size image ↗

From one sample to the next: observation features, the selected set, residuals, and sampling history form a state transition.

One episode, one random start

RLSAC runs several episodes over the same input, and each episode opens with a randomly sampled minimum set. This costs almost nothing and buys two things. It guarantees a performance floor, because every episode starts from an unbiased draw and the policy can only improve on it. And it gives the policy a meaningful state to react to, since a residual field computed against a terrible hypothesis is far more informative than one computed against nothing.

The policy itself is an EdgeConv-style graph network. The choice is not cosmetic. Observations have no canonical order, and a policy that read them as an ordered vector would behave differently depending on how the input happened to be shuffled. The graph aggregates each observation’s neighbors into its own representation, which keeps the policy permutation-invariant and lets points exchange local context before any of them is proposed.

The network outputs a probability per observation, and the top-scoring set becomes the proposal. One detail worth stating plainly: that probability is not a claim about how good an individual observation is. It is a claim about the long-run return from including it. The two are not the same, and treating them as the same is a common way to misread what the policy has learned.

Duplicate sets are rejected. Every previously used minimum set is recorded, and if the top-scoring set has already been tried, the next one is drawn according to the probabilities and checked again until an unused set is found. Without this, a policy that has learned a good region will keep re-proposing the same set and burn the entire budget on a hypothesis it already has.

Training uses discrete Soft Actor-Critic with a replay buffer, sampling probabilistically so the policy keeps exploring. At test time the sampling switches to greedy selection. The split is the standard exploration-exploitation tradeoff, and we compared the other three combinations against it before settling on this one.

Episodes terminate on any of three conditions during training: the inlier count stops changing for two steps, the inlier ratio fails to beat the episode’s best for three steps, or the episode hits its step cap. Testing keeps only the cap. The distinction we wanted was between giving up on a search that has stopped making progress and stopping one that is still improving but has run out of budget.

Watching it work on a line

Set a few hundred points in a square, draw a line through some of them, let the rest be outliers. Any pair of points defines a candidate line. Every other point has a distance to it, and that distance is the feedback. The hypothesis visibly walks toward the ground truth across successive steps, which is satisfying to watch but not the interesting part.

The interesting part is what the policy does after it gets there. Once it finds a hypothesis with a high inlier ratio, it keeps working locally, refining the set rather than jumping to a different region of the data. A sampler without memory cannot do this, because it has no way to know it is already close. This is the clearest argument for putting the residual and history features in the state, and it shows up in the qualitative traces much more plainly than in any aggregate number.

Our synthetic data also places outliers inside the inlier band, which is what happens in real scenes. That rules out memorizing coordinates. What is learnable is the distribution of point groups that make a well-conditioned model, and that is what we wanted the policy to pick up.

View full-size image ↗

Illustrative sequence from the paper: hypotheses and residual evidence change across policy transitions.

From points to image correspondences

Two-view geometry changes the observation but not the interface. An observation is now a match between two image locations, described by its coordinates, its matching score and a local descriptor. A minimum set of eight goes to the eight-point solver, which returns a fundamental matrix, and consensus scoring measures how well every correspondence agrees with the resulting epipolar geometry.

In this setting the policy settles on rigid, well-localized feature points on buildings and similar structures. Nobody told it to prefer those. They are simply the matches that survive the geometric score, episode after episode, and the policy found them.

View full-size image ↗

The learned sampler receives feedback from a classical geometric solver.

What carries across the two tasks is not a trained network. It is the contract: encode observations, propose a minimum set, solve, score, feed the evidence back. Moving to a new problem means supplying task-appropriate features, a solver and a residual definition. The paper’s claim about transferability rests on that interface, together with the random restart and the duplicate rejection, not on the state transition by itself.

What we take from this

The lesson we keep coming back to is that reward can substitute for differentiation. If a downstream metric can score an output, it can train the module that produced it. There is no need to make an argmax, a solver or a threshold differentiable, which means the interpretable classical pieces get to stay. That is what allows a traditional estimator and a learned one to be optimized toward the same objective instead of sitting on opposite sides of an interface.

Memory also turns out to be worth more than prediction here. Feeding the last attempt’s residuals back into the state is cheap and dense. Training a better one-shot predictor is expensive and, in our setting, not obviously better.

And a learned sampler should never be allowed to underperform the random baseline it replaced. Seeding every episode with a random draw is what buys that.

Where it stays limited

An honest list. The inlier threshold is still a hand-set hyperparameter and it directly determines the reward, which is the same sensitivity MAGSAC++ was designed to remove; we reintroduced it and did not study it. The state only tracks the top-N correspondences by matching score, so a good match outside that set can never be selected. Our comparisons for two-view geometry are against classical methods only, even though we discuss learning-based samplers throughout; readers should not read those as head-to-head results. Transfer to other tasks is argued from the interface, not demonstrated. And we never analyzed cost: multiple episodes times up to fifteen steps, each with a graph forward pass, a solver call and a full residual computation, is a real bill, and we did not compare accuracy at matched iteration budgets.

The larger point

The discussion section of the paper makes the argument we actually believe. One-shot deep estimators are not reliable outside the distribution they were trained on, and their poor interpretability makes them hard to deploy in engineering settings. Classical estimators have clear behavior and well-defined scope, but cannot be folded into a learned framework because they are not differentiable. Combining the two gets both properties. If a component can be scored, it can be learned, and it can keep being the component you trust.

We are curious whether the same treatment extends to other non-differentiable pieces of the geometric stack. Noise covariance estimation in SLAM is the one we keep thinking about.

3D Object Detection and Tracking Based on Lidar-Camera Fusion and IMM-UKF Algorithm Towards Highway Driving

Tue, 04 Apr 2023 00:00:00 +0000

The design constraint behind this work is not accuracy. It is latency.

Most fusion-based 3D detection algorithms run below 15 Hz on a workstation, and a workstation is not what drives a car. An onboard computing centre is slower still, and it is not dedicated to perception: planning, control and the rest of the stack have to run concurrently, which takes a further bite out of the same budget.

Now put a number on what 15 Hz costs. At highway speed the vehicle covers about 1.94 metres between two consecutive frames. For that distance the perception system effectively does not see anything that has just happened, and we call that being time blind. The consequences are not abstract. A system that reacts to where a car was two metres ago is reacting to a situation that no longer exists.

Thirty hertz substantially reduces the blind distance and leaves room for the other modules to run. That target, on affordable hardware rather than a development machine, is what shaped every choice in this paper. It also means we made a deliberate trade: LiDAR-only methods achieve better detection accuracy than fusion methods do, and we did not set out to beat them on that axis. We set out to be competitive while running at a frequency they cannot reach.

What the fusion families cost

There are three ways to combine an image with a point cloud, and each has a price.

Detecting in 2D and lifting the boxes into the point cloud, the frustum approach, makes the whole system depend on the 2D detector. An error in the image branch propagates into 3D with no way to recover it.

Fusing at the feature or proposal level and training end to end produces strong results, and it loses original information in the process of fusing at the region level. These methods also tend to be complex and to run slowly, which is the specific problem we were trying to solve.

The third family projects the point cloud into an image feature map and augments each point with what it finds there. The architecture stays close to a LiDAR-only detector, which is what we wanted, and the pillar encoder keeps the runtime low. We chose this family for exactly that reason.

Within LiDAR-only detection the lineage matters too. Voxel-based 3D convolution was accurate and slow. Sparse convolution improved efficiency without making it fast enough. Pillars, which reduce the problem to 2D convolutions, hit the best tradeoff between accuracy and speed, and we built on that.

For tracking, the choice of a classical filter over a learned one was deliberate rather than conservative. Learning-based trackers perform poorly in real time, and neural data association is time-consuming, so the Hungarian algorithm stays. Batch tracking is ruled out for a different reason: it uses all data, and a driving system only ever has history. 2D tracking is ruled out because it cannot tell you an object’s position and size in 3D.

The closest filter-based alternative is a well-known 3D tracker that is simple and efficient and uses a single constant-velocity model. A constant-velocity model has no way to represent a car that is turning, so its state estimate degrades in exactly the situations where prediction matters most. A monocular tracker uses an extended Kalman filter under one camera. An urban JPDA-IMM-UKF system exists and uses traditional clustering for detection, which produces fragmented trajectories and hurts tracking performance downstream.

IMM itself is standard in aviation and nearly absent in vehicles, and the reason is instructive. Aircraft maneuver drastically: large speed changes, fast heading changes, so a switching model earns its complexity. Vehicle motion is comparatively mild. And with little prior work in the automotive setting, the parameters that IMM needs are hard to set.

What makes it hard

Fusing at full image resolution, per point, at thirty hertz, is a budget problem before it is an accuracy problem. The number of image channels you attach to each point is the knob you turn, and we tuned it as a tradeoff rather than maximizing it.

Data-layer fusion is also fragile in a specific and quiet way. Because each point is matched to an image location, a calibration error puts the sampled feature at the wrong pixel for every point, and the resulting input looks entirely plausible. There is no error signal. The system does not know it is being fed garbage.

Choosing a motion model is a lose-lose if you must choose one. Constant velocity is simple and strong for linear motion. Constant turn rate and velocity handles rotation. Either alone is wrong in the other regime. Switching between them is the answer, and switching introduces a set of prior parameters: the state switching matrix, the state covariance, the process and measurement noise covariances. These are conventionally set from experience, and empirical values are not guaranteed to apply to a different scenario. That is a real methodological problem, not a tuning nuisance.

There is a nonlinearity problem too. Constant-velocity transitions are linear in the state. Constant-turn transitions involve position expressed through velocity and yaw rate multiplied by sine and cosine of the heading, which is not linear, so the filter needs sigma points rather than a plain Kalman update.

Tracking also has to survive gaps and resist noise at the same time. Occlusion causes detections to drop, and being too permissive about starting new tracks admits detector false positives. The two failure modes pull in opposite directions.

And the physics caps detection regardless of architecture. In one of our examples, three very distant cars return only three to five LiDAR points each. Too few points and the detection simply does not fire.

Attach a neighbourhood, not a pixel

The first half of the idea addresses the fragility of data-layer fusion. Instead of sampling a single projected pixel for each point, we sample a 5×5 window of image features around the projection and attach the whole neighbourhood to the point.

The consequence is that the network learns the matching relationship between the two modalities rather than trusting the calibration exactly. A point whose projection lands a pixel or two off still finds relevant appearance features in its window. This was the single most useful conceptual move in the detection half, and it is what makes the difference between fusion that survives calibration error and fusion that quietly degrades.

The features come from a full-resolution extractor, which matters for a related reason: the point has to map to a precise image location for the window to mean anything, and a downsampled feature map destroys that precision. The encoder is VGG-based with two modifications. We reduced the channel count to keep optimization manageable, and we cut the network off in the middle so that the feature map does not shrink too far, which would make it useless for generating fusion data. A top-down decoder upsamples and merges with same-resolution encoder maps, with 1×1 convolutions controlling the output dimension.

View full-size image ↗

A local window learns the correspondence between the two modalities instead of assuming the calibration is exact.

Collapsing the height axis to get the frame rate

The augmented points then go through a pillar encoder. Points are binned onto a bird’s-eye-view grid with cell size and a cap on how many points a cell keeps, with random subsampling beyond the cap. The height axis is collapsed to a single cell, which is the trick that turns 3D convolution into 2D convolution and is where the frame rate comes from.

Each point carries its geometry and appearance plus five offset features that describe where it sits relative to its cell’s mean and its voxel centre. A PointNet layer expands these, max-pooling reduces each cell to one representative, and the cells are scattered back into a pseudo-image. A 2D convolutional backbone and a region proposal head then predict oriented 3D boxes.

The cost of the pillar representation is that height structure inside a cell is discarded and the detection range is bounded by the camera’s field of view, since fusion needs pixels. We accept both as the price of the frequency target.

One loss term deserves a note because it looks redundant and is not. The yaw residual is computed as the sine of the difference between predicted and anchor angle, and sine is symmetric. A car facing forward and a car facing backward produce the same residual. Regression alone cannot break the tie, which is why a separate direction classification term exists in the loss.

View full-size image ↗

The fusion-augmented point cloud is encoded as a bird’s-eye-view pseudo-image.

Two motion models, and a search for the priors

The tracker predicts every existing trajectory, associates predictions to detections by the Hungarian algorithm on 3D overlap, updates matched tracks, and manages births and deaths for everything unmatched.

The interesting part is the update. Two models run in parallel, one with a constant-velocity state and one with a constant turn rate and velocity state. Mixing probabilities are computed from the previous model weights and the switching matrix, mixed estimates are formed for each model, each model’s sigma-point filter predicts forward, and the two are combined into the IMM prediction. After association, each model is updated for the matched detection and the model probabilities are recomputed from the measurement likelihoods. The pairing is complementary by construction: the constant-velocity model estimates linear motion cheaply and well, and the constant-turn model covers the rotational case it cannot.

Life-cycle management handles the gap-versus-noise tension with two guardrails. An unmatched trajectory keeps predicting for a bounded number of frames before it is terminated, because an unmatched track usually means occlusion rather than disappearance, and restarting the trajectory later would fragment it. An unmatched detection has to persist for several consecutive frames before a track is born, which filters out occasional false positives. The two rules guard opposite failure modes.

The priors are where we departed from convention. Rather than setting the switching matrix and the covariances from experience, we treat them as an optimization problem and fit them with particle swarm optimization. The name invites a misreading worth clearing up: particle swarm optimization is a derivative-free search over parameters, it has nothing to do with particle filtering, and no particle filter appears anywhere in this work.

Two caveats on that fitting, and we would rather state them than have them discovered. The optimization runs offline and per dataset, so it is not online adaptation; the parameters are fitted to the data they were fitted on. And a symmetric switching matrix does not by itself tell you that vehicle motion in a dataset favors the constant-velocity model, even though it is tempting to read it that way.

View full-size image ↗

Bridging a missed detection is often cheaper than detecting it.

What we take from this

Latency budgets are a design input, not a measurement you take afterward. Once the real-time threshold is crossed, the remaining budget belongs to accuracy, and the architecture should be organized around that ordering.

A local receptive field is a cheap substitute for perfect calibration. When two sensors are aligned imperfectly, learning the correspondence in a neighbourhood beats assuming a pixel.

Reduce the dimensionality of the problem before optimizing the network. Pillarization is the enabling trick, not an afterthought.

Hybrid representations outperform pure ones here: learned appearance features, an explicit geometric projection, and an explicit state estimator, each doing what it is good at. The modularity also helps when something goes wrong, because each component has a distinct role and a distinct failure signature.

Complementarity beats tuning. Two models that fail in different regimes are better than one model that is mediocre in both.

And when the priors genuinely cannot be guessed, search for them. The admission that empirical IMM values may not transfer is the most useful methodological statement in the paper.

What it does not do

The honest summary is that the system is competitive in accuracy rather than leading. It improves substantially on detection speed and remains behind the best LiDAR-only methods on accuracy, which is the trade we chose.

The IMM-UKF has trouble with objects whose motion direction changes little. A filter that switches between hypotheses needs observable maneuver to be worth switching for.

Detection range is bounded by the camera field of view, which is inherent to this fusion design. Sparse distant objects fail, as the three-to-five-point example shows. The tracker is a separate stage from the detector rather than jointly optimized with it, which is the direction we point to for future work, along with improving accuracy without giving up the frame rate.

Closing

What we built integrates 3D detection and 3D multi-object tracking into a single pipeline measured against the real-time requirements of highway driving: a detector that produces oriented boxes efficiently, and a tracker that follows objects through uncertain motion using association, an interacting multiple model filter, and tuned parameters. The claim is not that it is the most accurate system available. It is that the accuracy is real at a frequency the fusion alternatives cannot reach, and that on a highway, being two metres behind the world is its own kind of inaccuracy.

Chang Nie

Teach and Grow: An Agent-Centered Architecture for General Robot Learning

A Few Demonstrations Become Skills That Stay

Where End-to-End Robot Learning Gets Expensive: The Retraining Tax

From Planning AI Agents to GPT-6 Centered Robotic Manipulation

Formal Modeling: Executable Capabilities as Explicit Skill Blocks

Subgoal Reasoning on Top, Metric Geometry and Control Underneath

1. Cross-Modal Demonstration Parsing and Subgoal Alignment

2. Scene Grounding and Metric Geometric Mapping

3. Sensor-Driven Physical Verification Contracts

Dual Memory Architecture: Skill Library and Experience Memory

1. Skill Library ($\mathcal{L}_{\text{skill}}$)

2. Experience Memory ($\mathcal{M}_{\text{exp}}$)

Lifelong Learning and Analytical Cost Scaling

Slow Teacher and Fast Student: Dual-System Synthesis

Empirical Benchmark Validation and Insights

Summary and Engineering Takeaways

Voxelized 3D Gaussian Representation for Dense Visual SLAM on Embedded Vision System

Dense Mapping and Storage Constraints

The redundancy is made by SLAM, not by Gaussians

Why online compaction is awkward

Fewer anchors, then fewer bits

Growing where the optimizer is still struggling

The mask is a dimmer, not a switch

Tracking depends on the map you just compacted

Evidence, and the shape of it

What we take from this

What it does not handle

Why embedded matters

MID: A Self-Supervised Multimodal Iterative Denoising Framework

The clean version often does not exist

Why the existing answers are conditional

What makes it hard

Restore by adding noise

Two networks, two different questions

Denoising is a way of framing a problem

How it is built

What we take from this

Where it stops

What kind of model this is

Towards the Vision-Sound-Language-Action Paradigm: The HEAR Framework for Sound-Centric Manipulation

Robots that cannot hear the physics

The cue is gone before the policy is asked

What earlier attempts leave out

Separate the sensing rate from the decision rate

The four modules

Where the data came from, since it did not exist

Rigorous Real-Robot Experiment Design: Why These Tasks?

1. Moka Pot Coffee Brewing: A True Barista Robot vs. Superficial Button-Pushing

2. Shaking a Bottle: Active Acoustic Probing

3. Real Alarm Clock: Resolving Temporal Aliasing Under Quasi-Static Vision

4. Pouring into Opaque Containers and Answering Phone Calls

Real-robot experiments

Core Insights and Engineering Takeaways

What It Cannot Do Yet

The Larger Frame

MRASfM: Multi-Camera Reconstruction and Aggregation through Structure-from-Motion in Driving Scenes

Driving is the wrong shape for photogrammetry

What prior methods assume

What makes it hard

The camera set is the unit

Building a reconstruction

Getting the road out of the way

Bundle adjustment on the rig

Merging drives that share no images

What we take from this

What it still assumes

Where it leads

MovSAM: A Single-image Moving Object Segmentation Framework Based on Deep Thinking

A photograph contains no motion

What earlier methods assume

What makes it hard

The mask as something you can argue with

Turning language into pixels: Cross-modal feature aggregation

The reasoning loop: Propose, segment, critique

What is and is not trained

What we take from this

What this opens

ERMV: Editing 4D Robotic Multi-view images to enhance embodied agents

Demonstrations Are Expensive, Yet We Already Have Them