Tianchen Deng | Chang Nie

Voxelized 3D Gaussian Representation for Dense Visual SLAM on Embedded Vision System

Wed, 13 May 2026 00:00:00 +0000

Dense Mapping and Storage Constraints

Sparse feature-based SLAM is accurate and its maps are too thin to be useful. Navigation, path planning and obstacle avoidance need surface information, not a cloud of keypoints, which is why dense mapping has been pursued for as long as it has.

The dense alternatives each carry a cost. Classical dense fusion systems such as DTAM, Kintinuous and ElasticFusion are memory-hungry and slow, and their accuracy suffers for it. Neural radiance field SLAM produces better geometry but carries an implicit decoder, which means rendering is a network evaluation rather than a rasterization. 3D Gaussian splatting looked like the resolution: real-time rendering with high fidelity, no decoder, no volumetric integration.

Then you try to deploy it. A Gaussian-splatting SLAM system typically needs more than 500 MB to represent a small room-sized scene. That number is the difference between a demonstration and something that runs on a robot, and it is where this work starts.

The redundancy is made by SLAM, not by Gaussians

The obvious conclusion would be that Gaussian primitives are simply expensive, and that the fix is compression. We do not think that is right, and the diagnosis is the part of this paper we would most want read.

We looked at the Gaussian ellipsoids that these systems create and found that their geometric similarity is far higher than in the original 3D Gaussian splatting. Measuring similarity between normalized, mean-free Gaussians reduces to comparing their covariance, which isolates shape and size from position, and the distribution we see in SLAM systems is a tall narrow spike compared with the broad low curve of the offline method. These systems are storing near-duplicates of the same ellipsoid, over and over.

The cause is not the primitive. It is the optimization schedule, and there are two structural differences that explain it.

Offline 3D Gaussian splatting receives every image at once. It optimizes across all of them jointly, and when it selects keyframes it samples randomly from the entire set. A SLAM system receives frames in timestamp order. It waits for each frame’s training to finish before moving to the next, and it only ever optimizes selected keyframes. Same primitive, sequential and locally scoped optimization, and a very different resulting distribution. The online regime has, to a certain extent, exacerbated the geometric similarity.

That distinction matters because it decides which existing tools are usable. The offline compression literature, including compact Gaussian representations, lightweight Gaussian pruning and self-organizing Gaussian grids, all assume you have every image, every pose and the corresponding point cloud available from the start. SLAM systems are incrementally optimized. Every one of those methods is disqualified by construction, however well it works offline.

View full-size image ↗

Anchors carry structure; per-view decoding generates Gaussian attributes.

Why online compaction is awkward

Three things make the problem harder than compressing a finished map.

Offline 3D Gaussian splatting seeds its Gaussians from a complete posed point cloud. In SLAM that cloud only exists for where the camera has already been, so the decision of where to add representational capacity has to be made from a partial map, online, without knowing what is coming.

Deletion has to be causal. You cannot consult the full training set to decide whether a Gaussian is still needed, because the future views have not arrived.

And opacity alone is the wrong criterion for removal. A Gaussian can carry low opacity and still be critical across multiple views, or occupy a useful volume while contributing little colour. Volume and opacity together are closer to what we want.

There is also a failure mode specific to learning the decision. Our mask loss pushes mask values down, which is what a sparsity prior does, and if you leave it running without interruption the optimization will eventually eliminate every Gaussian in the map. A one-directional regularizer with no reset destroys the thing it is regularizing.

Finally, compression here is not a post-processing step. In offline compression you quantize a finished artifact. Here the codebooks are learned jointly with attributes that are still being optimized, so the quantizer is chasing a moving target.

Fewer anchors, then fewer bits

Two layers of compaction, in that order.

The first is structural. Instead of storing Gaussians, we store anchors. An anchor is a voxel centre together with a small learned feature, a scale, and a set of offsets. The Gaussian centres are generated as the voxel centre plus each offset scaled by that anchor’s scale, and the remaining attributes, opacity, rotation, scale and colour, are decoded from the anchor feature together with the current viewing distance and direction. Structure carries the representation rather than individual primitives, and because decoding is view-conditional, detail is generated per view rather than stored per Gaussian.

The second is parametric, and it applies once the anchor count is under control. The offsets and scales are the geometry every anchor has to carry, so those are where the bits go. We compress them with cascaded residual vector quantization: the first codebook approximates the vector, and each subsequent stage quantizes only the residual the earlier stages left behind, with indices chosen by nearest-neighbour search against the running residual.

The analogy comes from neural audio coding, and it is a good one. The first codebook gives a rough description of the number. Each later codebook describes only what is left over. The codebooks are shared across the whole scene the way a palette is shared across an image. We initialize them with K-means.

Two tradeoffs are worth stating. More stages mean more residual levels to store and slower inference for gains that are only marginal. A larger codebook leaves many indices unused while still costing memory and training time, and a smaller one costs accuracy. The paper’s configuration uses six residual stages over a codebook of size 64, and the abbreviation is worth spelling out for readers who have met particle filters: residual vector quantization and particle swarm optimization have nothing to do with each other, and no particle filter appears anywhere in this work.

View full-size image ↗

Each stage quantizes only what the previous stages left behind.

Growing where the optimizer is still struggling

With no structure-from-motion cloud to seed from, the question of where to put capacity has to be answered online. The signal we use is the gradient.

A voxel whose constituent Gaussians keep producing large averaged gradients over a run of iterations is a place the optimizer is still fighting, which is a reasonable indication that the current representation cannot express what the data wants. If that averaged gradient exceeds a threshold and no anchor exists at that voxel centre yet, we plant one. Candidates are eliminated randomly to keep growth from exploding, and trivial anchors get removed by accumulating the opacity of their Gaussians.

Three resolutions run in parallel, with voxel size shrinking by a factor of four at each level and the gradient threshold doubling. Coarse voxels absorb bulk geometry such as walls and floors. Fine voxels handle detail such as a chair leg. A single voxel size would force one choice to serve both, which is why the ladder exists.

Initialization is separate: at map start, points are sampled where accumulated opacity is low or where the rendered depth disagrees with the measurement.

The mask is a dimmer, not a switch

Inside a sliding window, each Gaussian carries a learnable mask parameter, which a stop-gradient straight-through form turns into a binary mask. That mask multiplies the Gaussian’s scale and opacity before rendering, so a masked primitive fades out of the image rather than being deleted from the data structure. Removal happens later, at the window boundary, where we use hit frequency together with the learned mask.

The window is short-term memory with a defined composition: the current frame, the keyframe it overlaps most, and the two earlier keyframes with the highest frustum overlap with the current depth point cloud. Everything outside the current view is frozen by frustum culling, which preserves past geometry and reduces the number of parameters being optimized. Masking runs on keyframes only.

The reset is the detail that makes the mechanism safe. At the window boundary, once removals are applied, both the gradients and the masks are cleared so the system can relearn. Without that reset, the sparsity pressure described earlier runs unopposed and the map erases itself.

Window size is a genuine tradeoff rather than a hyperparameter to tune casually. Too large a window masks away Gaussians that were doing useful work, which costs accuracy. Too small a window costs memory and optimization quality.

View full-size image ↗

Masking modulates scale and opacity; removal happens at the window boundary, after which the masks and gradients are reset.

Tracking depends on the map you just compacted

Three signals drive tracking, and all of them come out of the same alpha-compositing pass. Colour, depth, and a silhouette channel that accumulates only the visibility weights. Because they share the rasterization, the silhouette is nearly free.

Tracking minimizes squared colour error plus squared depth error over rays with valid depth, gated by the silhouette: only pixels where the renderer is confident it is actually covering something are trusted. An ICP term ties Gaussian centres to their nearest points across frames, on the reasoning that Gaussians are a point cloud and classical geometric association therefore applies to them. Pose is initialized by constant-velocity forward projection, which the ablation calls vital.

The ablations here are the clearest argument that these components are load-bearing. With only the depth term, tracking fails outright. With only colour, tracking works at substantially worse error. Without the silhouette gate, tracking fails outright. A renderer-derived trust region turns out to be doing real work, and a photometric pipeline apparently needs both a geometry term and a region it can believe.

Bundle adjustment runs at two scales. Local adjustment refines the recent window, and a global stage samples rays from a deliberately larger keyframe database to refine map and poses together, with an additional structural similarity term. The purpose is to bound accumulated drift over long trajectories and large scenes without touching every historical frame.

Evidence, and the shape of it

The system is evaluated on Replica, ScanNet and TUM RGB-D, on an embedded module and a laptop-class GPU, and on a multisensor mobile-robot dataset we collected ourselves with LiDAR, an inertial unit and a depth camera, covering scenes over a thousand square metres with high-quality trajectory ground truth. The dataset exists to answer a question the standard benchmarks cannot: whether the representation holds up when the map is being built on a moving robot rather than on a desk.

The most useful evaluation is not a table. We inserted the compact representation into SplaTAM, MonoGS and Gaussian-SLAM and re-ran them. The same direction of improvement appeared in each. That is evidence about the diagnosis rather than about our system, since a representation that helps three different systems is addressing something those systems share.

View full-size image ↗

Compactness is what makes on-robot deployment possible.

What we take from this

Diagnose the representation you inherited from the training regime before you build a compressor. The interesting move here was not designing a better quantizer. It was noticing that sequential online optimization produces a different and more redundant distribution than joint optimization, and measuring it before writing any code.

Related to that, any module that needs the entire dataset is disqualified by construction in an incremental system, no matter how well it performs offline. That single constraint eliminates most of the available literature.

Compact in two layers, structural and then parametric. Quantizing alone leaves millions of primitives to encode. Masking alone leaves a large per-primitive payload.

Gradients are a capacity-allocation signal, not just an optimization artifact, and using them that way turns where should the map be detailed into a decision the system makes for itself.

And any learned pruning pressure that only pushes in one direction needs a reset. That pattern generalizes well beyond this paper.

What it does not handle

Fast motion produces motion blur in both the colour and depth images, and tracking depends on exactly those losses, so rapid movement degrades it. Scenes with many dynamic objects produce artifacts and drift, and the system is still limited in dealing with highly dynamic environments.

Loop closure and relocalization are not implemented, which bounds how well drift can be corrected on large loops. The system is RGB-D only, so there is no monocular or stereo path. And we owe the reader a clearer account of the reported memory figure: it is checkpoint size rather than peak memory footprint, and those are different numbers.

Why embedded matters

The paper’s claim is not that a compact map is an acceptable compromise. It is that a compact map is still a good map, and the same quality is available at a fraction of the storage. That claim only has teeth in a setting where the storage is a real constraint, which is why the embedded experiments are the ones we care about. A representation that fits on the robot is the one that gets used.

MID: A Self-Supervised Multimodal Iterative Denoising Framework

Thu, 30 Apr 2026 00:00:00 +0000

The clean version often does not exist

Good data is the precondition for everything downstream, and acquisition noise is nearly universal: sensor limitations, transmission errors, environmental interference, and the stochastic nature of physical measurement itself. The usual response is to denoise first.

Two practical facts make that harder than it sounds, and both shaped this paper.

The first is that the noise that actually matters is rarely Gaussian. Poisson noise in low-light imaging, compression artifacts, and what we would call structured noise, such as a set of line segments that do not correspond to any real structure in the scene, all violate the assumptions that classical filters are built on. The second is more fundamental: for many of the domains we care about, the clean target simply does not exist. In clinical imaging and remote sensing there is frequently no noise-free reference to compare against, and no way to acquire one. The MRI experiments in the paper are scored with proxy measures precisely because clean ground truth is unavailable for those datasets.

So the training signal has to come from somewhere other than a paired clean sample. That constraint is the whole problem.

Why the existing answers are conditional

Classical model-based denoising, from filtering to transform-domain thresholding to non-local means, is tailored to specific noise models and data types. Its effectiveness falls away as soon as the noise statistics deviate from the assumption it was built on.

Supervised deep denoising sidesteps the modelling assumption but needs large paired datasets of clean and noisy samples, which are costly or infeasible to collect in exactly the domains where denoising matters most.

Self-supervised blind-spot methods were the natural answer, and the criticism we make of them is specific rather than general. They avoid pairing, and they carry hidden modality assumptions. The tricks that make a blind-spot network work, such as masking pixels and predicting them from neighbours, rely on spatial correlation in images. Point clouds are unordered, and amino acid sequences are discrete and unstructured. A method that assumes nothing about the data can travel to those settings. One that quietly assumes spatial smoothness cannot, and the observable symptoms are oversmoothing and loss of fine detail.

Domain-specific specialists have their own conditionalities. Noise2Noise can oversmooth fine anatomical structures. Patch2Self exploits redundancy across volumes but requires training a separate model per volume, which risks spatial inconsistency between them. DDM2 is sensitive to the initial noise level and to modality-specific tuning. In surface electromyography, high-pass filtering removes the ECG interference and removes important low-frequency components of the signal along with it, because the two overlap in frequency; template subtraction assumes a rigid distributional form; and neural approaches can be unstable to train. For protein sequence curation, the standard greedy selection on average Hamming distance rests on a single criterion that does not capture sequence quality more broadly.

We summarize the positioning as a capability matrix rather than a benchmark table. The question is which methods are self-supervised and work on images, signals and point sets at once. Most of the field manages two of the three.

What makes it hard

The inverse of a nonlinear corruption has no usable closed form. Reversing Poisson noise, compression, or a set-structured nuisance directly is difficult, and our own ablation shows what happens if you try: a single-pass network leaves the data essentially unchanged. Networks struggle to learn a large jump from noisy to clean in one step. That result is not an implementation failure. It is the reason the method is shaped the way it is.

You also do not know how corrupted the input is. Nothing about a noisy sample announces its position on a corruption trajectory, so there is no way to decide how much to remove or how many steps to take.

Self-supervision has to be manufactured rather than found. Without pairs, the only available labels are ones you create yourself, and they have to serve two purposes: how far along the corruption the input sits, and what the next increment looks like.

And sometimes the noise is not a value at all. An incorrect correspondence, an invalid line segment, or a redundant sequence is not a magnitude added to anything. It is a decision about an element. Before any of that can be subtracted, the nuisance has to be recast as a per-element property, which is why those tasks use a binary classification objective rather than a regression on residuals.

Unordered and discrete data break the usual priors outright. A point cloud has no ordering and a multiple sequence alignment is a set of discrete strings, so weight sharing over a spatial grid and convolution are simply unavailable.

There is also a tradeoff baked into the formulation. The first-order approximation underneath our method only holds if the increments are small. Small increments buy linearity, and then cost you a long inference loop.

Restore by adding noise

The idea we started from is counterintuitive enough to state plainly. To learn how to remove noise, you first add more of it.

A noisy observation is not a degraded version of a clean sample sitting off to one side. It is one state on a corruption trajectory, and you can continue that trajectory yourself. Take the noisy input as the starting point, apply the corruption process again and again, and you now have a labelled ladder: at every rung you know exactly how much noise separates it from the rung below, because you put it there.

The move that makes this usable is a local linearization. Writing the corruption as a general operator and expanding to first order,

$$ s_t \approx s_{t-1} + \Delta s_{t-1}. $$

The step from one rung to the next is treated as additive, which means its inverse is a subtraction. The residual the noise network is asked to predict is nothing more than the difference between two consecutive states that we generated. This is an approximation, and we say so: it implicitly assumes the Jacobian is close to identity, or that its effect is absorbed into the learned mapping.

Everything else follows from that. The reverse step becomes tractable, and the training signal is free.

Two networks, two different questions

The framework asks two separate questions, and keeping them separate is what makes one recipe portable across domains.

The first is: where am I? A step prediction network regresses the input’s position on the trajectory, with the step normalized and a mean-squared objective. This is necessary before any amount of subtraction can be chosen, and it is where the method differs most from diffusion. Diffusion trains against a noise prediction derived from a fixed forward schedule. Here the step is estimated from the data itself, without assuming a fixed forward process.

The second question is: what should be removed? The noise prediction network takes the current state and the estimated step and predicts the increment. The two networks are trained jointly, and the decoupling is deliberate: one decides how far to walk back, the other decides what to take off at each step.

At inference the loop is a few lines. Estimate the current step, then subtract predicted increments until the step count reaches zero. A lightly corrupted input takes fewer steps than a heavily corrupted one, so the iteration count adapts to the data without a schedule.

The reason for iterating at all, rather than removing everything at once, is that many small corrections preserve detail better than one large one. Oversmoothing is what happens when a single pass has to guess the whole distance.

View full-size image ↗

A step predictor and a noise predictor are trained jointly and used in sequence at inference.

View full-size image ↗

The reverse self-supervised restoration loop walks a corrupted input back down the trajectory one increment at a time.

Denoising is a way of framing a problem

The most interesting part of this work, for us, is not the image results. It is what happens when the same formulation is pointed at problems that nobody would call denoising.

Image correspondences become a four-dimensional point cloud, where a match is a point with two image coordinates in each view. Wrong matches are points that do not belong to the structure, and the network learns the broader features of incorrect correspondences rather than relying on a match score alone. Line segments used for vanishing point estimation become an unordered point cloud, and the model is trained by iteratively adding random segments as noise. Multiple sequence alignments are handled by treating redundant sequences with low average Hamming distance as noise. In that last case, the noise consists of real amino acid sequences, which is a fair demonstration that the framework is evaluating data quality rather than a synthetic artifact.

What this shows is that denoising is a task framing rather than a data type. Once the nuisance is expressed as something removable, outlier rejection in geometric estimation, correspondence pruning, line filtering and sequence subsampling turn out to be the same problem wearing different clothes.

Two design details follow from the same insight. Multiplicative noise can be made additive in the log domain, which the paper notes and uses. And for point-set tasks the noise objective is a per-element binary cross-entropy, not an additional penalty term on top of a regression. The network is not asked how much to subtract from a wrong correspondence. It is asked whether that element should be there at all.

View full-size image ↗

The same formulation that removes image noise also identifies incorrect matches.

How it is built

For images and medical volumes, a CNN backbone handles the step predictor with fully connected layers on top, and a CNN encoder-decoder predicts the increments. For point sets and one-dimensional signals, including correspondences, line segments, amino acid sequences and surface EMG, both networks use fully connected layers with a Transformer encoder-decoder. The reason is the one given above: convolutions assume a spatial ordering that unordered sets do not have, so the architecture changes with the data rather than the other way around.

None of the domains get a bespoke formulation. That was the point of building it this way. One method, applied to image speckle, MRI noise, muscle-signal contamination, geometric outliers, wrong correspondences, invalid line segments, and redundant protein sequences, with only the backbone changing.

View full-size image ↗

The nuisance here consists of real sequences rather than synthetic artifacts.

What we take from this

Supervision can be manufactured by continuing a corruption you can simulate, rather than by obtaining the clean endpoint. That single substitution is what makes the framework applicable where paired data does not exist.

Local linearization plus iteration converts an intractable global inverse into a sequence of tractable local ones, and the ablation is the evidence. When the single-pass model fails, the right response is not a larger model. It is decomposition.

Separating where am I from what should be removed is what makes one framework portable. That decoupling, more than any architectural choice, is the reusable part.

Self-supervision strategies carry hidden modality assumptions, and it is worth auditing them. A formulation that asserts nothing about the structure of the data will travel further than one that quietly requires smoothness.

And iteration is a detail-preservation mechanism, not just an optimization convenience.

Where it stops

Extreme noise is the stated failure mode. When the signal structure is almost entirely obscured, iterative subtraction has too little partial structure to work from, and generative priors are the direction we point to for future work.

The second cost is time. Iterating is inherently slower than a single-pass denoiser, which is a bottleneck for real-time use. Knowledge distillation and more efficient sampling are the remedies we suggest.

Two things are visible in the method that we did not fully characterize. The number of corruption steps and the magnitude of each increment are design choices rather than derived quantities, and we did not study the sensitivity to them. Errors in the estimated step also compound across the subtractions that follow, and that propagation deserves more attention than we gave it.

What kind of model this is

It is worth being explicit about one boundary, because it comes up in every conversation about this work. MID is not a generative model. Diffusion models minimize a noise prediction loss derived from a fixed forward schedule and can sample new data from the learned distribution. MID is an iterative, self-supervised noise subtractor that learns the structure of the noise present in its target domain and removes it. The output is a cleaner version of the input, not a new sample.

That distinction is what keeps the framework honest about what it can promise. It removes what it was trained to recognize. Anything else is a different problem.

Towards the Vision-Sound-Language-Action Paradigm: The HEAR Framework for Sound-Centric Manipulation

Tue, 17 Mar 2026 03:22:30 +0000

Robots that cannot hear the physics

Current vision-language-action models see, understand language, and move. What they do not do is listen. They operate in what amounts to a silent world, even though the robot’s own environment is full of sound that encodes physical state the cameras cannot recover: liquid starting to sputter as it boils, the sharp click of a collision, the difference in timbre between a full container and an empty one, the moment a material stops flexing and starts cracking.

Contact acoustics are the cheapest tactile sensor available. A microphone is a few dollars, it needs no contact, and it covers the whole workspace at once. The case for using it is not speculative. The problem is that nobody has built a policy that can.

The cue is gone before the policy is asked

The difficulty is not a modelling problem, and framing it that way is why earlier approaches have not solved it. It comes from the control architecture that makes large VLA models practical.

A large backbone cannot run at control frequency. So the standard design chunks actions: the model produces a short sequence of future commands, the robot executes them open-loop, and only then does the model get queried again. That gap is where the physics happens. A collision, a boil-over, a completed press: these are brief, non-repeatable acoustic events, roughly on the order of a hundred milliseconds, and they routinely fall entirely inside the interval where the policy is not looking. The average chunk at 30 Hz is about a second long.

The consequence is worth stating precisely, because it is not what people usually assume. A late reaction would be a delay. What actually happens is that the evidence is destroyed. The audio arrives, nothing is listening, and by the time the policy is queried again, the window contains only silence. If an event occupies samples that end before the next window opens, the window does not contain a weak version of that event. It contains nothing. We call the interval between decisions the Blind Execution Interval, and the word blind is doing real work.

There is a second failure that has nothing to do with timing. Look at a robot that has been waiting beside a running appliance. If the scene is quasi-static, the current image looks the same whether it has been waiting one second or ten. The expert demonstrations differ, because the right next action depends on elapsed time, but the observation does not. A snapshot policy has no way to tell those situations apart, so it either drifts or freezes. This is the aliasing problem that temporal context is supposed to solve, and without audio it is genuinely unobservable.

What earlier attempts leave out

Turning audio into text is the obvious first instinct, and it loses the thing we need. Speech recognition front ends discard non-speech cues and prosody entirely, and they compress timing into discrete tokens. A beep produces nothing at all. More subtly, an affirmative “Yes!” and a doubtful “Yes?” collapse into nearly identical token sequences, so the difference in pragmatic intent disappears while transcription accuracy looks perfect.

Rendering the waveform as an image and treating it as another camera view is the second instinct. It captures amplitude changes, and it obscures spectral structure: frequency contours and timbre, which is where material and contact information live. It is also sensitive to where the event happens to land inside the rendered window, so performance changes for reasons that have nothing to do with the physics.

A more recent line adds speech to VLA models and preserves prosody and speaker identity. That is real progress, and it targets explicit verbal commands rather than the ambient acoustic physics of manipulation, which is a different problem. The omni-modal systems are closer still, and the reservation is structural: they are built to listen before acting, using sound to establish context or infer intent around an action, rather than as low-level feedback during contact. None of them address the misalignment between transient events and chunked execution, which is the actual bottleneck.

Audio foundation models such as CLAP and ImageBind were trained on fixed-length clips, offline and independently. Deployed statelessly, they carry no temporal context across clips. Compact audio-native policies run fast and can handle contact, but their scale makes long-horizon, multi-stage behavior brittle. Processing audio natively turns out to be necessary and not sufficient.

The shortcut worth naming is faster replanning. Truncate the chunk, query more often, and the blind interval shrinks. We tried the variants. It helps less than it looks like it should, and it pays in motion quality: discontinuous, jerky trajectories that introduce localized action loops. It also cannot fix system latency, which no query rate eliminates. A cue lost to latency is lost regardless of how often you ask.

And the benchmarks are no help, because every standard simulator is silent. RLBench and ManiSkill test spatial reasoning and visual control in environments with no acoustic channel at all. Navigation benchmarks that do use sound let the agent move its base to improve its listening geometry, which is not an option for a manipulator that has to keep working while generating its own mechanical noise.

Separate the sensing rate from the decision rate

Our answer comes from noticing that this is a rate mismatch, not an information deficit. The sound is there. The policy is simply asked too rarely to catch it.

So instead of making the policy faster, we stop tying the two rates together. A persistent memory is updated continuously from the audio stream, packet by packet, independently of when the policy is queried. When a decision is finally requested, the memory already contains everything that arrived, including events that happened during blind intervals. The robot is not reacting to the last window. It is reacting to the last second of listening, with the recent history summarized.

That is the core move, and everything else in the architecture is bookkeeping around it. There is something that remembers, something that interprets, something that predicts what comes next acoustically, and something that acts. We named them accordingly.

Two constraints shaped the design. The first is that the audio path has to be causal in the strict sense: nothing may look at samples that have not arrived, and the tokenizer must produce codes for a segment from that segment alone. The second is that the memory horizon has to be a target, not a hope. It needs to cover the worst-case decision gap plus the measured system latency, otherwise the guarantee is only statistical.

The four modules

Historizer. Between two decision boundaries, the robot may receive a long run of consecutive audio packets, each about forty milliseconds. The Historizer consumes them as they arrive and updates a streaming stateful Transformer memory. The memory is read out at decision times and never otherwise. Its entire justification is the evidence-vanishing argument above: a window-only interface cannot represent something that happened just before the window opened. We also tried a causal GRU and exponential moving average pooling over audio features; the streaming attention’s extra capacity earned its place. We chose a short window with a persistent summary over a very long window, because long windows mix distinct interaction phases together and make the result sensitive to exactly where the window boundary falls.

Envisioner. Interpretation is hierarchical, because the two jobs have very different frequencies. A high-level model reads the multi-view images, the current acoustic window, the instruction, proprioception and the memory, and produces three things: a semantic latent, a key-value cache, and a short structured description of the current task stage, something like {"stage": "wait", "subgoal": "listen for beep"}. That stage text comes from the high-level model’s own text head. A smaller low-level model then reuses the cache with the current robot state to produce the control feature. The split keeps expensive semantic interpretation out of the fast path. The stage supervision matters more than it might appear: without it, tasks that require a quick behavioral shift suffer, because nothing forces the latent to name the phase it is in, and the phases that matter acoustically are often visually identical.

Advancer. This one predicts the audio that will arrive between the current decision and the next one, as discrete codes from the streaming tokenizer, trained by cross-entropy. It is used only during training and is dropped at deployment, so it costs nothing at inference time. Its purpose is to give the latent a concrete sense of elapsed time and process progression. Pouring, boiling, an alarm, a conversation: each has a different acoustic future, and predicting it forces the shared representation to encode where in the process the robot currently is. Without it, the failures look different from the Historizer’s. They happen late. The robot keeps pouring after the sound has already told it to stop. That asymmetry is how we knew the two modules were solving different problems rather than duplicating each other.

Realizer. Actions are generated by conditional flow matching from a Gaussian prior to the action chunk, integrated in a handful of Euler steps, conditioned on the control feature through adaptive normalization. We used flow matching rather than diffusion for the step count. The detail worth dwelling on is why smoothness matters here. Replacing the Realizer with direct regression produces jitter, and jitter injects mechanical self-noise into the microphone signal, which masks the brief cues the rest of the system is trying to hear. A perception failure caused by a control choice. The sharpest version of the claim is that a policy which listens while moving has to move in a way that makes listening possible. Sensing and control are not separable in this setting.

View full-size image ↗

A high-level model produces semantics and task stage; a low-level model produces the control feature.

View full-size image ↗

The Advancer is used only during training and is removed at deployment.

The three objectives are summed with weights that ramp in over the first part of fine-tuning, so the model learns imitation before it is asked to be temporally grounded.

Where the data came from, since it did not exist

No robot dataset contains synchronized audio. Open X-Embodiment has no microphones at all, real synchronized audio is noisy and hardware-specific and does not scale across labs, and no standard simulator models acoustics.

We built both pieces. OpenX-Sound retrofits audio onto existing episodes from video, preserving every original signal, and we audited a sample of it by hand for synchronization and semantic relevance. We are explicit that this is a bootstrapping mechanism for representation learning and not a substitute for physical data: synthesized audio can omit subtle contact cues, shift event timing, or hallucinate a sound with no physical source. Every result in the paper fine-tunes an OpenX-Sound checkpoint on real task-specific demonstrations with real audio. The synthesized audio is what makes a pretrained acoustic prior possible, not what makes the behavior work.

HEAR-Bench is the other half. Built on a dual-arm simulator, it runs an asynchronous audio thread that injects ambient noise and trigger clips at the exact sample index implied by the control cycle, and it scores success only if the geometric goal is reached after the cue. That timing rule is the whole contribution. Without it, pressing a button the instant the episode starts and pressing it when the alarm actually rings look identical.

Rigorous Real-Robot Experiment Design: Why These Tasks?

When validating the acoustic manipulation capabilities of end-to-end multimodal Vision-Language-Action (VLA) foundation models, we deliberately avoided arbitrary toy benchmarks. Instead, we designed each real-world robotic evaluation specifically targeting the fundamental physical blind spots of vision-only policies:

1. Moka Pot Coffee Brewing: A True Barista Robot vs. Superficial Button-Pushing

The vast majority of existing demonstrations claiming “robot coffee brewing” merely involve a mechanical arm approaching a commercial capsule or fully automated pod coffee machine, executing an open-loop press on a static plastic button, and idly waiting while internal integrated circuits run a blind timer. Such setups require zero perception of the underlying physical state—they are glorified mechanical switch activators, indifferent to fluid dynamics, thermal phase changes, or culinary progression.

In contrast, our setup places an authentic dual-chamber metal Italian moka pot on an active electric cooktop. The moka pot consists of fully opaque metal walls; the boiling pressurized water in the boiler, the compacted coffee puck in the funnel filter, and the ascending extraction column inside the collector chimney remain entirely concealed from view. External RGB cameras and infrared depth sensors cannot penetrate the metal shell to observe fluid levels or thermodynamic transitions.

A seasoned master barista relies neither on stopwatches nor visual inspection; extraction cutoff is governed purely by ear:

Silent Heating Phase: Water temperature climbs gradually with negligible acoustic emission beyond faint thermal conduction;
Smooth Boiling & Laminar Extraction Phase: Saturated steam pressure forces water smoothly through the coffee puck, producing a continuous, gentle, muffled bubbling sound;
Violent Sputtering Phase (Cavitation & Steam Eruption): As boiler water nears depletion, high-temperature steam mixes with residual liquid to produce a violent, unmistakable high-frequency sputtering and hissing eruption through the extraction chimney.

This acoustic inflection point is the decisive boundary in coffee extraction: removing the pot one second too early yields sour, under-extracted, watery coffee; leaving it on the heat two seconds too late causes dry-boil scorching, bitterness, and permanent thermal damage to the moka pot base.

HEAR functions as a genuine barista robot: powered by the end-to-end VLA foundation model, it continuously analyzes high-frequency spectral transitions in real time. The moment the distinctive micro-sputtering signature emerges, the policy issues action chunks with millisecond-level responsiveness, smoothly lifting the moka pot away from the heating burner.

View full-size image ↗

Contrasting superficial open-loop button pressing with HEAR’s acoustic monitoring of internal thermodynamic phase transitions inside an opaque metal moka pot.

2. Shaking a Bottle: Active Acoustic Probing

In everyday physical environments, numerous pill bottles, opaque jars, and enclosed canisters conceal their contents entirely. Passive vision fails fundamentally in such settings—no matter how many camera viewpoints or lighting angles are deployed, the exterior surface remains completely static.

To overcome this, the robot must exhibit Active Exploration: it must grasp and shake the container to inject mechanical energy, exciting internal collisions. By decoding the resonant damping, impact frequencies, and high-frequency collision timbres of the internal objects against the container walls, the model disambiguates the presence, material properties, and fill level of the enclosed items. In our real-robot demonstration, we deploy an occupied container filled with loose objects, showing how the robot actively shakes the bottle, decodes transient collision acoustics, and executes informed downstream classification.

3. Real Alarm Clock: Resolving Temporal Aliasing Under Quasi-Static Vision

An alarm clock in its silent standby state and active ringing state can appear visually indistinguishable. Lacking acoustic feedback, naive end-to-end policies succumb to shortcut learning, prematurely hitting the stop button at the very start of the episode. Our benchmark enforces that the stop button must be actuated only after the acoustic trigger chime sounds, directly probing whether the multimodal VLA accurately aligns actions with transient acoustic timing cues.

4. Pouring into Opaque Containers and Answering Phone Calls

Pouring water into an opaque cup represents the canonical scenario where sound is the sole observable state signal. Fluid levels are strictly occluded; the true decision variable is the continuous frequency drift rate of the pouring stream resonating within the shrinking acoustic cavity. Similarly, answering a phone call assesses multi-stage temporal state machines: the robot approaches upon hearing the ringing chime, remains stationary during the caller’s acoustic pauses, and docks the receiver only when the terminal dial tone registers.

Real-robot experiments

All demonstrations include their recorded audio. Select a video to play it.

Moka CoffeeThe robot listens for the acoustic sputtering phase inside the opaque pot before removing it from heat.

Answer PhoneRinging, speech, and the end of a call distinguish visually similar task stages.

Shake BottleThe robot actively shakes an opaque container, discerning internal contents from collision acoustics.

Real Alarm ClockThe robot waits for the alarm before pressing the stop button.

Core Insights and Engineering Takeaways

Integrating high-frequency continuous acoustic feedback into end-to-end embodied foundation models (VLA) demands far more than appending an extra feature dimension to the visual tokenizer. Conventional paradigms treat sound as an auxiliary camera stream, yet the physics of sound is intrinsically event-driven, high-frequency, and transient. Grounding it in robotic action mandates resolving the fundamental architectural tension between continuous acoustic cadence and chunked control loops.

This structural tension extends well beyond sound. Any physical contact modality that generates rapid, non-repeatable events faster than the policy’s replanning frequency—most notably high-frequency tactile arrays and micro-vibrations—encounters identical rate-mismatch bottlenecks. This asymmetry cannot be bypassed by merely optimizing the inference throughput of large foundation models.

Artificially increasing the model’s query rate fails for structural reasons rather than tuning deficiencies: truncating action chunks trades long-horizon trajectory smoothness for erratic reactivity, injecting severe mechanical jitter and self-noise directly into the sensory stream. The only robust solution is decoupling perception cadence from decision loops via explicit streaming causal memory. Auxiliary acoustic future prediction objectives ground shared representations in physical time at zero inference overhead, while rigorously timed causal benchmarks transform “appearing visually plausible” into “authentically listening to the physical world.”

What It Cannot Do Yet

One workspace microphone means no source localization and no separation of overlapping events, which is functionally the cocktail party problem. We would want an array, or a base-plus-end-effector pair for beamforming. The system is trained purely by offline imitation, so its robustness is bounded by the diversity of noise and timing variation in the demonstrations; online fine-tuning or reinforcement learning in the deployed room is the obvious remedy. Audio and vision are processed as separate streams and fused at the representation level, and we suspect native audio-visual encoders that associate timbre with contact directly would do better. And unconstrained real-world cooking acoustics remain an open problem, with real deployment markedly harder than simulation.

The Larger Frame

The tension we started from, between high-frequency transient sensory events and low-frequency delayed policy updates, is not specific to sound. It will show up for every physical modality that carries information in its timing. HEAR is our attempt to show that the fix is architectural, and to leave behind the pieces—the VSLA formulation, the pretraining corpus, and the benchmark—that let other researchers build further on this foundation.