Embodied Intelligence | Chang Nie

Teach and Grow: An Agent-Centered Architecture for General Robot Learning

Mon, 17 Aug 2026 23:45:21 +0000

A Few Demonstrations Become Skills That Stay

Teach-and-Grow Learning (TGL) establishes an AI Agent and GPT-6 centered robotic manipulation architecture for general-purpose physical autonomy. Operating under the principle that pretrained foundation model weights remain entirely frozen, the framework places GPT-6 at the center of the embodied reasoning loop. The surrounding AI Agent interprets multi-modal spatial cues, decomposes complex long-horizon tasks, grounds high-level semantics into metric SE(3) geometries, and validates physical state transitions before accumulating verified skills into persistent dual-memory repositories.

Teach → Act and verify → Grow. Sparse demonstrations provide structural subgoal decompositions and causal dependencies; real-time physical feedback governs closed-loop execution and error recovery; and persistent, decoupled dual memory transforms the verified takeaways of preceding tasks into immediate springboards for subsequent unfamiliar missions, entirely eliminating the dependency on perpetual fine-tuning and weight optimization.

· ·

Where End-to-End Robot Learning Gets Expensive: The Retraining Tax

Contemporary research in deep imitation learning and end-to-end Vision-Language-Action (VLA) models relies primarily on gradient descent across high-dimensional parameter spaces $\theta \in \mathbb{R}^D$. When deploying these monolithic models into real-world, unstructured operational settings, however, a profound structural dilemma arises:

Whenever a robot encounters a novel container geometry, an uncalibrated camera perspective, highly specular surface finishes, or minor grasp slips, the actual metric variation may be minor. Yet, within a monolithic parameter-space paradigm, remediating this isolated deficiency typically necessitates collecting dozens or hundreds of additional expert trajectories, followed by computationally intensive joint multi-task optimization across the entire legacy training corpus. Worse still, because deep neural representations are heavily entangled across latent feature dimensions, adjusting parameter weights to accommodate new tasks frequently provokes unconstrained representation drift, causing previously mastered competencies to degrade unpredictably. We formalize this recurring computational, data collection, and verification overhead as the Retraining Tax.

Physical interaction data possesses a fundamentally distinct economic cost structure compared to internet-scale textual tokens or static web images. Physical contact cannot be synthetically generated losslessly via naive auto-regressive sampling; grasp stability must be empirically tested within a 3D dynamical workspace; and state changes in support relationships and surface friction must be validated through physical sensors. Geometric poses, background clutter, frictional coefficients, and manipulator nonlinear dynamics interact continuously. Covering the independent marginal distributions of isolated physical factors is fundamentally insufficient to cover their high-dimensional joint combinatorial distributions.

TGL addresses this foundational bottleneck: Can we decouple the acquisition of new robotic tasks from parameter-space gradient updates? Can we preserve physically validated manipulation experiences as explicit, inspectable objects endowed with verifiable geometric applicability bounds, deterministic preconditions/postconditions, and modular error-recovery rules? Under this paradigm, robotic foundation models centered on GPT-6 within an AI Agent serve as zero-shot semantic reasoning and geometric perception backbones, while TGL restructures how task knowledge is stored, audited, and retrieved within the robot system.

View full-size image ↗

Figure 1. Policy optimization and explicit skill acquisition, illustrated through the same unfamiliar manipulation task.

From Planning AI Agents to GPT-6 Centered Robotic Manipulation

This research originated from investigating how tool-using AI Agent architectures could directly actuate robot manipulators to accomplish open-ended manipulation tasks. Placing GPT-6 at the core of the embodied agent provides unprecedented physical causality modeling and world common-sense reasoning. Paired with open-vocabulary perception, 6D pose estimation, and kinematic motion planning tools, a GPT-6 centered AI Agent can assemble feasible action sequences through exploratory trial-and-error even in the complete absence of prior demonstrations.

However, relying exclusively on instantaneous agent deliberation exposes severe operational limitations: once an episode terminates and the object is placed, the system retains no persistent, deterministic knowledge. When confronted with an identical or closely related task in subsequent trials, the agent must invoke slow, high-latency language model inference anew, searching through vast combinatorial action spaces from scratch. Furthermore, physical hurdles encountered during previous runs (such as slippery rims on thin glassware or localized mechanical binding in drawer rails) are discarded alongside the context window.

TGL integrates GPT-6 into the AI Agent cognitive center, retaining the agile observe-reason-act-verify loop while coupling it to an evolving lifelong memory architecture. Successful task executions are consolidated into formal Skill Blocks governed by rigorous execution contracts, while unexpected execution faults, root-cause diagnostics, and successful recovery sequences are archived as contextual exemplars within Experience Memory. In downstream tasks, the robot prioritizes retrieving verified, high-confidence skills, invoking full agent deliberation only when genuine out-of-distribution variations occur.

Within this framework, sparse demonstrations (human teleoperation logs, simulated trajectories, or instructional procedural text) function as an accelerator rather than an immutable prerequisite. Demonstrations supply a macroscopic temporal sequence of subgoals, confining exploratory search to critical geometric transitions and slashing trial-and-error overhead.

Formal Modeling: Executable Capabilities as Explicit Skill Blocks

To isolate invariant behavioral strategies from transient physical variations specific to a single environment, TGL introduces the formal abstraction of the Skill Block. Consider the task of picking up a ceramic bowl and placing it onto an adjacent tray. Across diverse demonstrations, the manipulator’s Cartesian trajectories, joint velocities, and approach angles vary widely. However, the underlying causal structure remains invariant: acquire the designated bowl, establish a stable grasp, transfer toward the target placement relation, and compliantly release contact.

Mathematically, we formalize a Skill Block $S_k$ as a 6-tuple:

$$S_k = \langle \mathcal{G}_k, \mathcal{P}_{\text{pre}}, \mathcal{P}_{\text{post}}, \Pi_{\text{exec}}, \mathcal{V}_{\text{eval}}, \mathcal{R}_{\text{recov}} \rangle$$

Where:

$\mathcal{G}_k$ denotes the semantic and geometric subgoal targeted by the skill;
$\mathcal{P}_{\text{pre}}$ represents the explicit precondition manifold required for skill triggering (e.g., target object kinematically reachable within the workspace envelope, gripper fully open, approach vector free of static occlusions);
$\mathcal{P}_{\text{post}}$ defines the postcondition assertions that must hold upon completion (e.g., vertical object centroid displacement exceeding a threshold, gripper aperture matching object width);
$\Pi_{\text{exec}}$ denotes the family of compatible parameterized low-level executors (e.g., Contact-GraspNet antipodal samplers, Cartesian straight-line admittance controllers, or specialized local visuomotor policies);
$\mathcal{V}_{\text{eval}}$ represents the multi-modal physical verification operator responsible for outputting a deterministic success evaluation;
$\mathcal{R}_{\text{recov}}$ denotes the bounded recovery policy branch specifying local retreat and compensation trajectories upon anomalous execution.

Here, training-free has a precise mathematical meaning: acquiring, validating, and consolidating an incoming task involves zero gradient backpropagation or weight optimization across underlying neural networks. Upstream foundation models and downstream geometric tools remain frozen; capability growth is realized strictly through explicit topological expansion and parameter synthesis within the skill library and memory network.

View full-size image ↗

Figure 2. One task moves through teaching, composition, execution, verification, and persistent storage. The resulting library and memory become the starting point for the next task.

Subgoal Reasoning on Top, Metric Geometry and Control Underneath

TGL enforces a disciplined division of responsibilities across hierarchical tiers, ensuring high-level semantic generalization while maintaining low-level physical precision:

The AI Agent segments incoming multimodal demonstrations along boundaries of physical state transitions. Rather than naively memorizing time-series joint waypoints, the system identifies structural shifts in spatial contact topology (e.g., free-space transit $\to$ pre-contact alignment $\to$ stable dynamic constraint $\to$ release and decoupling). Although kinematic speeds and approach vectors vary, shared topological state transitions define the structural skeleton of the candidate Skill Block.

2. Scene Grounding and Metric Geometric Mapping

When the agent schedules a Skill Block, its abstract causal parameters are mapped to the local 3D workspace:

Perception and Segmentation: Onboard RGB-D sensors capture point clouds and color imagery, invoking open-vocabulary detectors and segmenters to isolate target geometric meshes;
Grasp Affordance Synthesis: Contact-GraspNet densely evaluates force-closure grasp candidates over the 3D surface mesh, selecting antipodal normals that maximize contact stability;
Collision-Free Motion Planning: MPLib and OMPL motion planners generate kinematically continuous trajectories that navigate around obstacles represented in a local Euclidean Signed Distance Field (ESDF).

3. Sensor-Driven Physical Verification Contracts

Following trajectory execution, the system forbids blind open-loop transitions. Instead, the multi-modal verification operator $\mathcal{V}_{\text{eval}}$ audits execution states against empirical ground truth:

For GraspObject, reaching commanded joint waypoints is insufficient. The controller concurrently evaluates gripper motor current feedback, infrared optical beam interruptions between gripper fingers, and wrist 6-axis force-torque wrench differentials. Only when force signatures confirm the gravitational payload of the target object does the postcondition pass;
If verification registers an unladen gripper, an incipient slip is flagged, dynamically routing execution to the bounded recovery branch $\mathcal{R}_{\text{recov}}$ to adjust approach yaw and re-attempt grasp acquisition locally, preventing error compounding.

View full-size image ↗

Figure 3. The contract connects a semantic subgoal to grounded execution and observable evidence. Strategy is carried forward; geometry and control are recomputed.

Dual Memory Architecture: Skill Library and Experience Memory

To balance generic capability retrieval with specialized domain adaptation, TGL structures knowledge across two distinct, complementary repositories:

1. Skill Library ($\mathcal{L}_{\text{skill}}$)

The Skill Library maintains a curated catalog of validated atomic skill templates. Each entry encapsulates formal subgoal specifications, geometric precondition checkers, scene grounding rules, and compatible motion planner interfaces. The library prioritizes abstraction and domain independence, allowing skills to be instantiated across diverse objects and functional workspaces.

2. Experience Memory ($\mathcal{M}_{\text{exp}}$)

Experience Memory functions as an episodic, case-based repository that logs contextual metadata and diagnostic histories from real-world execution episodes:

Catalogs ambient lighting parameters, surface material properties, grasp quality scores, and empirical execution outcomes;
Archives failure traces alongside successful recovery trajectories (e.g., recording that lateral pinch grasps on polished metallic cylinders induced rotational slip, whereas a vertical top-down pinch established stable force closure);
When the robot encounters analogous operational contexts in future missions, Experience Memory injects contextual priors into the planning prompt, biasing the agent away from historically vulnerable approaches and selecting pre-validated parameter sets.

This decoupled architecture provides unmatched engineering auditability: practitioners can inspect individual skill boundaries, refine recovery routines, or deprecate faulty tool configurations without initiating uncontrollable global model retraining.

Lifelong Learning and Analytical Cost Scaling

Conventional reinforcement learning and imitation learning face compounding difficulties when scaling across broad task catalogs due to parameter interference. TGL introduces the Effective Reusable Experience hypothesis: as a robot operates across extended horizons, accumulated skill blocks and contextual memory nodes progressively cover the predominant geometric and contact interactions within its operational domain.

View full-size image ↗

Figure 4. Conceptual predictions for future-task error and teaching demand. The curves are schematic.

Analytically, we compare the marginal computational cost of traditional parameter-space updates against TGL’s modular expansion:

Global Parameter Update Paradigm: Adding the $N$-th task requires joint optimization across all preceding $N$ tasks to prevent catastrophic forgetting. The required training samples and compute scale super-linearly as $\mathcal{O}(N^\alpha)$ ($\alpha > 1$);
TGL Modular Expansion Paradigm: Synthesizing and validating an incoming Skill Block requires verifying only local execution contracts against the specific task scene. Marginal acquisition overhead remains bounded at $\mathcal{O}(1)$.

Provided that scene grounding, verification checks, and skill retrieval maintain lightweight computational profiles, explicit skill accumulation delivers dramatic long-term efficiency advantages over perpetual fine-tuning.

View full-size image ↗

Figure 5. Analytic cost regimes under different assumptions about coverage and local compatibility.

Slow Teacher and Fast Student: Dual-System Synthesis

In real-world industrial and service settings, relying entirely on agent deliberation for every single motion introduces latencies of several hundred milliseconds to seconds, which is ill-suited for high-speed dynamic execution.

To resolve this latency-generality trade-off, TGL natively supports a Slow Teacher, Fast Student dual-system execution framework:

Slow Teacher (System 2: TGL Deliberative Loop): The AI Agent functions as a deliberative System 2, performing long-horizon reasoning, tool composition, and fault recovery in unfamiliar tasks. In doing so, it filters out execution failures and buffers clean, physically verified expert trajectories into an execution repository;
Fast Student (System 1: Distilled Reactive Policy): Verified trajectories are used to asynchronously train high-frequency reactive policies (such as a 50 Hz Diffusion Policy or Action Chunking Transformer). These compact models execute fluid, low-latency primitives under nominal operational conditions;
Seamless Anomaly Hand-off: If the reactive policy experiences an out-of-distribution observation or fails a physical verification check, low-level monitors instantly trigger an interrupt, returning control to System 2 to perform root-cause diagnosis, execute recovery, and update Experience Memory.

Empirical Benchmark Validation and Insights

We conducted rigorous empirical evaluations across the standard robotic manipulation benchmark LIBERO as well as its out-of-distribution counterpart, LIBERO-Plus. Experiments focused on key evaluation axes:

Few-Shot Skill Acquisition: Measuring TGL’s capacity to extract invariant subgoal structures from 1 to 3 demonstrations and deploy them successfully across novel spatial layouts and visual textures;
Mitigation of Retraining Overhead: Quantifying cumulative computational overhead and historical task retention when acquiring 10 consecutive tasks compared against parameter fine-tuning baselines;
Fault Tolerance via Physical Contracts: Evaluating task success rates under induced physical disturbances (e.g., external perturbations during grasping, visual feature occlusions) with and without bounded recovery routines.

Key empirical findings include:

TGL achieved substantially higher out-of-distribution generalization compared to few-shot behavioral cloning baselines while requiring minimal demonstration data;
Because base model weights remain frozen, performance across historical tasks exhibited zero degradation, completely eliminating catastrophic forgetting;
Explicit verification contracts and localized recovery routines resolved the majority of operational disturbances autonomously, sustaining reliable long-horizon execution.

Summary and Engineering Takeaways

TGL demonstrates that general-purpose robotic manipulation does not strictly mandate monolithic end-to-end neural network optimization for every newly encountered task.

By unifying foundation model semantic reasoning, metric geometric toolsets, deterministic impedance control, and structured dual-memory repositories, a robot can acquire, adapt, and retain complex capabilities without weight perturbation. Preserving knowledge as explicit, inspectable, and reusable objects offers an economically scalable and physically grounded blueprint for lifelong robot learning.

Towards the Vision-Sound-Language-Action Paradigm: The HEAR Framework for Sound-Centric Manipulation

Tue, 17 Mar 2026 03:22:30 +0000

Robots that cannot hear the physics

Current vision-language-action models see, understand language, and move. What they do not do is listen. They operate in what amounts to a silent world, even though the robot’s own environment is full of sound that encodes physical state the cameras cannot recover: liquid starting to sputter as it boils, the sharp click of a collision, the difference in timbre between a full container and an empty one, the moment a material stops flexing and starts cracking.

Contact acoustics are the cheapest tactile sensor available. A microphone is a few dollars, it needs no contact, and it covers the whole workspace at once. The case for using it is not speculative. The problem is that nobody has built a policy that can.

The cue is gone before the policy is asked

The difficulty is not a modelling problem, and framing it that way is why earlier approaches have not solved it. It comes from the control architecture that makes large VLA models practical.

A large backbone cannot run at control frequency. So the standard design chunks actions: the model produces a short sequence of future commands, the robot executes them open-loop, and only then does the model get queried again. That gap is where the physics happens. A collision, a boil-over, a completed press: these are brief, non-repeatable acoustic events, roughly on the order of a hundred milliseconds, and they routinely fall entirely inside the interval where the policy is not looking. The average chunk at 30 Hz is about a second long.

The consequence is worth stating precisely, because it is not what people usually assume. A late reaction would be a delay. What actually happens is that the evidence is destroyed. The audio arrives, nothing is listening, and by the time the policy is queried again, the window contains only silence. If an event occupies samples that end before the next window opens, the window does not contain a weak version of that event. It contains nothing. We call the interval between decisions the Blind Execution Interval, and the word blind is doing real work.

There is a second failure that has nothing to do with timing. Look at a robot that has been waiting beside a running appliance. If the scene is quasi-static, the current image looks the same whether it has been waiting one second or ten. The expert demonstrations differ, because the right next action depends on elapsed time, but the observation does not. A snapshot policy has no way to tell those situations apart, so it either drifts or freezes. This is the aliasing problem that temporal context is supposed to solve, and without audio it is genuinely unobservable.

What earlier attempts leave out

Turning audio into text is the obvious first instinct, and it loses the thing we need. Speech recognition front ends discard non-speech cues and prosody entirely, and they compress timing into discrete tokens. A beep produces nothing at all. More subtly, an affirmative “Yes!” and a doubtful “Yes?” collapse into nearly identical token sequences, so the difference in pragmatic intent disappears while transcription accuracy looks perfect.

Rendering the waveform as an image and treating it as another camera view is the second instinct. It captures amplitude changes, and it obscures spectral structure: frequency contours and timbre, which is where material and contact information live. It is also sensitive to where the event happens to land inside the rendered window, so performance changes for reasons that have nothing to do with the physics.

A more recent line adds speech to VLA models and preserves prosody and speaker identity. That is real progress, and it targets explicit verbal commands rather than the ambient acoustic physics of manipulation, which is a different problem. The omni-modal systems are closer still, and the reservation is structural: they are built to listen before acting, using sound to establish context or infer intent around an action, rather than as low-level feedback during contact. None of them address the misalignment between transient events and chunked execution, which is the actual bottleneck.

Audio foundation models such as CLAP and ImageBind were trained on fixed-length clips, offline and independently. Deployed statelessly, they carry no temporal context across clips. Compact audio-native policies run fast and can handle contact, but their scale makes long-horizon, multi-stage behavior brittle. Processing audio natively turns out to be necessary and not sufficient.

The shortcut worth naming is faster replanning. Truncate the chunk, query more often, and the blind interval shrinks. We tried the variants. It helps less than it looks like it should, and it pays in motion quality: discontinuous, jerky trajectories that introduce localized action loops. It also cannot fix system latency, which no query rate eliminates. A cue lost to latency is lost regardless of how often you ask.

And the benchmarks are no help, because every standard simulator is silent. RLBench and ManiSkill test spatial reasoning and visual control in environments with no acoustic channel at all. Navigation benchmarks that do use sound let the agent move its base to improve its listening geometry, which is not an option for a manipulator that has to keep working while generating its own mechanical noise.

Separate the sensing rate from the decision rate

Our answer comes from noticing that this is a rate mismatch, not an information deficit. The sound is there. The policy is simply asked too rarely to catch it.

So instead of making the policy faster, we stop tying the two rates together. A persistent memory is updated continuously from the audio stream, packet by packet, independently of when the policy is queried. When a decision is finally requested, the memory already contains everything that arrived, including events that happened during blind intervals. The robot is not reacting to the last window. It is reacting to the last second of listening, with the recent history summarized.

That is the core move, and everything else in the architecture is bookkeeping around it. There is something that remembers, something that interprets, something that predicts what comes next acoustically, and something that acts. We named them accordingly.

Two constraints shaped the design. The first is that the audio path has to be causal in the strict sense: nothing may look at samples that have not arrived, and the tokenizer must produce codes for a segment from that segment alone. The second is that the memory horizon has to be a target, not a hope. It needs to cover the worst-case decision gap plus the measured system latency, otherwise the guarantee is only statistical.

The four modules

Historizer. Between two decision boundaries, the robot may receive a long run of consecutive audio packets, each about forty milliseconds. The Historizer consumes them as they arrive and updates a streaming stateful Transformer memory. The memory is read out at decision times and never otherwise. Its entire justification is the evidence-vanishing argument above: a window-only interface cannot represent something that happened just before the window opened. We also tried a causal GRU and exponential moving average pooling over audio features; the streaming attention’s extra capacity earned its place. We chose a short window with a persistent summary over a very long window, because long windows mix distinct interaction phases together and make the result sensitive to exactly where the window boundary falls.

Envisioner. Interpretation is hierarchical, because the two jobs have very different frequencies. A high-level model reads the multi-view images, the current acoustic window, the instruction, proprioception and the memory, and produces three things: a semantic latent, a key-value cache, and a short structured description of the current task stage, something like {"stage": "wait", "subgoal": "listen for beep"}. That stage text comes from the high-level model’s own text head. A smaller low-level model then reuses the cache with the current robot state to produce the control feature. The split keeps expensive semantic interpretation out of the fast path. The stage supervision matters more than it might appear: without it, tasks that require a quick behavioral shift suffer, because nothing forces the latent to name the phase it is in, and the phases that matter acoustically are often visually identical.

Advancer. This one predicts the audio that will arrive between the current decision and the next one, as discrete codes from the streaming tokenizer, trained by cross-entropy. It is used only during training and is dropped at deployment, so it costs nothing at inference time. Its purpose is to give the latent a concrete sense of elapsed time and process progression. Pouring, boiling, an alarm, a conversation: each has a different acoustic future, and predicting it forces the shared representation to encode where in the process the robot currently is. Without it, the failures look different from the Historizer’s. They happen late. The robot keeps pouring after the sound has already told it to stop. That asymmetry is how we knew the two modules were solving different problems rather than duplicating each other.

Realizer. Actions are generated by conditional flow matching from a Gaussian prior to the action chunk, integrated in a handful of Euler steps, conditioned on the control feature through adaptive normalization. We used flow matching rather than diffusion for the step count. The detail worth dwelling on is why smoothness matters here. Replacing the Realizer with direct regression produces jitter, and jitter injects mechanical self-noise into the microphone signal, which masks the brief cues the rest of the system is trying to hear. A perception failure caused by a control choice. The sharpest version of the claim is that a policy which listens while moving has to move in a way that makes listening possible. Sensing and control are not separable in this setting.

View full-size image ↗

A high-level model produces semantics and task stage; a low-level model produces the control feature.

View full-size image ↗

The Advancer is used only during training and is removed at deployment.

The three objectives are summed with weights that ramp in over the first part of fine-tuning, so the model learns imitation before it is asked to be temporally grounded.

Where the data came from, since it did not exist

No robot dataset contains synchronized audio. Open X-Embodiment has no microphones at all, real synchronized audio is noisy and hardware-specific and does not scale across labs, and no standard simulator models acoustics.

We built both pieces. OpenX-Sound retrofits audio onto existing episodes from video, preserving every original signal, and we audited a sample of it by hand for synchronization and semantic relevance. We are explicit that this is a bootstrapping mechanism for representation learning and not a substitute for physical data: synthesized audio can omit subtle contact cues, shift event timing, or hallucinate a sound with no physical source. Every result in the paper fine-tunes an OpenX-Sound checkpoint on real task-specific demonstrations with real audio. The synthesized audio is what makes a pretrained acoustic prior possible, not what makes the behavior work.

HEAR-Bench is the other half. Built on a dual-arm simulator, it runs an asynchronous audio thread that injects ambient noise and trigger clips at the exact sample index implied by the control cycle, and it scores success only if the geometric goal is reached after the cue. That timing rule is the whole contribution. Without it, pressing a button the instant the episode starts and pressing it when the alarm actually rings look identical.

Rigorous Real-Robot Experiment Design: Why These Tasks?

When validating the acoustic manipulation capabilities of end-to-end multimodal Vision-Language-Action (VLA) foundation models, we deliberately avoided arbitrary toy benchmarks. Instead, we designed each real-world robotic evaluation specifically targeting the fundamental physical blind spots of vision-only policies:

1. Moka Pot Coffee Brewing: A True Barista Robot vs. Superficial Button-Pushing

The vast majority of existing demonstrations claiming “robot coffee brewing” merely involve a mechanical arm approaching a commercial capsule or fully automated pod coffee machine, executing an open-loop press on a static plastic button, and idly waiting while internal integrated circuits run a blind timer. Such setups require zero perception of the underlying physical state—they are glorified mechanical switch activators, indifferent to fluid dynamics, thermal phase changes, or culinary progression.

In contrast, our setup places an authentic dual-chamber metal Italian moka pot on an active electric cooktop. The moka pot consists of fully opaque metal walls; the boiling pressurized water in the boiler, the compacted coffee puck in the funnel filter, and the ascending extraction column inside the collector chimney remain entirely concealed from view. External RGB cameras and infrared depth sensors cannot penetrate the metal shell to observe fluid levels or thermodynamic transitions.

A seasoned master barista relies neither on stopwatches nor visual inspection; extraction cutoff is governed purely by ear:

Silent Heating Phase: Water temperature climbs gradually with negligible acoustic emission beyond faint thermal conduction;
Smooth Boiling & Laminar Extraction Phase: Saturated steam pressure forces water smoothly through the coffee puck, producing a continuous, gentle, muffled bubbling sound;
Violent Sputtering Phase (Cavitation & Steam Eruption): As boiler water nears depletion, high-temperature steam mixes with residual liquid to produce a violent, unmistakable high-frequency sputtering and hissing eruption through the extraction chimney.

This acoustic inflection point is the decisive boundary in coffee extraction: removing the pot one second too early yields sour, under-extracted, watery coffee; leaving it on the heat two seconds too late causes dry-boil scorching, bitterness, and permanent thermal damage to the moka pot base.

HEAR functions as a genuine barista robot: powered by the end-to-end VLA foundation model, it continuously analyzes high-frequency spectral transitions in real time. The moment the distinctive micro-sputtering signature emerges, the policy issues action chunks with millisecond-level responsiveness, smoothly lifting the moka pot away from the heating burner.

View full-size image ↗

Contrasting superficial open-loop button pressing with HEAR’s acoustic monitoring of internal thermodynamic phase transitions inside an opaque metal moka pot.

2. Shaking a Bottle: Active Acoustic Probing

In everyday physical environments, numerous pill bottles, opaque jars, and enclosed canisters conceal their contents entirely. Passive vision fails fundamentally in such settings—no matter how many camera viewpoints or lighting angles are deployed, the exterior surface remains completely static.

To overcome this, the robot must exhibit Active Exploration: it must grasp and shake the container to inject mechanical energy, exciting internal collisions. By decoding the resonant damping, impact frequencies, and high-frequency collision timbres of the internal objects against the container walls, the model disambiguates the presence, material properties, and fill level of the enclosed items. In our real-robot demonstration, we deploy an occupied container filled with loose objects, showing how the robot actively shakes the bottle, decodes transient collision acoustics, and executes informed downstream classification.

3. Real Alarm Clock: Resolving Temporal Aliasing Under Quasi-Static Vision

An alarm clock in its silent standby state and active ringing state can appear visually indistinguishable. Lacking acoustic feedback, naive end-to-end policies succumb to shortcut learning, prematurely hitting the stop button at the very start of the episode. Our benchmark enforces that the stop button must be actuated only after the acoustic trigger chime sounds, directly probing whether the multimodal VLA accurately aligns actions with transient acoustic timing cues.

4. Pouring into Opaque Containers and Answering Phone Calls

Pouring water into an opaque cup represents the canonical scenario where sound is the sole observable state signal. Fluid levels are strictly occluded; the true decision variable is the continuous frequency drift rate of the pouring stream resonating within the shrinking acoustic cavity. Similarly, answering a phone call assesses multi-stage temporal state machines: the robot approaches upon hearing the ringing chime, remains stationary during the caller’s acoustic pauses, and docks the receiver only when the terminal dial tone registers.

Real-robot experiments

All demonstrations include their recorded audio. Select a video to play it.

Moka CoffeeThe robot listens for the acoustic sputtering phase inside the opaque pot before removing it from heat.

Answer PhoneRinging, speech, and the end of a call distinguish visually similar task stages.

Shake BottleThe robot actively shakes an opaque container, discerning internal contents from collision acoustics.

Real Alarm ClockThe robot waits for the alarm before pressing the stop button.

Core Insights and Engineering Takeaways

Integrating high-frequency continuous acoustic feedback into end-to-end embodied foundation models (VLA) demands far more than appending an extra feature dimension to the visual tokenizer. Conventional paradigms treat sound as an auxiliary camera stream, yet the physics of sound is intrinsically event-driven, high-frequency, and transient. Grounding it in robotic action mandates resolving the fundamental architectural tension between continuous acoustic cadence and chunked control loops.

This structural tension extends well beyond sound. Any physical contact modality that generates rapid, non-repeatable events faster than the policy’s replanning frequency—most notably high-frequency tactile arrays and micro-vibrations—encounters identical rate-mismatch bottlenecks. This asymmetry cannot be bypassed by merely optimizing the inference throughput of large foundation models.

Artificially increasing the model’s query rate fails for structural reasons rather than tuning deficiencies: truncating action chunks trades long-horizon trajectory smoothness for erratic reactivity, injecting severe mechanical jitter and self-noise directly into the sensory stream. The only robust solution is decoupling perception cadence from decision loops via explicit streaming causal memory. Auxiliary acoustic future prediction objectives ground shared representations in physical time at zero inference overhead, while rigorously timed causal benchmarks transform “appearing visually plausible” into “authentically listening to the physical world.”

What It Cannot Do Yet

One workspace microphone means no source localization and no separation of overlapping events, which is functionally the cocktail party problem. We would want an array, or a base-plus-end-effector pair for beamforming. The system is trained purely by offline imitation, so its robustness is bounded by the diversity of noise and timing variation in the demonstrations; online fine-tuning or reinforcement learning in the deployed room is the obvious remedy. Audio and vision are processed as separate streams and fused at the representation level, and we suspect native audio-visual encoders that associate timbre with contact directly would do better. And unconstrained real-world cooking acoustics remain an open problem, with real deployment markedly harder than simulation.

The Larger Frame

The tension we started from, between high-frequency transient sensory events and low-frequency delayed policy updates, is not specific to sound. It will show up for every physical modality that carries information in its timing. HEAR is our attempt to show that the fix is architectural, and to leave behind the pieces—the VSLA formulation, the pretraining corpus, and the benchmark—that let other researchers build further on this foundation.

ERMV: Editing 4D Robotic Multi-view images to enhance embodied agents

Wed, 23 Jul 2025 12:41:11 +0000

Demonstrations Are Expensive, Yet We Already Have Them

Vision-Language-Action (VLA) foundation models rely heavily on synchronized multi-view video demonstrations aligned with dense action trajectories. Capturing such teleoperated physical data is notoriously slow, labor-intensive, and non-scalable: a single dexterous manipulation episode entails thousands of high-resolution image frames across heterogeneous camera viewpoints. The scarcity of expert demonstrations constitutes a primary bottleneck restricting embodied policies from zero-shot generalization in open-world environments.

This investigation began with a fundamental observation: the behavioral intelligence already exists within recorded datasets; what is missing is visual diversity. A demonstration of picking up a mug conveys the essential physical mechanics, but reflects only a single physical instantiation of the environment. If an agent has only ever encountered that mug on a specific countertop under fixed lab illumination, it cannot transfer robustly to novel kitchens or unstructured homes.

To unlock this data bottleneck, embodied AI fundamentally requires an action-conditioned, geometrically consistent Robotic World Model. If a world model can take established high-quality action sequences as conditions, simulate physical rollouts across diverse visual counterfactuals (varied textures, lighting, cluttered backgrounds) while preserving the exact contact physics and execution dynamics, the value of existing robotic datasets can be magnified exponentially.

This defines the core identity of ERMV: a Controllable 4D Multi-View Robotic World Model. Rather than executing naive 2D image modifications, ERMV models the entire continuous 4D spatiotemporal physical trajectory as an action- and state-conditioned generative state transition flow.

The defining challenge lies in strict action-physics coupling. A robot demonstration is not a generic monocular video clip; it comprises synchronized multi-camera streams coupled with high-frequency kinematic states, end-effector poses, and gripper commands. If a world model introduces geometric hallucinations, disrupts projection consistency, or alters the physical contact boundaries between gripper and object, the generated rollout becomes an invalid, physically violating demonstration—poisoning rather than improving downstream policy training.

View full-size image ↗

Visual variation propagates through synchronized views while the recorded actions stay intact.

Why Conventional Generative Models Fail at Physical Simulation

In recent years, large-scale general video generators (e.g., Sora) have demonstrated astonishing visual fidelity, prompting efforts to adopt them directly as embodied world models. However, in precision robotic manipulation, unconstrained monocular video generation and autonomous driving world models fail fundamentally:

Rigid priors in driving world models break down in manipulation: Autonomous driving world models rely on static circumferential camera rigs and flat Bird’s-Eye-View (BEV) ground planes, where cross-view attention is hardcoded to fixed spatial baselines. In robot manipulation, the multi-camera setup is highly dynamic: eye-in-hand cameras move continuously relative to third-person perspectives, nullifying rigid geometric topology priors.
Video models without strict action conditioning cause physical hallucinations: Unconditioned video diffusion hallucinates plausible-looking textures but cannot enforce strict conditioning on joint angles, gripper states, and 6-DoF end-effector velocities. Objects morph, drift, or teleport without contact, and robotic fingers penetrate through solid boundaries—rendering the output useless for training robust VLA policies.
Frame-by-frame inpainting destroys 4D spatiotemporal coherence: Applying single-frame diffusion inpainting across frames lacks temporal and inter-camera constraints, leading to severe flickering, perspective tearing, and geometric inconsistency.
Static 3D reconstruction struggles with dynamic contact and motion blur: While NeRF or 3D Gaussian Splatting provides rigid 3D consistency, it struggles with complex transient contacts, deformable materials, and high-velocity motion blur frequently present in fast robot manipulation.

Embodied AI demands a world model engineered from the ground up to respect robot kinematics, guarantee 4D multi-camera geometric consistency, and support long-horizon physical rollouts.

What makes it hard

The camera rig is not fixed. Manipulation footage comes from a moving, egocentric multi-camera system, so any consistency mechanism keyed to known static inter-camera geometry stops applying.

Motion blur breaks the geometry we would normally rely on. Because the camera and the objects move at the same time, features sampled along a precise epipolar line in a blurry image may not correspond to the true pixel positions at all. The constraint that makes multi-view attention correct is, under motion, wrong.

Errors compound. Edited frames are fed back as history for the next window, so a small error does not stay small. Over long horizons this shows up as semantic drift and blurred detail. In this domain the tolerance is especially low, because the robot arm and the manipulated object have to stay consistent through the entire sequence. When they do not, the sequence does not merely look wrong. It stops being valid training data.

The compute tension is structural rather than incidental. Long-range memory wants a large working window, and dense attention over a window of L timesteps and N views wants a large amount of GPU memory. We wanted something that trains on a single consumer GPU.

Preserving the objects that matter has no cheap solution. Training a general segmenter for manipulated objects is not feasible, because those objects are diverse, often novel, and viewed from challenging egocentric angles. Labelling every frame by hand across thousands of images is not feasible either.

And text prompts lack the granularity the task needs. Asking for a background change to an office leaves the colour, the type of room, and its orientation undefined, and the result may conflict with what the robot is doing.

Use the structure that robot data already has

Robot data is unusually well structured. It comes with calibrated cameras, recorded camera poses, recorded robot states, and a shared clock. Our idea is to stop trying to invent a specification for the edit and instead reuse that structure, plus a single hand-edited keyframe.

The first frame of the primary camera is edited, by an inpainting model or by hand, and that edited image becomes the visual specification. The paper calls it a blueprint, which is the right word. It carries colour, layout and orientation unambiguously in a way that a sentence cannot, and it takes a person one edit rather than thousands.

Two design decisions follow from the structure, and both are about relaxing constraints rather than satisfying them.

The first concerns epipolar geometry. A moving gripper is smeared across pixels by motion blur, and the epipolar line computed from the poses points to where the gripper would be if it were stationary. Enforcing that constraint exactly matches the wrong pixel. So we keep epipolar geometry as the search structure and learn the deviation. A small network predicts the motion-induced offset from local image features and the motion condition, and attention is taken along the shifted line. The constraint is relaxed rather than discarded.

The second concerns attention. Dense spatio-temporal attention is the default way to keep a sequence consistent, and it is expensive. The domain-specific observation that justifies not doing it is that in many manipulation scenarios the background is largely static and the images change slowly. There are not many inter-frame distinctions that need to be captured by a dense mechanism. So we sample.

View full-size image ↗

A fixed budget of observations provides long-range context for the whole window.

Sparse context, and giving back what sparsity removed

Sparse Spatio-Temporal context takes a sliding window of timesteps and samples a fixed-size subset of images from it, which is far smaller than the full time-by-view grid. The sampling is random, and it is worth saying so plainly rather than attributing intelligence to it that is not there.

What makes it work is the second half. Each sampled image carries its original time index and camera index, and those identities are explicitly encoded as a condition. Subsampling destroyed the spatio-temporal structure, so the structure is re-injected. A frame from three seconds ago is still labelled as three seconds ago, which means it contributes appearance evidence without being mistaken for the current state. This is a small idea with wide applicability: when you subsample structured data, put the structure back.

There is a second decision that distinguishes this from the usual autoregressive scheme. The sampled history is injected into the network as a condition, and it is also generated alongside the future frames. Future frames can therefore extract geometric structure from history during generation, rather than only conditioning on it.

The memory saving from sparse sampling at a fixed window size is roughly a factor of two, and that factor is what makes single-GPU training possible at all.

Where the motion conditioning lives

Each target image is conditioned on the camera pose, the robot state, and the temporal deltas of both. The static terms ground each view geometrically, and the deltas are what allow the model to render motion blur at all. This is more consequential than it sounds. Removing the motion conditioning produces images that look sharper, because blur is a kind of softness, and they lack the physical characteristics a real camera would have captured. Perceptual sharpness is not physical realism, and a policy trained on impossibly sharp images is trained on a distribution that does not exist.

Training assumes synchronized pose and state records exist. That assumption is satisfied for robot demonstrations, which is why this works here and would not work on arbitrary video.

View full-size image ↗

The epipolar line is the search structure; a learned offset accounts for motion blur.

A loop that spends human effort where it is needed

Generated frames feed back as history, and history is exactly where errors accumulate. Rather than trying to solve this with a better generator, we handle it at the system level with a verifier.

A multimodal model acts as an automated checker, comparing each generated image against the original with a chain-of-thought prompt. It scores degradation of the core objects on a ten-point scale and emits a consistency judgement. When the check fails, the system asks the expert for a segmentation mask of the core objects, and that mask becomes an additional condition for a corrective regeneration.

We describe this as MLLM review plus expert correction, and we think of it as a practical intermediate path rather than a fully solved problem. The two obvious extremes do not work. A universal segmenter for manipulated objects is not trainable, and labelling every frame is not affordable. The loop keeps human annotation confined to the few cases where the model falters, which is where expert time is actually worth spending.

We should be clear about what this means for how automatic the pipeline is. It is not fully automatic. A person produces the guide frame, and a person may be asked for a mask. Making that boundary move is future work, and we point to semantic segmentation and object detection as the tools that would replace parts of the current manual intervention.

View full-size image ↗

An automated check catches degradation, and human correction is requested only where it fails.

What we take from this

A violated physical constraint should be relaxed rather than dropped. Blur breaks epipolar geometry, and the useful response is to learn the deviation while keeping the geometry as scaffolding. Discarding the constraint would have thrown away the structure that makes cross-view attention work at all.

The justification for sparsity should come from domain statistics, not from generic efficiency arguments. The background is largely static is a real property of manipulation scenes, and it is what makes sparse attention defensible rather than merely cheap.

When you subsample structured data, re-inject the structure you subsampled away. Two indices per sample is a small cost for preserving what the model needs to interpret them.

Error accumulation can be handled at the system level rather than inside the generator. A cheap external verifier is a legitimate alternative to a better model, and it has the side benefit of localizing where the model is weak.

And augment rather than re-collect. That is the practical thesis, and it is the reason the method is worth building even though it needs a guide frame and occasional human help.

System Boundaries and Physical Constraints

As a 4D world model built for real-world manipulation, we must be explicit about the system’s operational boundaries:

Lightweight Representation vs. Explicit 3D Geometry: ERMV pairs multi-view 2D diffusion with motion-aware epipolar attention to maintain 4D consistency implicitly, without yet adopting explicit representations like depth maps, point clouds, or 3D Gaussian Splatting (3DGS). Incorporating explicit 3D geometry remains a natural architectural avenue to further mitigate perspective distortion during rapid spatial displacements.
Human-in-the-Loop Verification: While the MLLM verifier significantly curbs human annotation burden, transparent objects, specular surfaces, and complex fluid dynamics can still produce subtle perceptual false negatives. Retaining a sparse expert intervention channel for degraded masks is a pragmatic engineering design; achieving fully autonomous unmonitored rollouts requires higher-capacity vision backbones.
Physical Dynamics and Kinematic Grounding: ERMV is a generative world model conditioned on recorded actions and robot kinematics. It faithfully synthesizes lighting, textures, backgrounds, and occlusions across time, but it is not an analytic physics engine. When arbitrary actions produce severe unmodeled fractures or turbulent non-rigid chaos, the model relies on the physical manifold captured within its training distribution.

Where It Leads: The Future of Embodied World Models

The primary contribution of ERMV is demonstrating that building a high-fidelity, controllable 4D multi-view robotic world model does not require unconstrained, computationally prohibitive black-box video models. Instead, deeply integrating robot kinematics and multi-view epipolar geometry yields an exceptionally compute-efficient engine for physical rollout and counterfactual generation.

As an embodied world model, ERMV unlocks several transformative applications:

Counterfactual World Engine: Conditioned on a single expert demonstration’s actions and states, the world model rolls out parallel physical realities across heterogeneous visual domains (countertop textures, ambient lighting, cluttered backgrounds, reflective surfaces), supplying downstream VLA foundation models with virtually limitless training data.
Action-Conditioned Rollout for Offline Verification: Given an initial physical scene and a candidate planned trajectory, the world model rolls out future multi-view 4D observation streams, serving as a safe offline sandbox to diagnose potential collisions or failures without endangering hardware.
A Generative Sim-to-Real Bridge: Large volumes of action trajectories can be sampled inside inexpensive physical simulators, and then transformed through ERMV’s action-aligned world model into multi-view streams endowed with real-world sensor artifacts (lens noise, dynamic motion blur, authentic material responses)—bridging the perceptual sim-to-real chasm through generative physical rollouts.

Embodied Intelligence | Chang Nie

Teach and Grow: An Agent-Centered Architecture for General Robot Learning

A Few Demonstrations Become Skills That Stay

Where End-to-End Robot Learning Gets Expensive: The Retraining Tax

From Planning AI Agents to GPT-6 Centered Robotic Manipulation

Formal Modeling: Executable Capabilities as Explicit Skill Blocks

Subgoal Reasoning on Top, Metric Geometry and Control Underneath

1. Cross-Modal Demonstration Parsing and Subgoal Alignment

2. Scene Grounding and Metric Geometric Mapping

3. Sensor-Driven Physical Verification Contracts

Dual Memory Architecture: Skill Library and Experience Memory

1. Skill Library ($\mathcal{L}_{\text{skill}}$)

2. Experience Memory ($\mathcal{M}_{\text{exp}}$)

Lifelong Learning and Analytical Cost Scaling

Slow Teacher and Fast Student: Dual-System Synthesis

Empirical Benchmark Validation and Insights

Summary and Engineering Takeaways

Towards the Vision-Sound-Language-Action Paradigm: The HEAR Framework for Sound-Centric Manipulation

Robots that cannot hear the physics

The cue is gone before the policy is asked

What earlier attempts leave out

Separate the sensing rate from the decision rate

The four modules

Where the data came from, since it did not exist

Rigorous Real-Robot Experiment Design: Why These Tasks?

1. Moka Pot Coffee Brewing: A True Barista Robot vs. Superficial Button-Pushing

2. Shaking a Bottle: Active Acoustic Probing

3. Real Alarm Clock: Resolving Temporal Aliasing Under Quasi-Static Vision

4. Pouring into Opaque Containers and Answering Phone Calls

Real-robot experiments

Core Insights and Engineering Takeaways

What It Cannot Do Yet

The Larger Frame

ERMV: Editing 4D Robotic Multi-view images to enhance embodied agents

Demonstrations Are Expensive, Yet We Already Have Them

Why Conventional Generative Models Fail at Physical Simulation

What makes it hard

Use the structure that robot data already has

Sparse context, and giving back what sparsity removed

Where the motion conditioning lives

A loop that spends human effort where it is needed

What we take from this

System Boundaries and Physical Constraints

Where It Leads: The Future of Embodied World Models