ERMV: Editing 4D Robotic Multi-view images to enhance embodied agents

Demonstrations Are Expensive, Yet We Already Have Them

Vision-Language-Action (VLA) foundation models rely heavily on synchronized multi-view video demonstrations aligned with dense action trajectories. Capturing such teleoperated physical data is notoriously slow, labor-intensive, and non-scalable: a single dexterous manipulation episode entails thousands of high-resolution image frames across heterogeneous camera viewpoints. The scarcity of expert demonstrations constitutes a primary bottleneck restricting embodied policies from zero-shot generalization in open-world environments.

This investigation began with a fundamental observation: the behavioral intelligence already exists within recorded datasets; what is missing is visual diversity. A demonstration of picking up a mug conveys the essential physical mechanics, but reflects only a single physical instantiation of the environment. If an agent has only ever encountered that mug on a specific countertop under fixed lab illumination, it cannot transfer robustly to novel kitchens or unstructured homes.

To unlock this data bottleneck, embodied AI fundamentally requires an action-conditioned, geometrically consistent Robotic World Model. If a world model can take established high-quality action sequences as conditions, simulate physical rollouts across diverse visual counterfactuals (varied textures, lighting, cluttered backgrounds) while preserving the exact contact physics and execution dynamics, the value of existing robotic datasets can be magnified exponentially.

This defines the core identity of ERMV: a Controllable 4D Multi-View Robotic World Model. Rather than executing naive 2D image modifications, ERMV models the entire continuous 4D spatiotemporal physical trajectory as an action- and state-conditioned generative state transition flow.

The defining challenge lies in strict action-physics coupling. A robot demonstration is not a generic monocular video clip; it comprises synchronized multi-camera streams coupled with high-frequency kinematic states, end-effector poses, and gripper commands. If a world model introduces geometric hallucinations, disrupts projection consistency, or alters the physical contact boundaries between gripper and object, the generated rollout becomes an invalid, physically violating demonstration—poisoning rather than improving downstream policy training.

View full-size image ↗

Why Conventional Generative Models Fail at Physical Simulation

In recent years, large-scale general video generators (e.g., Sora) have demonstrated astonishing visual fidelity, prompting efforts to adopt them directly as embodied world models. However, in precision robotic manipulation, unconstrained monocular video generation and autonomous driving world models fail fundamentally:

1. Rigid priors in driving world models break down in manipulation: Autonomous driving world models rely on static circumferential camera rigs and flat Bird’s-Eye-View (BEV) ground planes, where cross-view attention is hardcoded to fixed spatial baselines. In robot manipulation, the multi-camera setup is highly dynamic: eye-in-hand cameras move continuously relative to third-person perspectives, nullifying rigid geometric topology priors.
2. Video models without strict action conditioning cause physical hallucinations: Unconditioned video diffusion hallucinates plausible-looking textures but cannot enforce strict conditioning on joint angles, gripper states, and 6-DoF end-effector velocities. Objects morph, drift, or teleport without contact, and robotic fingers penetrate through solid boundaries—rendering the output useless for training robust VLA policies.
3. Frame-by-frame inpainting destroys 4D spatiotemporal coherence: Applying single-frame diffusion inpainting across frames lacks temporal and inter-camera constraints, leading to severe flickering, perspective tearing, and geometric inconsistency.
4. Static 3D reconstruction struggles with dynamic contact and motion blur: While NeRF or 3D Gaussian Splatting provides rigid 3D consistency, it struggles with complex transient contacts, deformable materials, and high-velocity motion blur frequently present in fast robot manipulation.

Embodied AI demands a world model engineered from the ground up to respect robot kinematics, guarantee 4D multi-camera geometric consistency, and support long-horizon physical rollouts.

What makes it hard

The camera rig is not fixed. Manipulation footage comes from a moving, egocentric multi-camera system, so any consistency mechanism keyed to known static inter-camera geometry stops applying.

Motion blur breaks the geometry we would normally rely on. Because the camera and the objects move at the same time, features sampled along a precise epipolar line in a blurry image may not correspond to the true pixel positions at all. The constraint that makes multi-view attention correct is, under motion, wrong.

Errors compound. Edited frames are fed back as history for the next window, so a small error does not stay small. Over long horizons this shows up as semantic drift and blurred detail. In this domain the tolerance is especially low, because the robot arm and the manipulated object have to stay consistent through the entire sequence. When they do not, the sequence does not merely look wrong. It stops being valid training data.

The compute tension is structural rather than incidental. Long-range memory wants a large working window, and dense attention over a window of L timesteps and N views wants a large amount of GPU memory. We wanted something that trains on a single consumer GPU.

Preserving the objects that matter has no cheap solution. Training a general segmenter for manipulated objects is not feasible, because those objects are diverse, often novel, and viewed from challenging egocentric angles. Labelling every frame by hand across thousands of images is not feasible either.

And text prompts lack the granularity the task needs. Asking for a background change to an office leaves the colour, the type of room, and its orientation undefined, and the result may conflict with what the robot is doing.

Use the structure that robot data already has

Robot data is unusually well structured. It comes with calibrated cameras, recorded camera poses, recorded robot states, and a shared clock. Our idea is to stop trying to invent a specification for the edit and instead reuse that structure, plus a single hand-edited keyframe.

The first frame of the primary camera is edited, by an inpainting model or by hand, and that edited image becomes the visual specification. The paper calls it a blueprint, which is the right word. It carries colour, layout and orientation unambiguously in a way that a sentence cannot, and it takes a person one edit rather than thousands.

Two design decisions follow from the structure, and both are about relaxing constraints rather than satisfying them.

The first concerns epipolar geometry. A moving gripper is smeared across pixels by motion blur, and the epipolar line computed from the poses points to where the gripper would be if it were stationary. Enforcing that constraint exactly matches the wrong pixel. So we keep epipolar geometry as the search structure and learn the deviation. A small network predicts the motion-induced offset from local image features and the motion condition, and attention is taken along the shifted line. The constraint is relaxed rather than discarded.

The second concerns attention. Dense spatio-temporal attention is the default way to keep a sequence consistent, and it is expensive. The domain-specific observation that justifies not doing it is that in many manipulation scenarios the background is largely static and the images change slowly. There are not many inter-frame distinctions that need to be captured by a dense mechanism. So we sample.

View full-size image ↗

Sparse context, and giving back what sparsity removed

Sparse Spatio-Temporal context takes a sliding window of timesteps and samples a fixed-size subset of images from it, which is far smaller than the full time-by-view grid. The sampling is random, and it is worth saying so plainly rather than attributing intelligence to it that is not there.

What makes it work is the second half. Each sampled image carries its original time index and camera index, and those identities are explicitly encoded as a condition. Subsampling destroyed the spatio-temporal structure, so the structure is re-injected. A frame from three seconds ago is still labelled as three seconds ago, which means it contributes appearance evidence without being mistaken for the current state. This is a small idea with wide applicability: when you subsample structured data, put the structure back.

There is a second decision that distinguishes this from the usual autoregressive scheme. The sampled history is injected into the network as a condition, and it is also generated alongside the future frames. Future frames can therefore extract geometric structure from history during generation, rather than only conditioning on it.

The memory saving from sparse sampling at a fixed window size is roughly a factor of two, and that factor is what makes single-GPU training possible at all.

Where the motion conditioning lives

Each target image is conditioned on the camera pose, the robot state, and the temporal deltas of both. The static terms ground each view geometrically, and the deltas are what allow the model to render motion blur at all. This is more consequential than it sounds. Removing the motion conditioning produces images that look sharper, because blur is a kind of softness, and they lack the physical characteristics a real camera would have captured. Perceptual sharpness is not physical realism, and a policy trained on impossibly sharp images is trained on a distribution that does not exist.

Training assumes synchronized pose and state records exist. That assumption is satisfied for robot demonstrations, which is why this works here and would not work on arbitrary video.

View full-size image ↗

A loop that spends human effort where it is needed

Generated frames feed back as history, and history is exactly where errors accumulate. Rather than trying to solve this with a better generator, we handle it at the system level with a verifier.

A multimodal model acts as an automated checker, comparing each generated image against the original with a chain-of-thought prompt. It scores degradation of the core objects on a ten-point scale and emits a consistency judgement. When the check fails, the system asks the expert for a segmentation mask of the core objects, and that mask becomes an additional condition for a corrective regeneration.

We describe this as MLLM review plus expert correction, and we think of it as a practical intermediate path rather than a fully solved problem. The two obvious extremes do not work. A universal segmenter for manipulated objects is not trainable, and labelling every frame is not affordable. The loop keeps human annotation confined to the few cases where the model falters, which is where expert time is actually worth spending.

We should be clear about what this means for how automatic the pipeline is. It is not fully automatic. A person produces the guide frame, and a person may be asked for a mask. Making that boundary move is future work, and we point to semantic segmentation and object detection as the tools that would replace parts of the current manual intervention.

View full-size image ↗

What we take from this

A violated physical constraint should be relaxed rather than dropped. Blur breaks epipolar geometry, and the useful response is to learn the deviation while keeping the geometry as scaffolding. Discarding the constraint would have thrown away the structure that makes cross-view attention work at all.

The justification for sparsity should come from domain statistics, not from generic efficiency arguments. The background is largely static is a real property of manipulation scenes, and it is what makes sparse attention defensible rather than merely cheap.

When you subsample structured data, re-inject the structure you subsampled away. Two indices per sample is a small cost for preserving what the model needs to interpret them.

Error accumulation can be handled at the system level rather than inside the generator. A cheap external verifier is a legitimate alternative to a better model, and it has the side benefit of localizing where the model is weak.

And augment rather than re-collect. That is the practical thesis, and it is the reason the method is worth building even though it needs a guide frame and occasional human help.

System Boundaries and Physical Constraints

As a 4D world model built for real-world manipulation, we must be explicit about the system’s operational boundaries:

1. Lightweight Representation vs. Explicit 3D Geometry: ERMV pairs multi-view 2D diffusion with motion-aware epipolar attention to maintain 4D consistency implicitly, without yet adopting explicit representations like depth maps, point clouds, or 3D Gaussian Splatting (3DGS). Incorporating explicit 3D geometry remains a natural architectural avenue to further mitigate perspective distortion during rapid spatial displacements.
2. Human-in-the-Loop Verification: While the MLLM verifier significantly curbs human annotation burden, transparent objects, specular surfaces, and complex fluid dynamics can still produce subtle perceptual false negatives. Retaining a sparse expert intervention channel for degraded masks is a pragmatic engineering design; achieving fully autonomous unmonitored rollouts requires higher-capacity vision backbones.
3. Physical Dynamics and Kinematic Grounding: ERMV is a generative world model conditioned on recorded actions and robot kinematics. It faithfully synthesizes lighting, textures, backgrounds, and occlusions across time, but it is not an analytic physics engine. When arbitrary actions produce severe unmodeled fractures or turbulent non-rigid chaos, the model relies on the physical manifold captured within its training distribution.

Where It Leads: The Future of Embodied World Models

The primary contribution of ERMV is demonstrating that building a high-fidelity, controllable 4D multi-view robotic world model does not require unconstrained, computationally prohibitive black-box video models. Instead, deeply integrating robot kinematics and multi-view epipolar geometry yields an exceptionally compute-efficient engine for physical rollout and counterfactual generation.

As an embodied world model, ERMV unlocks several transformative applications:

1. Counterfactual World Engine: Conditioned on a single expert demonstration’s actions and states, the world model rolls out parallel physical realities across heterogeneous visual domains (countertop textures, ambient lighting, cluttered backgrounds, reflective surfaces), supplying downstream VLA foundation models with virtually limitless training data.
2. Action-Conditioned Rollout for Offline Verification: Given an initial physical scene and a candidate planned trajectory, the world model rolls out future multi-view 4D observation streams, serving as a safe offline sandbox to diagnose potential collisions or failures without endangering hardware.
3. A Generative Sim-to-Real Bridge: Large volumes of action trajectories can be sampled inside inexpensive physical simulators, and then transformed through ERMV’s action-aligned world model into multi-view streams endowed with real-world sensor artifacts (lens noise, dynamic motion blur, authentic material responses)—bridging the perceptual sim-to-real chasm through generative physical rollouts.