Embodied AI | Chang Nie

A Robotic Chemist: Full-Stack Dual-Arm Mobile Manipulation for Biochemical Experiments

Sun, 06 Sep 2026 00:00:00 +0000

In the emerging paradigm of automated biochemistry and AI for Science, computational chemistry pipelines and large language models (LLMs) can autonomously predict retrosynthetic pathways and design high-throughput reaction conditions. However, the definitive bottleneck in scientific discovery remains closed-loop physical validation in wet laboratories. Traditional specialized automated liquid handling stations are bulky, inflexible, and mechanically plumbed to dedicated microplates, rendering them incapable of operating standard multi-purpose human-scale laboratory instruments (such as chemical fume hoods, rotary evaporators, benchtop centrifuges, analytical balances, and magnetic stirrers). Developing mobile dual-arm robotic chemists with human-scale dexterity and spatial mobility represents a pivotal breakthrough toward fully autonomous biochemical experimentation.

Nevertheless, transitioning collaborative dual-arm robots from structured industrial assembly lines into unconstrained wet-lab environments exposes severe physical and perceptual bottlenecks. The robotic platform must interact with completely transparent, specular glassware, execute precision mechanical assembly in narrow centrifuge slots, suppress dynamic liquid sloshing, and sustain multi-step execution over dozens of interconnected phases.

This project developed a full-stack mobile dual-arm robotic chemist system deployed on an omnidirectional mobile base paired with dual collaborative manipulators. Integrating multi-view geometric vision, kinematic demonstration retargeting, parameterized atomic skills, and explicit physical state contracts, the system accomplishes autonomous end-to-end wet-lab operations: handling fragile test tubes and beakers, microliter fluid transfers, and symmetrical dynamic counterbalancing in high-speed centrifuges.

Real Laboratory Challenges & Architectural Layers

Unlike structured factory assembly lines with rigid metal parts and dedicated fixturing, a biological or chemical laboratory imposes extreme demands on spatial perception, compliant bimanual dexterity, and execution fault tolerance:

System Layer	Core Engineering Challenge	Technical Solution
Perception & Proprioception	Transparent borosilicate glass and quartz vessels cause active depth sensors to suffer missing data and severe multipath reflections	Fusing multi-view 2D boundary contours, spatial cone geometry, and CAD priors to recover robust 6D grasping poses
Mobility & Workspace Layout	Mobile base docking drift and tight reachability constraints inside chemical fume hoods	Base navigation provides coarse staging; eye-in-hand cameras perform secondary visual servoing to optimize dual-arm manipulability
Precision Contact Skills	Brittle thin-walled glassware vulnerable to damage; narrow centrifuge slots require strict coaxial entry	Decoupling manipulation into parameterized atomic skill blocks governed by compliant admittance control and adaptive micro-motion search
Fluid Dynamic Stability	Dynamic fluid sloshing, surface tension meniscus clinging, and droplet spillage during rapid arm transit	Parameterizing jerk-bounded, smooth acceleration motion profiles paired with visual feedback during volumetric pouring
Agentic Planning & Recovery	Compounding multi-step execution errors across multi-hour experimental protocols	VLM-driven task hierarchy checking explicit pre/post physical state contracts before and after each action, triggering localized self-healing retries

System & Algorithmic Architecture

Experimental Intent → Skill Learning → Physical Execution → Closed-Loop Feedback

Mobile base traverses workstations; coordinated dual arms execute precision manipulation; hierarchical agents invoke skills with closed-loop physical verification.

Skill-learning route

01Demonstrate and mapTemporal motion estimation · human-to-robot mapping

02Learn across sim and realScene reconstruction · teacher/student policies · transfer

03Reusable skillsGrasp · place · pour · workstation interaction

Task-execution route

04Interpret task and sceneInstructions · object states · prerequisites

05Coordinate the whole bodyMobile base · two arms · reachability and contact

06Assess and recoverCheck completion · retry or replan

Execution feedback → scene updates and skill improvementResearch objective: use physical interaction failures to guide the next learning cycle.

Core Technical Solutions: From Physics Priors to Closed-Loop Self-Healing

To guarantee dependable unattended operation at the physical chemistry bench, we developed an integrated processing pipeline spanning multi-modal perception, ergonomic retargeting, smooth fluid control, and contract-governed self-healing.

1. Robust 6D Pose Estimation on Transparent & Specular Glassware

In wet laboratories, beakers, test tubes, and graduated cylinders are predominantly fabricated from transparent borosilicate glass or fused quartz. Structured-light active infrared patterns pass directly through the glass walls, generating complex internal refractive caustics that destroy active depth triangulation and yield massive depth dropouts.

To bypass active depth degradation, we constructed a multi-view boundary contour intersection and parametric CAD constraint pipeline:

Contour Extraction & Spatial Cone Intersection: Multi-angle high-resolution cameras capture glassware from calibrated vantage points. By back-projecting extracted rim contours into space, the system builds intersecting visual cones whose common intersection resolves the 3D position and surface orientation of the opening;
Parametric Model Alignment: Extending along the central symmetry axis, the system fits a cylindrical geometric model against valid local point clouds, bounding 6D pose errors within reliable tolerances for compliant grasping.

2. Kinematic Retargeting of Human Demonstration & Singularity Avoidance

Expert chemists handle volumetric pipettes, unscrew threaded caps, and pour liquids with innate smoothness and dexterity. To capture this expert intuition, we deployed an optical-inertial motion-capture demonstration rig. However, collaborative manipulators are strictly bounded by joint limits, link lengths, and kinematic singularities. Blind Cartesian waypoint replication frequently triggers joint velocity spikes or self-collisions against the workbench.

We engineered a task-space kinematic retargeting algorithm with manipulability optimization:

Invariant Feature Extraction: Isolating inter-vessel relative poses, approach/retreat normal vectors, and characteristic velocities while filtering non-rigid human tremor;
Manipulability Optimization: Resolving inverse kinematics by projecting manipulability criteria into the null space, ensuring the robot mirrors human bimanual coordination while operating within singularity-free, high-stiffness joint configurations.

3. Precision Alignment and Compliant Centrifuge Assembly

Benchtop centrifuges feature angled cylindrical rotor holes with tight assembly clearances. If the tube axis deviates upon entry, the tube rim binds against the rotor block, risking tube damage or joint overcurrent trips.

We designed a three-stage cascade assembly controller:

Coarse Staging: Following mobile base docking, the arms convey the tube to a staging plane directly above the designated rotor slot;
Eye-in-Hand Coaxial Servoing: High-resolution wrist cameras perform contour fitting on the rotor aperture, extracting relative lateral offsets and axial tilt, dynamically driving coaxial alignment errors toward rapid convergence;
Axial Compliant Insertion & Helical Micro-Search: The manipulator engages axial impedance control, translating vertically downward at low speed while monitoring contact force. If resistance indicates edge binding, the controller executes a localized micro-motion search, allowing the tube to seat cleanly and reliably into the rotor socket.

4. Fluid Motion Stabilization & Meniscus-Monitored Pouring

Transporting open beakers and vials introduces dynamic fluid sloshing, risking chemical spillage and center-of-mass perturbations:

Smooth Trajectory Generation: Filtering acceleration and deceleration commands with jerk-bounded smooth profiles tailored to container dimensions and fluid fill levels, eliminating resonance during high-speed transit and rapid directional changes;
Pouring Meniscus Vision Feedback: When pouring reagents between vessels, dual arms coordinate tilt angle while wrist cameras observe the fluid surface tension meniscus and droplet detachment, providing closed-loop control over volumetric transfer.

Beaker grasp and transfer

Slender-object grasping

Tube alignment and placement

5. Physical State Contracts & Long-Horizon Self-Healing Execution

In long-horizon biochemical workflows (e.g., retrieve vial, unscrew cap, weigh on balance, pipette reagent, vortex mix, symmetric centrifuge load, spin, separate supernatant), open-loop execution inevitably fails when a single micro-slip occurs.

We formulated a hierarchical self-healing execution engine governed by physical state contracts:

Hierarchical Task Orchestration: High-level experimental protocols are compiled into a topological graph of task milestones. A Vision-Language-Action agent handles multi-step sequence reasoning, mid-level coordinators instantiate parameterized skill primitives, and low-level controllers maintain high-frequency feedback loops;
Pre- and Post-Condition Physical Assertions: Every atomic skill is bound to deterministic multi-modal verification contracts (e.g., gripper aperture checks and weight step detection upon grasping, and diametric symmetry verification before centrifuge loading);
Localized Self-Healing Fallback Branches: If a post-condition verification fails, the agent does not abort the experiment. Instead, it activates targeted local recovery branches (open gripper, retract to safe observation pose, re-scan multi-view geometry, and re-attempt grasp with adapted parameters), substantially increasing overall protocol completion.

Real-Robot Experiments & Demos

Precision Tube & Beaker Coordinated ManipulationReal-time recording: the dual-arm robot autonomously identifies labware, executes stable adaptive grasps, and performs precision fluid transfer.

Autonomous Centrifuge Workstation OperationReal-time recording: the robotic arm localizes narrow centrifuge slots with high precision, executing smooth tube insertion and balanced loading.

The video recordings above show the robot in action at a real wet-lab bench:

Transparent Labware Retrieval & Fluid Transfer: Autonomous 6D pose extraction of transparent tubes and open beakers under complex lab lighting, followed by smooth bimanual liquid aspiration, transfer, and dispensing without sloshing or surface spatter;
Precision Centrifuge Workstation Manipulation: Sub-millimeter coaxial insertion of micro-tubes into tight, angled centrifuge rotor sockets, accompanied by symmetric counterbalance loading and smooth retraction out of the fume hood.

Engineering Summary & Technical Takeaways

Mobile dual-arm manipulation in biochemical laboratories represents the definitive proving ground for robotic dexterity bridging deterministic robotics and complex physical reality:

Overcoming Optical Degeneracy via Geometric Priors: Transparent and specular glassware defeats naive depth sensors. Fusing 2D sub-pixel boundary geometry, spatial cone intersections, and implicit 3D scene representations provides the prerequisite foundation for reliable physical contact.
Embedding Physical Dynamics into Trajectory Generation: Handling delicate glassware and fluid reagents requires multi-body dynamic constraints. Explicitly incorporating input shaping, slosh suppression, and force admittance into low-level controllers enables clean, professional-grade laboratory execution.
Explicit State Contracts as the Bedrock of Long Horizons: In open-ended, multi-step scientific workflows, open-loop assumptions are fatal. Arming every atomic manipulation skill with verifiable physical assertions and local recovery branches creates the physical closed loop essential for real-world laboratory automation.
Hardware-Software Co-Design for Unstructured Workspaces: In confined fume hood environments, decoupling mobile base global mobility from dual-arm local manipulability is essential for preventing joint limits and wrist singularities.

CUHK Collaboration: Mobile Dual-Arm Manipulation for Library Shelf Tidying and Book Reshelving

Mon, 21 Sep 2026 00:00:00 +0000

In semi-structured physical environments, returning a book to a shelf appears deceptively simple. For an autonomous mobile dual-arm robot, however, it presents severe multi-physics challenges spanning spatial geometry, contact dynamics, and long-horizon decision-making:

Books on a library shelf are packed tightly side by side, subject to substantial lateral compression and surface friction, leaving clearances often strictly in the range of 1 to 2 millimeters;
Book spines exhibit varied textures and reflective laminated plastic coverings, frequently inducing orientation estimation errors in optical 6D pose estimators;
If the mobile base stops merely a few centimeters off-target, the arm rapidly hits joint singularities or mechanical limits when reaching deep into the shelf slot, or occludes the optical line-of-sight of wrist-mounted cameras;
Extracting or inserting a book without lateral force regulation easily triggers a domino-style collapse of neighboring leaned volumes.

This project is an ongoing research collaboration with . We focus on mobile dual-arm manipulation for autonomous library inventory, shelf reshelving, and tidying. I lead the AI Agent decision-making and physical execution loop, investigating how high-level tidying goals translate into reliable, verifiable physical actions.

Project context

CUHK · Prof. Yunhui Liu's Group

Library book reshelving represents a challenging semi-structured long-horizon manipulation task: reflective laminated spines, tightly compressed leaning books, millimeter clearances, and tight coupling between base parking and arm manipulability.

Projected Goals & Research Focus: End-to-end workflow for dense shelf extraction and millimeter slot insertion; targeting >92% bimanual extraction success and >80% jam recovery success in anticipated hardware deployment.

My contributions

As project lead, established the mobile dual-arm manipulation framework; responsible for coaxial visual servoing and closed-loop agent execution, setting research targets for bimanual extraction (>92% success rate) and jammed slot recovery (>80% recovery rate) in 1.5–3.0 mm tight shelf clearances.

Physical Challenges and the Fundamental Paradigm Shift of Mobile Manipulation

Transitioning dual-arm manipulation from fixed industrial workcells to a mobile wheeled platform represents a fundamental paradigm shift in both theoretical formulation and physical execution. In a fixed workcell, an unobserved object is typically a localized line-of-sight occlusion problem; the arm base is rigidly anchored to the workstation, and inverse kinematics (IK) possesses a static, deterministic geometric reachability envelope. In mobile manipulation, however, the problem expands across three critical physical dimensions:

Environment-Scale Partial Observability: The target book may reside on an entirely different shelf rack, in an adjacent aisle, or placed in a temporary return cart by human visitors. The robot must not only infer localized occlusions within its current view frustum, but must also perform global spatial reasoning across rooms and aisles, planning active exploratory navigation paths while maintaining a persistent topological and object-centric memory graph.
Base Placement as a First-Class Decision: The terminal parking pose of the mobile base directly dictates the kinematic feasibility of all subsequent manipulation motions. Stopping too close to the shelf prevents the arm from unfurling its elbow joints, risking collisions with narrow aisle walls behind the robot; stopping too far leaves deep shelf slots unreachable. Furthermore, the base yaw angle directly constraints the line-of-sight of onboard cameras toward spine boundaries and target slot gaps. Base docking can never be relegated to an independent 2D navigation stack; it must be treated as a first-class decision variable driven by predicted manipulation payoff.
Uncertainty Accumulation across Navigation-Manipulation Cycles: After executing a long transit path and braking to a halt, wheel odometry slippage, mechanical gear backlash, and chassis suspension compliance introduce residual base pose errors on the order of 1 to 3 centimeters and several degrees in heading. Executing pre-planned arm trajectories open-loop from global navigation coordinates inevitably results in hard physical collisions. The system must enforce a deterministic sequence of secondary local perception, pose refinement, and compliant force-closed feedback.

Research architecture

Observe the shelf. Plan the placement.

Parking changes visibility and reachability. Re-observe after moving; verify after placing.

Observe and decide

01Identify books and slotsIdentity · spatial relations · occlusion

02Update the World ModelRegion memory · local state · uncertainty

03Select a pose and skillsAI Agent · visibility · reachability

Act and verify

04Navigate, park, re-observeCorrect parking error · align locally

05Extract and reshelveTwo-arm coordination · support · insertion

06Verify and recover locallyCorrect book · correct slot · bounded retries

Observations and outcomes → revised state and planSimulation and demonstrations support skill development; the control layer constrains physical motion.

Software-Hardware Division and the Deterministic Control Boundary

To deploy complex AI reasoning onto physical hardware safely, our architecture establishes an uncompromising division of responsibilities: the AI Agent and the World Model operate strictly at the task and skill level, governing high-level discrete state transitions, base parking candidates, verification scheduling, and bounded recovery policies. Low-level trajectory interpolation, kinematics, collision checking, torque limits, and emergency stopping are enforced deterministically by the robot control stack.

High-Level Agent Planning System: Implemented within the ROS 2 ecosystem, running a dedicated local language model adapter (agent_planner). It consumes the structured symbolic task state, registered skill definitions, and the object-centric world model, outputting structured plans strictly adhering to predefined JSON schema declarations.
Low-Level Deterministic Control Stack: Built upon ROS 2 and MoveIt 2 for kinematic computation and planning scenes. Free-space arm motions utilize OMPL for collision-free trajectory generation, while slot insertion and retraction strictly invoke Cartesian straight-line interpolation. Mobile base navigation relies on Nav2 maintaining layered costmaps, coordinated with an omnidirectional mobile chassis to provide smooth translational and rotational braking.
Onboard Compute and Real-Time Bus: Computation is hosted on an onboard industrial GPU workstation running perception segmentation, world model maintenance, local trajectory synthesis, and skill status monitoring. Low-level motor controllers communicate with arm joints and base wheel drives over high-speed real-time Ethernet buses, guaranteeing millisecond-level emergency stops whenever an anomalous wrench or deviation is detected.

Joint Optimization of Base Docking and Arm Manipulability

In narrow library aisles where aisle clearance is typically restricted to 0.8 to 1.0 meters, decoupling mobile base parking from manipulator trajectory generation causes severe kinematic deadlocks. We formulate base pose selection as a multi-objective constrained optimization problem, jointly considering arm manipulability (steering clear of singularities and travel limits), wrist camera line-of-sight margins (preventing link self-occlusion), and navigation travel and safety clearances.

Once the base completes navigation and comes to a complete halt, the robot triggers a two-stage local visual refinement routine: onboard stereo and wrist cameras reconstruct local shelf fiducials and target spine edges, computing an incremental correction matrix that absorbs the 1-3 cm base drift before any arm motion commences.

Coordinated Bimanual Manipulation and Compliant Contact Mechanics

The most fragile phase of library shelf manipulation lies in the lateral force interactions during book extraction and slot re-insertion. Under tight packing, pulling a book out with a single arm drags adjacent leaning volumes through surface friction, provoking domino-style collapses. Conversely, during insertion, even sub-degree angular misalignment locks the book against slot edges. We address this through a specialized bimanual strategy.

1. Stabilizing Arm: Lateral Preloading and Anti-Collapse Support

Prior to any contact between the primary manipulator and the target book, the secondary stabilizing arm extends and positions a compliant, high-friction pad against the outer edge of the adjacent leaning book cluster, applying a regulated lateral preload (typically maintained at 5 N to 8 N):

This steady normal force functions as an artificial rigid sidewall, counteracting lateral friction forces induced when the target volume slides out;
A 6-axis force-torque sensor at the stabilizing wrist continuously tracks lateral reaction forces. If anomalous cluster slippage is detected, the controller adaptively adjusts contact depth to preserve the quasi-static equilibrium of the shelf.

2. Primary Manipulating Arm: Tangential Compliant Extraction

The primary arm is equipped with a slim, adaptive parallel-jaw gripper. Perception algorithms locate the upper and lower spine vertices alongside the surface normal, guiding the gripper tips to slide tangentially into the narrow book gaps:

Gripper closing combines motor current sensing with optical displacement feedback, clamping the book spine firmly without crushing fragile paper covers;
Extraction proceeds along the shelf normal. The manipulator operates in an admittance-controlled mode along lateral (Y) and pitch axes, allowing minor passive compliance to accommodate slight alignment errors without tearing delicate book jackets.

3. Tilted Micro-Search and Compliant Slot Insertion

Re-inserting a book into an ultra-narrow slot represents the highest contact-dynamic hurdle. We implement a three-phase insertion protocol:

Tilted Initial Contact: The bottom corner of the book is tilted at a slight angle of 12 to 15 degrees, reducing the initial contact geometry from an area contact to an edge-point contact, substantially diminishing sliding resistance;
Admittance Control and Compliant Search: In the Cartesian task frame, the manipulator tracks a compliant admittance model. The forward insertion axis is parameterized with low virtual stiffness, automatically slowing down upon sudden thrust increases while applying subtle oscillatory dither motions to guide the book along the natural boundaries of the slot;
Full Home Seating: When reaction forces drop to nominal levels and optical distance sensors confirm the spine is flush with the shelf plane, the gripper releases smoothly and retracts.

Object-Centric Shelf World Model

Maintaining the physical state of dozens of individual books across an aisle exceeds the capability of raw dense voxel grids or unstructured point clouds, which incur prohibitive computational overhead and lack semantic handles. We developed an object-centric shelf world model specifically tailored for dense manipulation.

1. Structured Node Representations

The world model represents the environment as an explicit geometric-semantic heterogeneous graph. Each node corresponds to an individual book entity or shelf partition unit:

Spatial Geometry: 3D oriented bounding boxes (3D OBB), 6D spine reference frame, and posterior pose covariance $\Sigma_p$;
Semantic Identity: Optical character recognition (OCR) spine text, barcode/RFID identifiers, and visual feature embeddings;
Topology and Contact: Left neighbor ID, right neighbor ID, supporting shelf plane, tilt angle $\phi_{\text{tilt}}$, and lateral tightness metric.

2. Multi-View Spatial-Temporal Data Association

As the mobile base traverses the aisles, incoming point clouds and RGB streams are continuously associated with existing graph nodes via Hungarian matching over a cost matrix combining 3D Euclidean proximity, visual appearance similarity, and relative topological sequence. When an existing book is re-observed from a different vantage point, a Kalman filtering pipeline updates its 3D pose estimate and reduces uncertainty ellipses. When a previously occupied coordinate is observed to be empty, the model updates the node state to a vacant slot.

3. Counterfactual Simulation for Action Safety

The world model enables the AI Agent to conduct counterfactual forward lookahead before executing physical actions:

Evaluating whether the manipulator’s swept volume will violate collision margins with shelf uprights from a candidate base pose;
Simulating whether extracting a target volume will destabilize adjacent volumes based on gravity vectors and friction coefficients;
If counterfactual analysis indicates collapse risk, the agent proactively generates a coordinated bimanual plan incorporating the stabilizing arm rather than attempting single-arm manipulation.

AI Agent Task Orchestration and Bounded Error Recovery

As lead of the AI Agent workstream, I developed a unified task abstraction layer bridging high-level semantic reasoning with low-level execution via Behavior Trees (BehaviorTree.CPP) and parameterized skill contracts.

1. Structured Task Planning and Schema Validation

When presented with open-ended natural language instructions (such as “Retrieve three misplaced computer science books from the return trolley, place them onto Shelf B2, and tidy the leaning stack”):

The onboard agent_planner digests the symbolic state from the world model (including robot region, shelf vacancies, and trolley inventory) through a local language model, decomposing the directive into a directed acyclic task graph;
Action outputs are restricted to registered skills: NavigateToArea, DockAtShelf, ScanSpines, StabilizeCluster, ExtractBook, VerifyHolding, InsertBook, RetractArm;
A deterministic schema validator verifies that all parameters, coordinate frames, and target IDs are mathematically sound and physically present before generating the reactive behavior tree.

2. Explicit Physical Verification Contracts

To avoid catastrophic cascading failures caused by open-loop assumptions, each atomic skill is bound by strict sensor-driven preconditions and postconditions:

For ExtractBook, preconditions require that the stabilizing arm has verified normal force balance and the gripper is aligned with the spine axis;
Postconditions require multi-modal verification: optical beam interruption on the gripper fingers combined with finger encoder positions confirming that the gripped width corresponds to the expected book thickness. If verification fails, the skill returns an explicit failure state, halting forward progression.

3. Multi-Tier Bounded Error Recovery

Mechanical binding during slot insertion is an inherent physical phenomenon due to paper curvature and localized friction. We embed bounded recovery routines:

Local Micro-Retreat: If insertion force exceeds safety limits (such as thrust exceeding 25 N) prior to target depth, the behavior tree triggers a local recovery branch: retracting 20 mm, applying a 1.5-degree yaw adjustment, and re-attempting compliant insertion;
Bounded Escalation and Safe Abort: If three consecutive recovery attempts encounter persistent jamming, the system terminates insertion, pulls the book back safely, transfers it to the onboard carriage, marks the slot as jammed in the world model, logs the anomaly, and proceeds to the next item, preventing a localized mechanical failure from paralyzing the overarching mission.

Progressive Milestones & Projected Performance Targets

The framework is architected across four difficulty milestones (M1 to M4):

M1: Meeting-Room Clearing. Validating natural language instruction parsing, reachability-driven docking, basic single- and dual-arm manipulation, and onboard carriage payload handling in open environments;
M2: Library Shelf Tidying and Reshelving. Introducing narrow aisle navigation, dense book extraction under compression, millimeter-clearance compliant insertion, bimanual stabilization, and jammed-slot recovery;
M3: Traceable High-Value Transport. Adding sealed container handling, delicate vial grasping, barcode/RFID verification, and audit logging;
M4: Unseen Open-World Generalization. Evaluating long-horizon stability and object-centric topological reasoning across novel room configurations without prior maps.

For M2 dense shelf tidying simulation and experimental benchmarking, the project defines concrete target objectives:

In dense shelf arrays with clearances between 1.5 and 3.0 mm, coordinated bimanual extraction targets an overall success rate exceeding 92%, aiming to prevent neighbor domino collapses;
Base docking optimization targets placing the manipulator in high-dexterity operating envelopes on >88% of initial attempts, minimizing realignment latency;
Under induced binding disturbances, bounded error recovery is designed to resolve over 80% of frictional jams without human intervention.

Engineering Lessons and Technical Takeaways

This project provided deep practical insights into the physical realities of mobile dual-arm manipulation in dense environments:

The Inviolable Role of Deterministic Control: While modern end-to-end vision-language-action models offer broad generalization, their high-frequency output jitter remains hazardous during tight-clearance physical contact. Confining AI models to discrete task planning and counterfactual reasoning while delegating low-level execution to classical admittance control, OMPL, and behavior trees provides the necessary foundation for real-world hardware reliability.
Mobility and Manipulation Must Be Unified: A mobile manipulator is fundamentally more than a robotic arm mounted on a mobile platform. Treating base docking as an independent navigation waypoint is the primary cause of kinematic singularities and visual occlusions. Deriving optimal base docking manifolds directly from manipulation constraints translates mobility into an active extension of the arm’s workspace.
Physical Robustness Arises from Rigorous Error Defense: In long-horizon real-world operations, expecting zero physical failures is unrealistic. True system resilience lies not in preventing all errors, but in verifying physical states against reality and executing bounded local recovery when anomalies occur. The architectural paradigms developed here provide enduring foundations for dense shelf handling, semi-structured mobile manipulation, and embodied robotic systems.

Warwick Collaboration: Decoupled Hierarchical Embodied AI and Agentic Robotics for Long-Horizon Tasks

Mon, 21 Sep 2026 00:00:00 +0000

In embodied robotic manipulation within open, unstructured environments, robots face profound physical uncertainty. Whether performing multi-step industrial assembly, automated laboratory specimen preparation, or parcel sorting in semi-structured warehouses, practical missions frequently encompass dozens of consecutive physical interactions. In such extended horizons, minor unmodelled errors introduced during any individual action compound exponentially, precipitating global task failure if left unmitigated.

This project is an ongoing research collaboration with the robotics and sensor systems group at the University of Warwick. We focus on developing a hierarchical embodied AI and closed-loop AI Agent architecture designed specifically for long-horizon physical manipulation. I am responsible for designing and deploying the full-stack software and hardware robotic system, bridging multimodal perception, high-level task decomposition, metric geometric motion synthesis, and low-level Cartesian impedance control to establish an autonomous operating pipeline with physical self-healing capabilities.

Project context

University of Warwick · Professor Sai Gu

In complex real-world environments, pure end-to-end models struggle to balance multi-minute logical consistency with high-frequency geometric control. We develop a decoupled hierarchical framework where high-level agents orchestrate subgoals while low-level controllers handle geometry and contact dynamics.

Projected Objectives & Research Focus: Closed-loop 8–15 step physical embodied manipulation; targeting >84% end-to-end success, >70% localized recovery, and <3 mm 3D localization error under multimodal fusion.

My contributions

As project lead, formulated the hierarchical agent manipulation architecture; for 8–15 step long-horizon tasks, established target objectives to elevate end-to-end task success rate from 35% to >84%, with >70% autonomous recovery from localized physical anomalies.

Physical Brittleness and the Challenge of Compound Error Propagation

In embodied manipulation, the fundamental barrier to scaling long-horizon tasks lies in compound error propagation. Consider a representative physical task composed of 10 sequential manipulation stages (e.g., identifying a chemical vial on a cluttered bench, uncapping the vessel, aspirating a reagent, re-threading and torquing the cap, transferring the vessel to an assay tray, and stowing it in an incubator). Even if every individual atomic skill possesses an isolated execution reliability of 90%, under open-loop execution the cumulative reliability decays exponentially to roughly 35% across 10 stages, and even further as steps increase. This compounding decay demonstrates that without explicit physical verification contracts and bounded local recovery mechanisms, open-loop robotic systems are virtually guaranteed to fail midway. Contemporary research exhibits two prevalent paradigms, both of which reveal distinct physical vulnerabilities when applied to real-world hardware:

Vulnerabilities of Pure End-to-End Visuomotor Policies (VLA Models): End-to-end policies excel at reactive short-horizon reflex behaviors within bounded training distributions. However, across operational timelines spanning several minutes and dozens of diverse actions, latent vector representations suffer from severe temporal drift. Critically, contemporary vision-language-action models operate at inference rates between 5 Hz and 20 Hz, which is fundamentally inadequate for the hundreds-of-hertz control loops required for stable contact dynamics. Furthermore, when unexpected mechanical binding or slippage occurs, end-to-end models lack explicit mechanisms for backtracking, fault localization, and structured error recovery.
Geometric Hallucinations in Direct LLM Waypoint Generation (LLM-as-Planner): Inquiring large language models to directly generate end-effector SE(3) waypoints or joint trajectories yields semantic commonsense but lacks metric spatial reasoning and inverse kinematics (IK) feasibility awareness. The resulting trajectories routinely violate joint velocity or torque limits, penetrate obstacle envelopes, or command configurations within singular regions. Crucially, direct text generation fosters an open-loop hallucination that issuing an action directive is physically equivalent to successfully completing the interaction.

To overcome these structural limitations, our architectural paradigm enforces hierarchical decoupling between task planning and motion synthesis, combined with strict closed-loop physical state verification. High-level symbolic reasoning is isolated within the AI Agent, high-frequency trajectory tracking and contact compliance are governed by deterministic real-time controllers, and multimodal sensing provides empirical ground truth across every execution transition.

Research architecture

3D perception → hierarchical decisions → robot execution

Task planning and motion control coordinate through observations and execution state.

Sensing and spatial understanding

01Vision and mmWaveComplementary observations · multimodal sensing

023D scene representationObjects · geometry · spatial relations

03Language and temporal groundingInstruction meaning · relevant events

AI Agent-based robot system

04High-level task decisionsInterpret instructions · organize subgoals

05Low-level executionRobot skills · motion control

06Feedback and updatesObserved outcomes · continued planning

My responsibility: build the robot system linking perception, planning, and executionIntegrate the collaboration's research components in a shared robot task loop.

Three-Tier Decoupled Embodied AI Architecture

To achieve semantic adaptability while preserving physical execution determinism, the system is organized into three strictly partitioned hierarchical layers:

1. Level 3: Semantic and Long-Horizon Task Planning Layer (AI Agent Brain, 0.5 Hz - 2 Hz)

Responsible for parsing open-ended natural language directives or complex procedural task objectives into a directed acyclic task graph (DAG) or a reactive behavior tree:

The agent operates via an onboard language model adapter, consuming symbolic state inputs comprising the robot’s current status, a spatial-semantic scene graph, and temporal execution history;
Candidate agent outputs are strictly restricted to a registered library of formally verified parameterized atomic skills (such as ApproachObject, AlignEndEffector, ExecutePinchGrasp, VerifyEngagement, LinearExtract);
Level 3 produces no direct motor current or joint torque commands; its decision manifold resides entirely within the domain of task sequencing, skill invocation, and verification scheduling.

2. Level 2: Metric-Geometric Skill Synthesis Layer (Skill Synthesis & Trajectory Generator, 10 Hz - 50 Hz)

Translates parameterized skill specifications from Level 3 into metric SE(3) Cartesian trajectories and joint space paths:

Ingesting a dynamic Euclidean Signed Distance Field (ESDF) maintained from onboard depth cameras and radar point clouds, the layer invokes OMPL and the Pinocchio rigid-body dynamics library to compute time-parameterized polynomial trajectories compliant with joint velocity and acceleration bounds;
For contact-intensive phases (such as drawer sliding, cap threading, or peg-in-hole insertion), the planner seamlessly switches to Cartesian linear interpolation and helical micro-search primitives, continuously auditing kinematic continuity and self-collision clearance.

3. Level 1: High-Frequency Compliant Execution Layer (Deterministic Real-Time Control, 500 Hz - 1000 Hz)

Serves as the low-level hard real-time foundation, driving joint actuators over deterministic buses while governing physical contact forces:

Executes Cartesian space impedance control: in free-space transit, stiffness is elevated to guarantee tracking accuracy; during physical contact, virtual stiffness is attenuated along contact axes to provide compliant deflection, absorbing positional misalignments;
Actuator currents and 6-axis wrist force-torque sensors are queried at 1 kHz against hardware-level safety thresholds. In the event of an anomalous collision wrench, the controller triggers a controlled stop within 2 milliseconds, preventing hardware damage.

Multimodal 3D Spatial Perception: Visual-mmWave Radar Fusion

In unstructured operational environments, relying exclusively on optical vision exposes the robotic system to catastrophic sensor degradation. In backlit conditions, structured-light and active stereo depth sensors suffer pattern saturation, inducing substantial depth dropouts. On transparent glassware or specular metallic tooling, specular reflections cause false point-cloud shifts of several centimeters. Furthermore, airborne dust or aerosol suspensions completely attenuate optical propagation.

Leveraging Warwick’s extensive expertise in advanced electromagnetic sensing and industrial digitalization, we developed a synchronized RGB-D and 60-77 GHz FMCW Millimeter-Wave (mmWave) Radar spatial perception architecture:

1. Complementary Physical Sensing and Microsecond Synchronization

Optical Sensors: Deliver dense, megapixel-resolution RGB imagery with fine contour boundaries and rich semantic segmentations, capturing micro-scale visual geometry, yet remain highly vulnerable to ambient illumination and optical occlusions;
Millimeter-Wave Radar: Exploits millimeter-range electromagnetic wavelengths that penetrate dust, smoke, and moisture, remaining impervious to illumination extremes; FMCW radar directly returns radial distance and high-precision Doppler velocities, though it exhibits coarse angular resolution and susceptibility to multipath ghost reflections;
Spatial-Temporal Calibration: Utilizing specialized retro-reflective geometric calibration targets, we solve for the rigid spatial extrinsic transformation matrix; hardware synchronization utilizes a Pulse-Per-Second (PPS) reference signal, constraining temporal projection jitter below 1 millisecond.

2. Kinematic Clutter Filtering and Log-Odds Volumetric Occupancy Mapping

To eliminate multipath clutter, the perceptual front-end applies Constant False Alarm Rate (CFAR) adaptive thresholding alongside Doppler velocity distribution analysis, isolating static structural landmarks from moving obstacles.

For volumetric mapping, we maintain a 3D log-odds temporal voxel occupancy grid, fusing optical depth projections with radar backscatter cross-sections via a recursive Bayesian framework. Through this design, when optical cameras experience blinding glare, radar range returns maintain high occupancy confidence, preventing the manipulator from colliding with undetected obstacles; conversely, when radar returns exhibit angular blooming near metallic edges, optical high-resolution masks delineate crisp object surfaces, establishing an all-weather geometric foundation.

Video Temporal Grounding and Physical State Verification

In long-horizon execution, verifying that a preceding manipulation stage has successfully attained its objective is the cornerstone of closed-loop robustness. If the system relies purely on elapsed trajectory time or encoder confirmations to assert success, undetected slippage or jamming will cascade through downstream tasks.

We incorporate a video-semantic grounding mechanism: throughout execution, the system maintains a circular video buffer $\mathcal{V}_{[t-\Delta t, t]}$ spanning window length $\Delta t$. The high-level agent establishes explicit condition descriptors for each sub-action (such as “vessel base lifted completely clear of the tabletop” or “slider rail pulled to terminal mechanical stop”). Utilizing cross-modal cross-attention layers, the model grounds semantic milestones along the temporal video timeline, extracting verifiable timestamped evidence of physical state transitions.

To guard against optical occlusions, every atomic skill is bound to rigorous multi-modal physical assertions:

Visual Assertions: The 3D bounding box centroid of the target object must exhibit a rigid vertical translation exceeding threshold $\Delta z > h_{\text{thresh}}$, accompanied by the visual reappearance of the supporting tabletop texture;
Force-Dynamic Assertions: The wrist 6-axis force-torque sensor must register a vertical gravitational wrench increase $\Delta F_z$ concordant with the object’s calibrated mass ($ \Delta F_z \approx m_{\text{object}} \cdot g \pm \epsilon_f $), while internal gripper finger encoders confirm a grasp aperture matching the item’s metric profile;
Deterministic Gating: Only when both visual geometric evidence and force metrics concurrently satisfy verification criteria does the skill node return a SUCCESS token to the behavior tree. If either assertion fails, execution is immediately intercepted.

3. Hierarchical Bounded Error Recovery

Upon verification failure, the system avoids indiscriminate aborts or blind task restarts. Instead, it triggers localized bounded recovery trees based on the failure classification:

Incipient Slip and Empty Grasps: If force sensing indicates an unweighted gripper, slippage is identified. The behavior tree executes a local re-observation and grasp re-estimation routine, adjusting gripper approach yaw and re-attempting acquisition;
Mechanical Binding and Over-Torque: During insertion or mating, if resistive wrenches surge prior to reaching target depth, alignment binding is diagnosed. The controller executes a structured sequence: retract 15 mm, apply micro-scale dither perturbations to find the slot axis, and resume compliant low-stiffness insertion;
Bounded Escalation and Safe Holding: If repeated recovery attempts reach a pre-defined threshold (e.g., 3 attempts) without resolution, the system gently places the object into a designated buffer dock, decrements object confidence within the scene graph, logs time-series diagnostic data, and switches to an alternate independent subtask, preserving overarching operational uptime.

System Integration & Projected Target Objectives

The system architecture combines a visual-mmWave multimodal perception rig, a hierarchical behavior tree engine, and a 6-DOF compliant manipulator, establishing clear target benchmarks across 8-to-15-step sequential physical manipulation tasks:

Long-Horizon End-to-End Target Completion: By enforcing physical verification contracts and bounded error recovery, the framework targets elevating end-to-end multi-step task success from an open-loop 35% baseline to >84%, with an objective of autonomously resolving >70% of localized execution anomalies without human intervention;
Perception Robustness Objectives under Degraded Conditions: In harsh illumination and airborne particulate environments, the visual-radar fusion pipeline aims to constrain 3D boundary and distance tracking RMSE within 3 millimeters;
Geometric and Kinematic Consistency: The Level 2 motion planning layer is architected to guarantee 100% inverse kinematics solvability and boundary clearance across all agent-synthesized directives, eliminating joint limit collisions.

Engineering Lessons and Technical Takeaways

Developing this hierarchical embodied architecture yielded several fundamental engineering insights into robotic deployment:

Embodied AI Is Fundamentally an Engineering Discipline in Managing Uncertainty: In purely virtual or simulation environments, action outcomes are conveniently abstracted into deterministic transitions. In the physical world, mechanical tolerances, surface friction, thermal drift, and lighting shifts are persistent realities. A robust embodied architecture succeeds not by naively assuming a perfect environment, but by adopting a defensive posture that rigorously audits physical states after every contact and provides deterministic recovery pathways when deviations occur.
Language Models Excel at Symbolic Reasoning; Continuous Mechanics Belongs to Physics-Based Control: Large language models possess unmatched breadth in semantic parsing and long-horizon task structuring. However, forcing them to output high-frequency continuous joint commands is structurally unsound. Positioning the language model as a high-level strategist while assigning low-level trajectory generation, collision checking, and impedance control to deterministic robotics libraries achieves the optimal balance between open-world intelligence and physical safety.
Sensor Diversity Requires Orthogonal Physical Modalities: Redundantly stacking identical optical sensors fails to resolve structural weaknesses such as ambient backlighting, optical glare, or particulate obscuration. Incorporating millimeter-wave radar introduces an orthogonal electromagnetic sensing modality that compensates for fundamental optical blind spots, raising the lower bound of perceptual reliability in harsh, real-world manipulation environments.

Embodied AI | Chang Nie

A Robotic Chemist: Full-Stack Dual-Arm Mobile Manipulation for Biochemical Experiments

Real Laboratory Challenges & Architectural Layers

Core Technical Solutions: From Physics Priors to Closed-Loop Self-Healing

1. Robust 6D Pose Estimation on Transparent & Specular Glassware

2. Kinematic Retargeting of Human Demonstration & Singularity Avoidance

3. Precision Alignment and Compliant Centrifuge Assembly

4. Fluid Motion Stabilization & Meniscus-Monitored Pouring

5. Physical State Contracts & Long-Horizon Self-Healing Execution

Real-Robot Experiments & Demos

Engineering Summary & Technical Takeaways

CUHK Collaboration: Mobile Dual-Arm Manipulation for Library Shelf Tidying and Book Reshelving

Project context

My contributions

Physical Challenges and the Fundamental Paradigm Shift of Mobile Manipulation

Software-Hardware Division and the Deterministic Control Boundary

Joint Optimization of Base Docking and Arm Manipulability

Coordinated Bimanual Manipulation and Compliant Contact Mechanics

1. Stabilizing Arm: Lateral Preloading and Anti-Collapse Support

2. Primary Manipulating Arm: Tangential Compliant Extraction

3. Tilted Micro-Search and Compliant Slot Insertion

Object-Centric Shelf World Model

1. Structured Node Representations

2. Multi-View Spatial-Temporal Data Association

3. Counterfactual Simulation for Action Safety

AI Agent Task Orchestration and Bounded Error Recovery

1. Structured Task Planning and Schema Validation

2. Explicit Physical Verification Contracts

3. Multi-Tier Bounded Error Recovery

Progressive Milestones & Projected Performance Targets

Engineering Lessons and Technical Takeaways

Warwick Collaboration: Decoupled Hierarchical Embodied AI and Agentic Robotics for Long-Horizon Tasks

Project context

My contributions

Physical Brittleness and the Challenge of Compound Error Propagation

Three-Tier Decoupled Embodied AI Architecture

1. Level 3: Semantic and Long-Horizon Task Planning Layer (AI Agent Brain, 0.5 Hz - 2 Hz)

2. Level 2: Metric-Geometric Skill Synthesis Layer (Skill Synthesis & Trajectory Generator, 10 Hz - 50 Hz)

3. Level 1: High-Frequency Compliant Execution Layer (Deterministic Real-Time Control, 500 Hz - 1000 Hz)

Multimodal 3D Spatial Perception: Visual-mmWave Radar Fusion

1. Complementary Physical Sensing and Microsecond Synchronization

2. Kinematic Clutter Filtering and Log-Odds Volumetric Occupancy Mapping

Video Temporal Grounding and Physical State Verification

1. Cross-Modal Video Temporal Grounding

2. Multi-Modal Physical Verification Contracts

3. Hierarchical Bounded Error Recovery

System Integration & Projected Target Objectives

Engineering Lessons and Technical Takeaways