Teach and Grow: An Agent-Centered Architecture for General Robot Learning

publications

A Few Demonstrations Become Skills That Stay

Teach-and-Grow Learning (TGL) establishes an AI Agent and GPT-6 centered robotic manipulation architecture for general-purpose physical autonomy. Operating under the principle that pretrained foundation model weights remain entirely frozen, the framework places GPT-6 at the center of the embodied reasoning loop. The surrounding AI Agent interprets multi-modal spatial cues, decomposes complex long-horizon tasks, grounds high-level semantics into metric SE(3) geometries, and validates physical state transitions before accumulating verified skills into persistent dual-memory repositories.

Teach → Act and verify → Grow. Sparse demonstrations provide structural subgoal decompositions and causal dependencies; real-time physical feedback governs closed-loop execution and error recovery; and persistent, decoupled dual memory transforms the verified takeaways of preceding tasks into immediate springboards for subsequent unfamiliar missions, entirely eliminating the dependency on perpetual fine-tuning and weight optimization.

Read the paper · arXiv · Project website and demonstrations · Open Source Code

Where End-to-End Robot Learning Gets Expensive: The Retraining Tax

Contemporary research in deep imitation learning and end-to-end Vision-Language-Action (VLA) models relies primarily on gradient descent across high-dimensional parameter spaces $\theta \in \mathbb{R}^D$. When deploying these monolithic models into real-world, unstructured operational settings, however, a profound structural dilemma arises:

Whenever a robot encounters a novel container geometry, an uncalibrated camera perspective, highly specular surface finishes, or minor grasp slips, the actual metric variation may be minor. Yet, within a monolithic parameter-space paradigm, remediating this isolated deficiency typically necessitates collecting dozens or hundreds of additional expert trajectories, followed by computationally intensive joint multi-task optimization across the entire legacy training corpus. Worse still, because deep neural representations are heavily entangled across latent feature dimensions, adjusting parameter weights to accommodate new tasks frequently provokes unconstrained representation drift, causing previously mastered competencies to degrade unpredictably. We formalize this recurring computational, data collection, and verification overhead as the Retraining Tax.

Physical interaction data possesses a fundamentally distinct economic cost structure compared to internet-scale textual tokens or static web images. Physical contact cannot be synthetically generated losslessly via naive auto-regressive sampling; grasp stability must be empirically tested within a 3D dynamical workspace; and state changes in support relationships and surface friction must be validated through physical sensors. Geometric poses, background clutter, frictional coefficients, and manipulator nonlinear dynamics interact continuously. Covering the independent marginal distributions of isolated physical factors is fundamentally insufficient to cover their high-dimensional joint combinatorial distributions.

TGL addresses this foundational bottleneck: Can we decouple the acquisition of new robotic tasks from parameter-space gradient updates? Can we preserve physically validated manipulation experiences as explicit, inspectable objects endowed with verifiable geometric applicability bounds, deterministic preconditions/postconditions, and modular error-recovery rules? Under this paradigm, robotic foundation models centered on GPT-6 within an AI Agent serve as zero-shot semantic reasoning and geometric perception backbones, while TGL restructures how task knowledge is stored, audited, and retrieved within the robot system.

View full-size image ↗

From Planning AI Agents to GPT-6 Centered Robotic Manipulation

This research originated from investigating how tool-using AI Agent architectures could directly actuate robot manipulators to accomplish open-ended manipulation tasks. Placing GPT-6 at the core of the embodied agent provides unprecedented physical causality modeling and world common-sense reasoning. Paired with open-vocabulary perception, 6D pose estimation, and kinematic motion planning tools, a GPT-6 centered AI Agent can assemble feasible action sequences through exploratory trial-and-error even in the complete absence of prior demonstrations.

However, relying exclusively on instantaneous agent deliberation exposes severe operational limitations: once an episode terminates and the object is placed, the system retains no persistent, deterministic knowledge. When confronted with an identical or closely related task in subsequent trials, the agent must invoke slow, high-latency language model inference anew, searching through vast combinatorial action spaces from scratch. Furthermore, physical hurdles encountered during previous runs (such as slippery rims on thin glassware or localized mechanical binding in drawer rails) are discarded alongside the context window.

TGL integrates GPT-6 into the AI Agent cognitive center, retaining the agile observe-reason-act-verify loop while coupling it to an evolving lifelong memory architecture. Successful task executions are consolidated into formal Skill Blocks governed by rigorous execution contracts, while unexpected execution faults, root-cause diagnostics, and successful recovery sequences are archived as contextual exemplars within Experience Memory. In downstream tasks, the robot prioritizes retrieving verified, high-confidence skills, invoking full agent deliberation only when genuine out-of-distribution variations occur.

Within this framework, sparse demonstrations (human teleoperation logs, simulated trajectories, or instructional procedural text) function as an accelerator rather than an immutable prerequisite. Demonstrations supply a macroscopic temporal sequence of subgoals, confining exploratory search to critical geometric transitions and slashing trial-and-error overhead.

Formal Modeling: Executable Capabilities as Explicit Skill Blocks

To isolate invariant behavioral strategies from transient physical variations specific to a single environment, TGL introduces the formal abstraction of the Skill Block. Consider the task of picking up a ceramic bowl and placing it onto an adjacent tray. Across diverse demonstrations, the manipulator’s Cartesian trajectories, joint velocities, and approach angles vary widely. However, the underlying causal structure remains invariant: acquire the designated bowl, establish a stable grasp, transfer toward the target placement relation, and compliantly release contact.

Mathematically, we formalize a Skill Block $S_k$ as a 6-tuple:

$$S_k = \langle \mathcal{G}_k, \mathcal{P}_{\text{pre}}, \mathcal{P}_{\text{post}}, \Pi_{\text{exec}}, \mathcal{V}_{\text{eval}}, \mathcal{R}_{\text{recov}} \rangle$$

Where:

• $\mathcal{G}_k$ denotes the semantic and geometric subgoal targeted by the skill;
• $\mathcal{P}_{\text{pre}}$ represents the explicit precondition manifold required for skill triggering (e.g., target object kinematically reachable within the workspace envelope, gripper fully open, approach vector free of static occlusions);
• $\mathcal{P}_{\text{post}}$ defines the postcondition assertions that must hold upon completion (e.g., vertical object centroid displacement exceeding a threshold, gripper aperture matching object width);
• $\Pi_{\text{exec}}$ denotes the family of compatible parameterized low-level executors (e.g., Contact-GraspNet antipodal samplers, Cartesian straight-line admittance controllers, or specialized local visuomotor policies);
• $\mathcal{V}_{\text{eval}}$ represents the multi-modal physical verification operator responsible for outputting a deterministic success evaluation;
• $\mathcal{R}_{\text{recov}}$ denotes the bounded recovery policy branch specifying local retreat and compensation trajectories upon anomalous execution.

Here, training-free has a precise mathematical meaning: acquiring, validating, and consolidating an incoming task involves zero gradient backpropagation or weight optimization across underlying neural networks. Upstream foundation models and downstream geometric tools remain frozen; capability growth is realized strictly through explicit topological expansion and parameter synthesis within the skill library and memory network.

View full-size image ↗

Subgoal Reasoning on Top, Metric Geometry and Control Underneath

TGL enforces a disciplined division of responsibilities across hierarchical tiers, ensuring high-level semantic generalization while maintaining low-level physical precision:

1. Cross-Modal Demonstration Parsing and Subgoal Alignment

The AI Agent segments incoming multimodal demonstrations along boundaries of physical state transitions. Rather than naively memorizing time-series joint waypoints, the system identifies structural shifts in spatial contact topology (e.g., free-space transit $\to$ pre-contact alignment $\to$ stable dynamic constraint $\to$ release and decoupling). Although kinematic speeds and approach vectors vary, shared topological state transitions define the structural skeleton of the candidate Skill Block.

2. Scene Grounding and Metric Geometric Mapping

When the agent schedules a Skill Block, its abstract causal parameters are mapped to the local 3D workspace:

• Perception and Segmentation: Onboard RGB-D sensors capture point clouds and color imagery, invoking open-vocabulary detectors and segmenters to isolate target geometric meshes;
• Grasp Affordance Synthesis: Contact-GraspNet densely evaluates force-closure grasp candidates over the 3D surface mesh, selecting antipodal normals that maximize contact stability;
• Collision-Free Motion Planning: MPLib and OMPL motion planners generate kinematically continuous trajectories that navigate around obstacles represented in a local Euclidean Signed Distance Field (ESDF).

3. Sensor-Driven Physical Verification Contracts

Following trajectory execution, the system forbids blind open-loop transitions. Instead, the multi-modal verification operator $\mathcal{V}_{\text{eval}}$ audits execution states against empirical ground truth:

• For GraspObject, reaching commanded joint waypoints is insufficient. The controller concurrently evaluates gripper motor current feedback, infrared optical beam interruptions between gripper fingers, and wrist 6-axis force-torque wrench differentials. Only when force signatures confirm the gravitational payload of the target object does the postcondition pass;
• If verification registers an unladen gripper, an incipient slip is flagged, dynamically routing execution to the bounded recovery branch $\mathcal{R}_{\text{recov}}$ to adjust approach yaw and re-attempt grasp acquisition locally, preventing error compounding.

View full-size image ↗

Dual Memory Architecture: Skill Library and Experience Memory

To balance generic capability retrieval with specialized domain adaptation, TGL structures knowledge across two distinct, complementary repositories:

1. Skill Library ($\mathcal{L}_{\text{skill}}$)

The Skill Library maintains a curated catalog of validated atomic skill templates. Each entry encapsulates formal subgoal specifications, geometric precondition checkers, scene grounding rules, and compatible motion planner interfaces. The library prioritizes abstraction and domain independence, allowing skills to be instantiated across diverse objects and functional workspaces.

2. Experience Memory ($\mathcal{M}_{\text{exp}}$)

Experience Memory functions as an episodic, case-based repository that logs contextual metadata and diagnostic histories from real-world execution episodes:

• Catalogs ambient lighting parameters, surface material properties, grasp quality scores, and empirical execution outcomes;
• Archives failure traces alongside successful recovery trajectories (e.g., recording that lateral pinch grasps on polished metallic cylinders induced rotational slip, whereas a vertical top-down pinch established stable force closure);
• When the robot encounters analogous operational contexts in future missions, Experience Memory injects contextual priors into the planning prompt, biasing the agent away from historically vulnerable approaches and selecting pre-validated parameter sets.

This decoupled architecture provides unmatched engineering auditability: practitioners can inspect individual skill boundaries, refine recovery routines, or deprecate faulty tool configurations without initiating uncontrollable global model retraining.

Lifelong Learning and Analytical Cost Scaling

Conventional reinforcement learning and imitation learning face compounding difficulties when scaling across broad task catalogs due to parameter interference. TGL introduces the Effective Reusable Experience hypothesis: as a robot operates across extended horizons, accumulated skill blocks and contextual memory nodes progressively cover the predominant geometric and contact interactions within its operational domain.

View full-size image ↗

Analytically, we compare the marginal computational cost of traditional parameter-space updates against TGL’s modular expansion:

• Global Parameter Update Paradigm: Adding the $N$-th task requires joint optimization across all preceding $N$ tasks to prevent catastrophic forgetting. The required training samples and compute scale super-linearly as $\mathcal{O}(N^\alpha)$ ($\alpha > 1$);
• TGL Modular Expansion Paradigm: Synthesizing and validating an incoming Skill Block requires verifying only local execution contracts against the specific task scene. Marginal acquisition overhead remains bounded at $\mathcal{O}(1)$.

Provided that scene grounding, verification checks, and skill retrieval maintain lightweight computational profiles, explicit skill accumulation delivers dramatic long-term efficiency advantages over perpetual fine-tuning.

View full-size image ↗

Slow Teacher and Fast Student: Dual-System Synthesis

In real-world industrial and service settings, relying entirely on agent deliberation for every single motion introduces latencies of several hundred milliseconds to seconds, which is ill-suited for high-speed dynamic execution.

To resolve this latency-generality trade-off, TGL natively supports a Slow Teacher, Fast Student dual-system execution framework:

• Slow Teacher (System 2: TGL Deliberative Loop): The AI Agent functions as a deliberative System 2, performing long-horizon reasoning, tool composition, and fault recovery in unfamiliar tasks. In doing so, it filters out execution failures and buffers clean, physically verified expert trajectories into an execution repository;
• Fast Student (System 1: Distilled Reactive Policy): Verified trajectories are used to asynchronously train high-frequency reactive policies (such as a 50 Hz Diffusion Policy or Action Chunking Transformer). These compact models execute fluid, low-latency primitives under nominal operational conditions;
• Seamless Anomaly Hand-off: If the reactive policy experiences an out-of-distribution observation or fails a physical verification check, low-level monitors instantly trigger an interrupt, returning control to System 2 to perform root-cause diagnosis, execute recovery, and update Experience Memory.

View full-size image ↗

Empirical Benchmark Validation and Insights

We conducted rigorous empirical evaluations across the standard robotic manipulation benchmark LIBERO as well as its out-of-distribution counterpart, LIBERO-Plus. Experiments focused on key evaluation axes:

• Few-Shot Skill Acquisition: Measuring TGL’s capacity to extract invariant subgoal structures from 1 to 3 demonstrations and deploy them successfully across novel spatial layouts and visual textures;
• Mitigation of Retraining Overhead: Quantifying cumulative computational overhead and historical task retention when acquiring 10 consecutive tasks compared against parameter fine-tuning baselines;
• Fault Tolerance via Physical Contracts: Evaluating task success rates under induced physical disturbances (e.g., external perturbations during grasping, visual feature occlusions) with and without bounded recovery routines.

Key empirical findings include:

1. TGL achieved substantially higher out-of-distribution generalization compared to few-shot behavioral cloning baselines while requiring minimal demonstration data;
2. Because base model weights remain frozen, performance across historical tasks exhibited zero degradation, completely eliminating catastrophic forgetting;
3. Explicit verification contracts and localized recovery routines resolved the majority of operational disturbances autonomously, sustaining reliable long-horizon execution.

Summary and Engineering Takeaways

TGL demonstrates that general-purpose robotic manipulation does not strictly mandate monolithic end-to-end neural network optimization for every newly encountered task.

By unifying foundation model semantic reasoning, metric geometric toolsets, deterministic impedance control, and structured dual-memory repositories, a robot can acquire, adapt, and retain complex capabilities without weight perturbation. Preserving knowledge as explicit, inspectable, and reusable objects offers an economically scalable and physically grounded blueprint for lifelong robot learning.