DR-LfD

Decompose and ReorganizePlanning with Primitives and Visuomotor Policies Learned from Demonstrations

IEEE Transactions on Robotics (T-RO)
Yizhou Chen et al. · show all 10 authors and affiliations
1The University of Hong Kong2JD.com3Nanyang Technological University4Southern University of Science and Technology5Huawei Technologies6The Chinese University of Hong Kong

DR-LfD decomposes demonstrations into reusable skills and composes them with task and motion planning. Under unseen object placements in three real-world tasks, it succeeds in 83% of trials, compared with 18% for π₀.₅.

Video Introduction

Skill Reuse and Composition

Cup-Sponge-Screwdriver. DR-LfD composes three separately learned skills into a 19-step plan with three distinct bimanual operations. Trained on the same 60 demonstrations, DP and π₀.₅ succeed in 0% and 10% of 20 trials under OOD placements, while DR-LfD succeeds in 50%.

Skill composition and reuse

TAMP composes three separately learned skills into one long-horizon task: screwdriver packing, cup handoff, and cup wiping.

Rollouts under different initial object placements (6× speed).

Skill improvement under clutter. With four distractors on the table, the success rate of DR-LfD drops from 50% to 0%. Because skills are learned separately, retraining each skill with five cluttered-background demonstrations restores the success rate to 55% (4+R in the figure), without recollecting any long-horizon demonstration.

Real-world stress tests from the paper: clutter scene, task success and completed steps with zero to four distractors, and comparisons on novel object instances.
Real-world stress tests under clutter (a, b) and novel object instances (c, d).

Spatial Generalization

Diffusion Policy

OOD setting · Failure · 4× speed

DR-LfD

Leaky setting · Success · 4× speed

SE(3)-equivariant diffusion primitives trained on ≈20 demonstrations adapt to new object positions and orientations without explicit 6-DoF object-pose estimation.

Object Handoff — OOD placements (4× speed).

Screwdriver Packing — OOD placements (4× speed).

Screwdriver Packing on Franka — single-arm platform, placements randomized by tossing (4× speed).

Randomization via Tossing

Tossing leaves the screwdriver at a random position and orientation, and the equivariant primitive adapts to each new placement.

One-shot video, the robot adapts to any displacements

Constraint Handling

Reachability and obstacle avoidance

DR-LfD exposes the reachability and collision constraints of each visuomotor policy to TAMP, which inserts corrective actions when a constraint is violated.

Real-world reachability map with the cup sleeve out of reach, and collision map with a plunger blocking cup wiping, each followed by the planned recovery steps.

Unreachable Setup

When an object lies beyond one arm’s reach, DR-LfD uses the other arm to move it into their shared workspace before insertion.

Obstructed Setup

When a pole blocks the workspace, DR-LfD relocates it to a safe pose before bimanual insertion.

Reactive Recovery

After each learned skill, a wrist-camera contact detector checks whether the expected contact is established. If it is not, DR-LfD re-perceives the scene and replans locally.

Paper Fig. 3(d): execution feeds contact observations back to scene perception and detailed TAMP for local replanning.
Online planning and execution.

When an operator moves the target object away before the grasp, DR-LfD recovers and succeeds in 9/10 Object Handoff, 5/10 Screwdriver Packing, and 10/10 Cup Wiping trials.

Object handoff — cup pulled away mid-reach (2× speed).

Screwdriver packing — screwdriver taken away (2× speed).

Cup wiping — cup relocated before wiping.

Cup-sponge-screwdriver — recovery during the full long-horizon task (4× speed).

Key Contributions

Core idea: DR-LfD learns individual manipulation skills from demonstrations and composes them with TAMP to achieve new task goals.

  • TAMP-gated system — Contact changes define atomic skills that TAMP composes into task plans.
  • Object-centric primitives — Equivariant diffusion models learn object-centric primitives that adapt to new object positions and orientations.
  • TAMP-compatible visuomotor policies — Initiation conditions, termination conditions, and physical constraints connect learned policies to sampling-based TAMP.
  • Evaluation — Real-world and simulation experiments test skill composition, spatial generalization, and physical constraint handling against imitation learning baselines.

How It Works

Method pipeline and implementation details
1Demonstrate
2Decompose
by contact
3Learn skill
repertoire
4Plan with
TAMP
5Execute &
verify
  1. Demonstrate — Record about 20 teleoperated demonstrations per short skill.

    2. Decompose by contact

    A VLM identifies robot–object and object–object contact changes. These changes define atomic skills and their start and end conditions.

    VLM-based contact relationship grounding for demonstration decomposition

3. Learn each skill

Train object-centric primitives for pick-and-place actions and visuomotor policies for contact-rich actions. Use standard TAMP motion skills for free-space moves.

SO(3)-equivariant object-centric skill network
  1. Plan a new task — TAMP selects and orders the learned skills, adding regrasping or obstacle removal to satisfy safety and reachability constraints.
  2. Execute and verify — The robot perceives, plans, executes, and verifies each step. Failed steps trigger local replanning.

Real-world Results

Real-world evaluation / Unseen placements

Higher success across all three OOD tasks.

Success rates under OOD position and rotation shifts. ALOHA · 20 trials per task and method.

Object handoff

95%

+75 points over π₀.₅

DR-LfD
95%
DP
30%
π₀.₅
20%

Screwdriver packing

80%

+70 points over π₀.₅

DR-LfD
80%
DP
10%
π₀.₅
10%

Cup-sleeve insertion

75%

+50 points over π₀.₅

DR-LfD
75%
DP
35%
π₀.₅
25%
Full results, evaluation protocol, and limitations

Spatial Generalization — Real World (n = 20 per setup)

Each task uses 20 demonstrations. DP and DR-LfD are trained from scratch on these demonstrations, and π₀.₅ is fine-tuned on the same data. ID, OOD, and Leaky placements are defined in Real-world Task Settings.

TaskSetupDPπ₀.₅DR-LfD
Object HandoffID90%100%100%
OOD30%20%95%
Leaky60%90%100%
Screwdriver PackingID55%60%80%
OOD10%10%80%
Leaky0%75%65%
Cup-sleeve InsertionID75%85%80%
OOD35%25%75%
Leaky35%35%40%

In the Leaky setting, π₀.₅ leads on Screwdriver Packing. Cup-sleeve Insertion remains difficult for all methods; fine insertion and noisy single-view depth sensing contribute to failures.

Real-world Task Settings

Tasks and initial placements

All real-world tasks run on a bimanual ALOHA robot with one overhead RGB-D camera.

Comparative tasks

Object Handoff. The right arm hands a tape roll to the left arm.

Screwdriver Packing. The robot packs a screwdriver into its box and closes the lid.

Cup-sleeve Insertion. The two arms insert a paper cup fully into a cup sleeve.

Initial Object Placements

Colored regions mark where each object starts. ID tests use the same regions as the demonstrations, and OOD tests enlarge every region. For Screwdriver Packing, OOD also replaces the three demonstrated headings (−10°, 0°, 10°) with a continuous range. Leaky tests tilt the target objects against static supports.

Initial placements for cup-sleeve insertion, screwdriver packing, and object handoff under ID, OOD, and Leaky settings.

Cup-Sponge-Screwdriver

This long-horizon task composes three skills: screwdriver packing, cup handoff, and cup wiping. Each skill is learned from 20 demonstrations.

Training and Deployment

Each skill is demonstrated separately, with its objects in a small region at a fixed orientation (a–c). At deployment, all three skills run in one scene, and the objects start anywhere in much larger regions (d). The screwdriver heading varies within (−90°, 90°), and the cup and sponge headings vary within (−45°, 45°).

Demonstration placements for three skills and deployment placements for their combined task.

Simulation Benchmarks (LIBERO & DexMimicGen)

LIBERO. DR-LfD reaches 98% on both the Spatial and Object suites, versus 84% and 78% for Diffusion Policy trained with 50 demos per task. On Long8, adding online perception raises the success rate of DR-LfD from 30% to 90% over 10 rollouts.

Data efficiency on DexMimicGen. On the contact-rich Two-Arm Threading and Two-Arm Assembly tasks, we train DR-LfD, Diffusion Policy (DP), and SE(3)-equivariant SDP with 100 to 1000 demonstrations per task.

Success rates versus 100, 200, 500, and 1000 training demonstrations on Two-Arm Threading and Two-Arm Assembly, comparing DR-LfD, DP, and SDP.
Success rates on two DexMimicGen tasks with different numbers of demonstrations, evaluated over 50 trials per setting. Error bars show 95% Wilson confidence intervals.

DR-LfD trained on just 100 demonstrations outperforms DP and SDP trained with 1000 demonstrations.

Citation

@article{chen2026decompose,
  title   = {Decompose and Reorganize: Planning with Primitives and Visuomotor Policies Learned from Demonstrations},
  author  = {Chen, Yizhou and Xu, Hang and Yu, Dongjie and Lu, Yupu and Xu, Tengye and Zhang, Zeqing and Zhang, Wei and Ren, Yi and Chen, Ben M. and Pan, Jia},
  journal = {arXiv preprint arXiv:2607.25397},
  year    = {2026}
}