Fine-Tuning VLAs with Self-Demonstrated Generative Control for Multi-Task Manipulation

Prachi Garg, Steve Xing*Equal contribution, Prahit Yaugand*Equal contribution, Saurabh Gupta, Derek Hoiem
University of Illinois Urbana-Champaign
Paper ArXiv Code Checkpoints Datasets VideoCite

Abstract

State-of-the-art vision-language-action (VLA) models such as π0.5 exhibit strong semantic understanding, instruction following and task behavior. However, when deployed on new robots, even minor mismatches in hardware configuration relative to pretraining can cause severe performance drops. Finetuning the VLA on in-domain expert data from the new embodiment improves performance on the expert task but leads to a loss in its original instruction following and behavioral priors. In this paper, we propose a self-supervised method that generates online interaction rollouts from the zero-shot VLA as additional training data for finetuning. Our experiments show this yields strong multi-task manipulation policies on the target robot that (1) inherit pretrained tasks distilled from the zero-shot VLA without expert demos for them, (2) improve success and generalist instruction following beyond expert-only fine-tuning on expert-teleoperated tasks, and (3) mitigate forgetting of held-out tasks from the pretrained policy, not part of fine-tuning. We demonstrate the success of our approach across test sets probing generalization on a real ALOHA robot and a new simulation benchmark in RoboTwin.

Problem & key observation

0:00 / 0:00

Method

We leverage these priors through self-supervised generative control [Self-Demonstration]We rollout the policy online on our robot, asking it to show us how to do the task

BENCHMARK 1 · PICK AND PLACE · TSS

Pick-and-place

Self-supervised task, distilled purely from self-demos, no expert data collected

π₀.₅ Zero-Shot
Zero-shot pick-and-place · 0% task success
Multi-Task ES
Fine-tuned on expert pick-up · 0% task success
Multi-Task ES+SS (Ours)
FAST + Flow fine-tuning · 55% task success

20 evaluation trials per policy

Success rate0 → 55%

pick-and-place success, with no expert place demonstrations 20 evaluation trials

π₀.₅ Zero-ShotMulti-Task ESMulti-Task ES+SS (Ours)

Task-behavior transfer · the expert data contains pick-up, not placing.

2.2 min teleoperation9.1 min self-demos

Browse all 20 test rollouts ↗

Held-out skills

What if we recollect the pick-and-place data on our robot? Comparison against upper bound.

Held-out skills not part of any fine-tuning data source are preserved even when compared to upper bound! Results from a human user study show that self-demos are preferred in blind comparisons.

Push

Success rate (correct object) p<0.001

ES+ES
22%
ES+SS (Ours)
71%

While IF is comparable across the 2 policies (82% and 84%), motion prior is far better retained by our method.

Pour / turn over

Success rate (correct object) p<0.001

ES+ES
18%
ES+SS (Ours)
51%

While IF is comparable across the 2 policies (91% and 97%), motion prior is far better retained by our method.

6 raters · 240 paired blind judgments · 20 scenes per task

Comparison against the all-expert upper bound

ES+ES collects expert demonstrations for all tasks on our robot, replacing self-demos.

  • Instruction following is comparable; the all-expert upper bound has higher success rates.
  • Sub-optimal self-demos are competent w.r.t. an oracle that re-collects expert demos for those same “pick-and-place” tasks, so there is little reason to pay the price of data re-collection.
Multi-Task ES+ES (Recollect Expert Oracle)Multi-Task ES+SS (Ours)

Expert pick-up

Novel-object pick-up

Pick-and-place

Laundry in basket

Browse all 20 test rollouts ↗

BENCHMARK 2 · COLORED BIMANUAL GEAR INSERTION

Colored bimanual gear insertion

Expert-supervised task, self-demos improve performance

π₀.₅ Zero-Shot
Multi-Task ES
Caterpillar gear insertion · 40% success
Multi-Task ES+SS (Ours)
Caterpillar gear insertion · 90% success

10 evaluation trials per policy

Success rate40 → 90%

gear insertion success with expert gear colors 10 evaluation trials

π₀.₅ Zero-ShotMulti-Task ESMulti-Task ES+SS (Ours)
Browse all 10 test rollouts ↗

BENCHMARK 2 · UNSEEN GEAR COLORS

Unseen gear colors

Expert-supervised task · OOD gear colors

π₀.₅ Zero-Shot
Multi-Task ES
Multi-Task ES+SS (Ours)

10 evaluation trials per policy

Success rate10 → 20%

gear insertion success with unseen colors 10 evaluation trials

π₀.₅ Zero-ShotMulti-Task ESMulti-Task ES+SS (Ours)
Browse all 10 test rollouts ↗

Real robot test sets span

140test cases
68unique language instructions
6task families
380evaluation rollouts

Simulation

Mid-training
Default embodiment (E1)Hammering · TSS
Post-training
Default embodiment (E1)Expert stacking · TESMid-training and post-training use the same embodiment, isolating forgetting caused by fine-tuning alone.
Post-training
Topview embodiment (E2)Expert stacking · TESOnly the head-camera view changes to a top-down view, introducing a controlled embodiment gap.Same camera, 0.45 m forward, looking straight down.

Default embodiment (E1)

Task illustration

Task success (%)

Controlled embodiment gap (E2)

Changing only the head-camera view drops old-task success from 90.8% to 51.6%; expert-only post-training lowers it further to 21.4%, while adding self-demos improves it to 36.0% (+14.6 percentage points).

Old tasks (TSS), evaluated in the topview mean over 10 tasks × 50 seeds

Base policy · E1
90.8%
Base policy · E2 zero-shot
51.6%
ES
21.4%
ES+SS (Ours)
36.0%

New tasks (TES): topview stacking 2 tasks × 50 seeds

Base policy · E2 zero-shot
6.0%
ES
95.0%
ES+SS (Ours)
94.0%

Discussion

1. What do self-demos contain?

Our real-robot self-demos are all task failures, yet much of their behavior is semantically meaningful. In the 15 analyzed rollouts, 94% of grasp attempts and 89% of place attempts target the instructed object or container, but no grasp succeeds.

13 of 15 episodes retry, with a mean of 2.1 grasp attempts and 1.2 place attempts per episode.

Original Figure 5 plot: 15 self-demonstrations showing correct and incorrect grasp and place attempts over each episode. No grasp succeeds.
2. Multi-task fine-tuning mitigates loss of instruction following
  • Single-task fine-tuning forgets more instruction following. Compared with zero-shot, pre-grasp IF drops from 70% to 40% on novel objects and from 70% to 45% on pick-and-place.
  • Multi-task fine-tuning mitigates this loss. Splitting the same 30 expert demo budget into 15 red- and 15 green-cube pickup demos raises pre-grasp IF from 40% to 50% on novel objects and from 45% to 75% on pick-and-place (Single Task ES → Multi-Task ES).
  • Multiple instructions in the same scene help preserve text steerability. With the same 15 red + 15 green demos, including both objects instead of one per scene raises pre-grasp IF from 45% to 50% on novel objects and from 55% to 75% on pick-and-place (Single Object/Scene → both objects present). The policy must use both vision and text to distinguish the task.
Original Table 1. Novel-object pre-grasp IF: zero-shot 70, single-task ES 40, multi-task ES with one object per scene 45, multi-task ES 50, ours 55. Pick-and-place pre-grasp IF: 70, 45, 55, 75, 90. Pick-and-place success: 0, 0, 10, 0, 55 percent.

IF (post-grasp) measures how often the arm moves to the correct container after an attempted pick-up.

Even when expert data covers the same pick-and-place task, adding self-demos raises success from 55% to 65% (Table 9).

Pick-and-place trainingIF (pre-grasp)IF (post-grasp)Success rate
Multi-Task ES75%85%55%
Multi-Task ES+SS (Ours)95%80%65%

3. Qualitative Results

Personalized distillation. Our policy produces smooth rollouts that complete self-supervised tasks, with motions personalized to our robot. The zero-shot base policy is jerkier, often raises both arms higher than the task requires, and is unable to grasp, with 0% task completion on pick-and-place.

Recovery behavior. Ours has better recovery behavior when objects sit at table edges: it turns to look for objects behind the wrist camera, unlike the baselines. We suspect this comes from the broader task and state coverage of self-demos on the robot.

Scene 1

Multi-Task ES+ES (Oracle)
Multi-Task ES+SS (Ours)

Scene 2

Multi-Task ES+ES (Oracle)
Multi-Task ES+SS (Ours)

TL;DR of results: Post-training VLAs with policy's self-supervised online rollouts on the target robot distills pretrained skills, improves expert task performance, and preserves held-out skills. Yields a generalist, multi-task manipulation policy at deployment.

Takeaway: To mitigate forgetting:

  • Multi-task fine-tuning is important for instruction following and text steerability. This can use expert data (the all-task expert oracle) or self-demos, but why pay the price of recollection when self-demos retain instruction following to the same degree?
  • Self-supervision alone shines in distilling and preserving motion priors.

Video

0:00 / 0:00

BibTeX

@misc{garg2026finetuningvlasselfdemonstratedgenerative,
      title={Fine-Tuning VLAs with Self-Demonstrated Generative Control for Multi-Task Manipulation},
      author={Prachi Garg and Steve Xing and Prahit Yaugand and Saurabh Gupta and Derek Hoiem},
      year={2026},
      eprint={2608.19490},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2608.19490},
}