Towards High-DoF Dexterous Manipulation through VLA Post-Training

Picking up a marker, passing it between hands, or turning it between two fingers asks more of a robot than reaching the right position. Contact must be maintained as fingers move, grasps change, and small errors accumulate. A policy can know the broad shape of a task and still miss the details that determine whether it succeeds.

Vision-language-action (VLA) models provide a starting point, but adapting them to a particular dexterous hand remains difficult. The action interface must accommodate many joints. Human corrections must start without a jump in the robot’s hand command. And the robot needs a way to improve from experience without exploring arbitrary finger movements.

We approach these problems through a shared, learned representation of hand motion. The same representation connects four steps: learning a temporal action codec, supervised fine-tuning, collecting human corrections with DAgger, and learning a residual policy from real-world interaction.

The pipeline starts by learning a temporal codec for hand actions. Once trained, this codec is frozen and becomes the common interface for the remaining stages. Demonstrations initialize the skill, DAgger supplies corrections on states the policy actually visits, and residual RL improves the tasks that still fall short.

These stages are applied as needed. A task that is already solved after human correction does not require an additional RL stage. In-hand rotation takes a different route: because its object state cannot be restored at takeover, it proceeds from supervised fine-tuning directly to residual RL.

We study five tasks using the same marker: picking it up, transferring it between hands, rotating it within one hand, removing its cap, and writing on paper. Each starts from a task-specific reset condition and has an operator-checked success criterion.

The videos below show additional third-person demonstrations recorded separately from the manuscript evaluation. Each task video contains 20 successful demonstrations, with completion counters and playback-speed labels. The Transfer and Rotate videos include a separately labeled disturbance stage.

Dexterous control must respond when the object or the robot is displaced. In these additional recordings, an operator introduces physical disturbances, including obstruction and pushes, while the final post-trained Transfer and Rotate policies execute their tasks.

The videos show the contact adjustments and recoveries at real-time speed. They illustrate behavior under the disturbances shown, rather than a quantitative robustness benchmark or a controlled comparison with another policy.

The result supports a task-specific post-training recipe for dexterous manipulation. Demonstrations establish a useful starting policy, human intervention supplies targeted recoveries, and latent residual RL refines the skills that still need improvement.

Several limits remain. Residual learning depends on a sufficiently competent reference policy. The nine-dimensional hand interface is partly dictated by the pretrained model. And the experiments concern atomic skills of roughly thirty seconds, with local variation around the training conditions.

Systematic evaluation of long-horizon composition, new objects, substantially different scenes, and broader changes in lighting or initial state remains open. The additional continuous and disturbance demonstrations do not replace that evaluation.