Towards High-DoF Dexterous Manipulation through VLA Post-Training

1. Represent coordinated hand motion

Our platform has two 7-DoF arms and two 20-DoF hands, for a total of 54 joint coordinates. A temporal variational autoencoder represents each hand-action chunk using nine latent coordinates per time step. The arm coordinates pass through without compression.

The codec attends across the action chunk, capturing coordinated motion over time instead of treating each frame in isolation. The nine-dimensional hand representation fits the pretrained policy’s 32-dimensional action interface; it is not a claim that nine dimensions are optimal for every robot or task.

2. Adapt the VLA with demonstrations

We fine-tune π_0.5 to predict actions in this shared representation from three camera views, the language instruction, and measured joint state. The frozen decoder converts its output back into executable joint commands.

The training objective supervises both the latent action and the reconstructed hand command. This matters because small errors in latent coordinates do not necessarily produce small errors at the robot’s fingers. At deployment, the policy predicts a 32-frame chunk and executes its first 25 frames before receiving a new observation.

3. Make human corrections continuous

When an operator takes over, their live gesture may differ from the robot’s current commanded hand pose. Switching immediately can cause a command jump, while recording the operator’s initial realignment can mix pose synchronization with the correction we actually want the policy to learn.

We separate those two operations. A five-second rolling buffer lets the operator select a pre-failure state. The interface then helps align the operator’s hand pose with the robot’s pose, and a two-second command blend provides a continuous handoff. Recording the task recovery begins after synchronization.

4. Learn residual corrections from experience

After imitation learning, we freeze the reference VLA and learn a bounded correction to its latent action. A compact RL token summarizes the frozen model’s observations for a small actor and critic. The corrected action is decoded once, using the same hand codec as the reference policy.

The hand residual changes coordinated motion directions learned by the decoder, rather than perturbing every joint independently. A task mask determines which arm and hand coordinates can change. For in-hand rotation, only the active right-hand latent is adjusted; the arms and left hand receive zero residual.

Residual learning complements the imitation policy. It does not replace the need for a reference policy that already reaches useful task states and can obtain informative outcomes.