Rethinking Visual Embodiment Dependence in Visuomotor Policies

Abstract

Visuomotor policies observe both the task scene and the acting embodiment, allowing embodiment-specific visual cues to influence action prediction. We study this phenomenon as visual embodiment dependence (VED) and show, through cue-conflict interventions across representative policies, that visible robot configuration can become a shortcut to task progress. Rather than eliminating VED, we argue that it should be structured around embodiment information that supports control and generalization. We realize this through embodiment canonicalization in 3D point clouds, replacing the original embodiment with a canonical end-effector representation (CER) that preserves control-relevant geometry while abstracting embodiment-specific morphology. Its editable form further enables configuration-decorrelation augmentation for unfamiliar robot configurations. Experiments show that embodiment canonicalization substantially improves human-to-robot policy transfer without robot demonstrations, while simply removing the embodiment is insufficient without preserving control-relevant geometry. We further find that CER itself can become a configuration shortcut when robot configuration becomes decoupled from task progress; configuration-decorrelation augmentation mitigates this failure mode and restores robust recovery without sacrificing performance on seen configurations. Together, these results show that robust visuomotor learning benefits from structuring, rather than removing, visual embodiment information.

Diagnosing Visual Embodiment Dependence

A policy observes both the task scene and the acting hand or robot. This can introduce two forms of dependence: morphology dependence, which limits transfer between visually different embodiments, and configuration dependence, in which the visible robot configuration becomes a proxy for task progress.

To isolate configuration dependence, we construct cue-conflict observations: the task state comes from one stage, while the visible embodiment configuration comes from another. We evaluate BC-RNN and Diffusion Policy in simulated Pick Place Can, and ACT and RISE in real-world Rotate Plate. All diagnostic policies predict absolute actions and receive no proprioceptive robot state, so configuration is available only through vision.

Structuring Visual Embodiment Information

Preserve the task scene. Canonicalize the embodiment. Diversify its configuration.

1. Remove embodiment-specific morphology. Mask the acting hand or robot in the 3D point cloud while preserving the surrounding task scene.
2. Retain control-relevant geometry. Insert a compact, line-based canonical gripper. Two distal segments represent the gripper tips and opening width; a proximal segment supplies additional orientation information. The same representation is instantiated from human hand landmarks or the robot gripper pose.
3. Decouple configuration from task progress. At recovery-eligible states, perturb the CER pose and pair it with a valid recovery trajectory back to the demonstrated pose, followed by the original trajectory. Contact-constrained states are excluded when a pose change would also require changing the scene.

Both operations are defined in 3D, and we use RISE as the policy backbone. Configuration-decorrelation augmentation is applied on the fly with probability 0.5 at eligible states, retaining only perturbations and recovery trajectories that satisfy workspace and collision constraints.

Experiments

Human-to-Robot Policy Transfer

We collect 50 human demonstrations per task for four real-world tasks: Collect Block, Close Box, Open Drawer, and Pour Balls. Policies are trained from scratch using only these human demonstrations, with no robot demonstrations. Controlled variants share the same RISE backbone and action-processing pipeline.

Generalization to Novel Robot Configurations

We separately collect 50 teleoperated robot demonstrations per task for Rotate Plate and Pour Balls. At evaluation, robot configurations are paired with the initial task state to break the configuration–progress associations seen during training. Rotate Plate includes mild and large shifts; Pour Balls uses a novel configuration-state shift. Policies share the same 20 object initializations and robot configurations for each task.

Ablations and Analysis

Which Embodiment Geometry Matters?

On Collect Block, we compare masking and CER designs ranging from coarse position and pose cues to gripper tips and the full canonical gripper. An oracle representation, obtained by replaying trajectories on the robot and capturing its full geometry, tests whether the compact CER loses control-relevant information.

Masking at Training and Deployment

We compare masking during training and deployment on Collect Block, reporting both task success and inference latency. Deployment variants use SAM2 segmentation, a mask rendered from the robot model and camera calibration, or no mask; all retain CER.

What Does the Learned Representation Depend On?

Controlled interventions on held-out robot observations from Collect Block and Pour Balls independently remove the original embodiment or CER while keeping the task scene fixed. We measure changes in the final representation before the action decoder using cosine distance to examine which embodiment cues the policy depends on.

BibTeX

@article{fang2026ved,
  title  = {Rethinking Visual Embodiment Dependence in Visuomotor Policies},
  author = {Fang, Hongjie and Lu, Yuxuan and Wang, Chenxi and
            Qin, Haoxiang and Tang, Shirun and He, Zihao and
            Xia, Shangning and Chen, Jingjing and Liu, Wanxi and
            Wang, Shiquan and Lu, Cewu},
  journal = {arXiv preprint arXiv:2609.16815},
  year = {2026}
}