XPENG Robotics
Thesis Capabilities Model Training Simulation Policy

Cross-Embodiment World Model

AnyWorld

Human interaction, recomposed for robot learning.

Explore the research arXiv DOI Code

AnyWorld: Factorized Egocentric World Models for Cross-Embodiment Generalization

Cheng Chen, Jerry Bai, Jiacheng Wei, Boyu Chen, Xiaoji Zheng, Fan Wu, Minghao Yang, Tianrun Chen, Ruibo Li, Xiaoyu Yue, Xiaoyang Guo, Yixiao Ge, Guosheng Lin, Fayao Liu

Abstract

Collecting contact-rich robot experiences at scale remains a major bottleneck for generalizable manipulation. Beyond data quantity, robot learning also requires diverse experiences across embodiments, viewpoints, and scenes. Human egocentric videos provide abundant physical interactions, but each video captures only a narrow slice of experience under a single body, camera trajectory, and environment. We propose AnyWorld, a cross-embodiment world modeling framework that expands a single human interaction into diverse robot-native rollouts without paired human-robot demonstrations. Our model factorizes an interaction into action, camera, and embodiment: action controls capture the motion structure, camera controls specify viewpoint evolution, and the target embodiment context defines the acting body and its interaction geometry. This formulation enables independent recomposition of embodiment, viewpoint, and scene factors, allowing a single model to generate many robot-domain experiences while preserving the underlying dynamics and object interactions. We train the model with large-scale human interaction pretraining followed by mixed-embodiment fine-tuning. Experiments show that our model supports controllable recomposition across embodiments, viewpoints, and scenes, and we further demonstrate that the generated data can improve manipulation performance on the RoboCasa GR1 tabletop benchmark and a real IRON humanoid robot. Beyond aggregate gains, we test whether unpaired human experience can be recomposed into robot-native video-action pairs that target a policy gap. Controlled IRON interventions correct a spurious completion prior and establish language-grounded spatial target selection; an action-only counterfactual intervention fails to learn the latter reliably, showing that both action calibration and visual recomposition are necessary.

Full text (HTML, v2)
Key takeaways
01
Unpaired transferNo paired human–robot clips
02
Cross-embodimentSame interaction, new realization
03
Targeted interventionMissing states · counterfactual pairs

Research thesis

From controllable experience generation to targeted policy adaptation.

AnyWorld turns human interactions into controllable robot-domain experiences. The same factorized interface scales diverse training data and synthesizes targeted supervision for policy correction and capability acquisition.

Source Human interaction Scalable physical experience
AnyWorld Controllable factorization
ActionCameraTarget context
Scale Diverse robot-domain experience Embodiments, scenes, objects, lighting, and viewpoints
Intervene Targeted policy supervision Rare task states and contrastive instruction–action pairs
Policy impact Improve · Correct · Extend Behavioral gains on a physical IRON robot

01 · Controllable recomposition

Preserve the interaction. Recompose its realization.

AnyWorld preserves the underlying interaction and scales it into diverse realizations across embodiments, scenes, object configurations, lighting, and camera trajectories.

Cross-embodiment replay

The same human interaction, executed by IRON.

Action and viewpoint structure are retained while the acting body is changed.

Interaction 01Surface sweeping
Human Source interaction
IRON Re-embodied rollout
Interaction 02Garment cleaning
Human Source interaction
IRON Re-embodied rollout
Scene and object recomposition

Scale one interaction family across controllable worlds.

Objects, layouts, illumination, and environments can change without discarding the interaction.

A
SweepingObject and scene variation
Base IRON rollout
Objects Edited layout
Scene New environment
B
Object manipulationContext and lighting variation
Base Interaction rollout
Objects Added context
Lighting Relit environment
C
Fine-grained interactionObject and appearance variation
Base Piano interaction
Objects Edited context
Lighting Relit context
Explicit camera control

Compare each generated trajectory against the source viewpoint.

The reference video is shown alongside camera-controlled realizations of the same interaction.

Reference Source camera trajectory
Trajectory A Horizontal viewpoint shift
Trajectory B Roll and vertical motion
Trajectory C Diagonal viewpoint motion
Interaction preservedRealization scaled across controllable factorsBody · Scene · Objects · Lighting · Camera

02 · Model design

Factorized controls make interaction and realization independently composable.

Action and camera controls preserve the source interaction. A target first frame and embodiment condition specify the robot body, environment, and initial interaction geometry.

Source interactionHuman experience
Wiping interaction in the source human scene
ExtractAction sequenceCamera trajectory
Factorized condition fusionAnyWorld
Interaction control Action trajectory Pixel-space motion structure preserved across embodiments Latent conditioning
Viewpoint control Camera trajectory Plücker-ray geometry specifies viewpoint evolution Geometric adapter
Target recomposed environment context
Target visual state Recompose into a new environment The first frame anchors embodiment, scene layout, objects, and initial interaction geometry.
New embodimentNew sceneNew layout
Generated experienceComposed realization
The same interaction recomposed into a new embodiment, scene, and layout
Interaction retainedNew embodimentNew sceneNew layout

03 · Training

Learn interaction priors first, then ground them across embodiments.

Human video provides broad contact and motion priors. Unpaired mixed-domain fine-tuning binds the shared control interface to robot appearance and geometry.

Stage 1Human interaction pretraining
Human interaction example Human interaction example Human interaction example

Learn physical interaction priors and action–camera control fidelity.

Stage 2Unpaired mixed-embodiment grounding
IRON interaction example IRON interaction example RoboCasa GR1 interaction example

Associate shared interaction controls with IRON and GR1 appearance, viewpoints, and interaction geometry.

Key property No paired human–robot videos.

Cross-embodiment correspondence emerges through the unified action, camera, and embodiment interface.

04 · Physics-aware simulation

Physics-aware rollouts capture both success and failure.

Under the same scene, controlled changes in motion geometry lead to successful target contact or a miss. The rollout reflects interaction outcomes rather than only visual appearance.

C1 · Reference Nominal interaction trajectory
C2 · Success Adjusted motion reaches and grasps the target
C3 · Failure Rotated motion fails to establish contact

05 · World-model evidence

Stronger control without sacrificing video quality.

AnyWorld improves alignment to action, camera, and embodiment conditions while maintaining competitive visual consistency.

Overall controllabilityAverage of ActionAlign, CameraAlign, and EmbodAcc
Cosmos-Predict2.5
0.417
WAN Fun-Control
0.609
AnyWorld
0.778
World-model controllability
MethodActionCameraEmbodimentAvg.
Cosmos-Predict2.50.1700.3150.7650.417
WAN Fun-Control0.6550.4020.7690.609
AnyWorld0.6590.7890.8860.778
0.942 Subject 0.949 Background 0.995 Flicker 0.996 Smoothness

06 · Real-robot policy evidence

Targeted robot-domain data can correct and extend policy behavior.

We use AnyWorld not only to scale experience, but also to construct targeted robot-domain supervision for concrete policy failures.

01
Correct policy shortcutsAugment missing task states
02
Ground instruction-conditioned controlPair language, target, and action
03
Improve target-domain adaptationSupporting aggregate evidence

Evidence 01 · Policy debiasing

Targeted state augmentation corrects completion shortcuts.

The baseline learns a false completion prior: when a banana is already visible inside the box, it remains stationary even though another target must still be placed. AnyWorld re-embodies the human interaction as IRON and re-scenes it into this missing robot-domain state.

Human interaction→AnyWorld robot-domain state→Policy correction
Before interventionVLA baseline
No action
The policy treats the banana already inside the box as evidence that the instruction has been completed.
Human-derived intervention AnyWorld state augmentation
Human → IRONBanana-in-box stateSmall fine-tuning set
After intervention+ Human-derived AnyWorld data
Task resumed
The same policy now responds to the instruction and places the remaining banana into the box.
Training-data construction The corrective experience is synthesized from human interaction.
Human sourceInteraction prior
AnyWorld Re-embody + re-scene
Human → IRONAdd banana inside box
Generated supervisionTargeted IRON rollout

Evidence 02 · Instruction-grounded control

Contrastive pairing grounds language while preserving stable robot action.

The robot data contains only single-target, single-side grasps. AnyWorld re-embodies human motion as IRON and re-scenes it with symmetric left–right targets; paired supervision then binds language to the corresponding stable robot action.

Human wrist motion→AnyWorld dual-target experience→Instruction-grounded policy
Human sourceWrist-level motion
AnyWorld transformation
Re-embodyHuman → IRON
Re-sceneSymmetric left/right targets
Align actionRelative EEF + robot-space projection
AnyWorld outputIRON dual-target experience
Contrastive supervision Same observation. Opposite instruction. Corresponding action. This pairing forces the policy to use language rather than memorized visual or state priors.
Real-robot policy trained with human-derived AnyWorld data Language selects the target under matched visual conditions. The paired robot-domain experience changes only the instruction and corresponding action.
After · Human-derived AnyWorld pairs“Place the left-side banana into the box.”
Left target Language selects the target; the learned robot motion remains stable.
After · Human-derived AnyWorld pairs“Place the right-side banana into the box.”
Right target The policy changes target while preserving accurate execution.
Control space

Relative EEF

Preserves local motion direction during robot fine-tuning.

Human signal

Wrist-centric action

Avoids transferring incompatible human arm-scale motion.

Alignment

Robot-space projection

Constrains action magnitude while retaining intent.

Supervision

Contrastive pairs

Break visual and state overfitting by changing caption and action under the same observation.

Evidence 03 · Target-domain adaptation

Re-embodied rollouts provide additional policy gains.

As supporting evidence, mixing generated target-domain rollouts with robot data improves average task success in both simulation and real-robot evaluation.

RoboCasa GR118 pick-and-place tasks
Baseline
49.8%
+ AnyWorld
54.6%
+4.8 pts
IRON real robot20 grasp trials
Baseline
20.0%
+ AnyWorld
55.0%
+35.0 pts

Project

AnyWorld

Factorized Egocentric World Models for Cross-Embodiment Generalization