[Paper Notes] RynnWorld-Teleop: An Action-Conditioned World Model for Digital Teleoperation
Published:
RynnWorld-Teleop replaces the physical robot in a teleoperation session with an action-conditioned video world model. An operator’s hand-pose stream simultaneously drives a robot-view video generator and a human-to-robot retargeting pipeline, producing synchronized RGB observations and 54-dimensional robot actions for imitation learning.
My main takeaway concerns the operator’s “feel.” The system provides a responsive visual control loop: a distilled causal generator produces robot-centric video at 40 FPS with about 25 ms model latency. It does not provide haptic or force feedback. The Manus gloves capture pose; they do not return contact force, weight, friction, or slip to the hands. RynnWorld-Teleop is therefore best understood as a scalable data engine with visual telepresence, not a bilateral force-reflecting teleoperation system.
Paper Info
“RynnWorld-Teleop: An Action-Conditioned World Model for Digital Teleoperation” is by Haoyu Zhao, Xingyue Zhao, Hangyu Li, Biao Gong, Kehan Li, Siteng Huang, Xin Li, Deli Zhao, and Zhongyu Li, from DAMO Academy, Alibaba Group, the Hong Kong Embodied AI Lab, CUHK, Hupan Lab, and Ant Group. The notes below refer to arXiv:2607.06558v2, dated July 12, 2026.
Digital Teleoperation as Data Generation
Physical teleoperation couples every demonstration to robot availability, a prepared workspace, real objects, and manual resets. RynnWorld-Teleop moves the interaction into a learned visual world. Given a reference image (I_{\mathrm{ref}}) and a hand-gesture sequence
[ P={p_1,\ldots,p_T}, ]
the model generates a robot-centric egocentric video
[ V={v_1,\ldots,v_T}. ]
The same gesture stream is retargeted into robot joint positions. The output trajectory is therefore
[ \tau={(v_t,a_t)}_{t=1}^{T}, \qquad a_t\in\mathbb{R}^{54}, ]
where each action concatenates two 7-DoF arms and two 20-DoF dexterous hands. The generated observation and robot action share the same human control signal, which gives the system its action-grounded property.
The paper defines three requirements for useful digital teleoperation:
- robot-centric: the generated view depicts the target robot embodiment;
- action-grounded: each frame remains tied to recoverable joint-level actions;
- real-time: the operator stays in the visual loop and can compose longer skills.
What the Operator Actually Feels
The operator wears five HTC Vive trackers—chest, wrists, and upper arms—and Manus data gloves. In the physical data-collection setup, the trackers provide wrist-to-chest transforms at 100–120 Hz. The gloves are converted into a 21-point MediaPipe hand skeleton and then retargeted to each 20-DoF WUJI hand, with an exponential moving average for smoothing.
These devices are motion sensors. The paper describes no actuators in the gloves, fingertip force display, vibrotactile feedback, exoskeleton resistance, or other haptic channel. The feedback available during digital teleoperation is the generated first-person robot video.
| Feedback or control channel | Present in the paper? | Role |
|---|---|---|
| Hand and arm pose tracking | Yes | Captures operator intent |
| Real-time robot-view video | Yes | Visual closed-loop feedback |
| Motion smoothing and IK constraints | Yes | Stabilizes retargeted action labels |
| Contact-force reflection | No | No physical reaction force reaches the operator |
| Weight or inertia sensation | No | Inferred only from generated visual motion |
| Friction and slip sensation | No | No tactile slip feedback |
| Operator user study | No | No measured comfort, presence, or subjective hand-feel score |
“Fluid and responsive” in the paper is supported by throughput and latency measurements. It should not be interpreted as measured haptic realism.
1. Depth-Aware Skeletons Preserve Fine Motion
A standard 2D skeleton loses the hand’s distance from the camera, which is crucial when reaching behind, around, or into contact with an object. RynnWorld-Teleop renders a 21-joint skeletal video whose color and joint/bone diameter vary with camera-space depth. The result remains a 2D image-like signal while carrying explicit 3D cues.
A pretrained VAE maps this skeleton video to a control latent
[ c\in\mathbb{R}^{C\times T\times H\times W}, ]
spatially and temporally aligned with the target video latent. This representation gives the video model fine-grained articulation and depth information without requiring full meshes or robot simulation assets.
2. Pose-Conditioned Video DiT
The world model builds on Wan2.2-TI2V-5B. It introduces a control patch-embedding branch alongside the original video branch. Because skeleton and video latents have different statistics, the control latent is normalized into the video latent distribution before fusion:
[ \tilde c = \frac{c-\mu_c}{\sigma_c}\sigma_z+\mu_z, ]
[ x = \operatorname{PatchEmbed}_z(z_t) +\alpha\operatorname{PatchEmbed}_c(\tilde c). ]
The control branch is zero-initialized and (\alpha) starts small, allowing pose control to enter gradually while preserving the pretrained video prior. Direct concatenation disrupts that distribution: the reported FVD rises from 585 to 1191 in the LoRA ablation.
This conditioning mechanism is the link between an operator gesture and the visual consequence. It improves controllability, although it cannot guarantee that every generated contact obeys true rigid-body dynamics.
3. Human Priors First, Robot Embodiment Second
Training proceeds across two domains. Stage 1 uses large-scale egocentric human video to learn hand-object interaction priors:
- VITRA: 30.7 million frames;
- EgoDex: 74.0 million frames.
Stage 2 fine-tunes on 1,800 paired human-robot teleoperation episodes, totaling 0.43 million robot frames. The four tasks are dual picking, block pushing, bimanual lifting, and lid placement. Human gestures are paired with their robot executions through IK, teaching the generator how a human intention should appear through a specific robot embodiment.
The human-video stage supplies object permanence, contact patterns, and visual “common sense.” Removing it causes a severe collapse: FVD increases to 2598, with ghosting, blurred structures, and disappearing end effectors or objects. These learned priors make contact look plausible to the operator and make generated trajectories useful for policy training. They remain learned statistical dynamics, without explicit online force or contact solving.
4. Streaming Distillation Creates the Visual Loop
A bidirectional diffusion teacher is too slow for interactive control. The authors convert it into a causal student in two steps.
First, causal flow-matching warm-up teaches frame-causal generation. At time (t), attention can use only frames (1,\ldots,t). A fixed-size KV cache supports streaming, while the reference-image embedding persists as a sink token to preserve the original scene.
Second, Distribution Matching Distillation (DMD) transfers the teacher’s output distribution into a four-step generator. Training backpropagates through the persistent KV cache across successive chunks, reducing boundary artifacts and improving long-rollout consistency.
On one H100 at (480\times832), the causal model reports:
- 40.0 FPS;
- approximately 25 ms per generated frame;
- about 5% of latency in skeletal encoding;
- about 72% in four-step causal DiT denoising;
- about 23% in VAE decoding.
This rate matches or exceeds a typical 30 Hz robot camera stream and is the main basis for the paper’s claim of a responsive operator experience. The paper reports model inference latency, not complete motion-to-photon latency including trackers, communication, display, and buffering.
There is also a quality-speed trade-off. On EgoDex-Test, the non-causal SFT model obtains FVD 550 at 2.8 FPS; the causal model reaches 40 FPS with FVD 1226. Real-time interaction comes with reduced visual fidelity.
5. Retargeting Produces Smooth Robot Actions
The operator’s wrist pose is expressed relative to the chest, scaled into the robot workspace, and transformed into the robot base frame:
[ T_{\mathrm{target}} = T_{\mathrm{base}} \cdot \operatorname{Scale}(T_{\mathrm{chest}}^{-1}T_{\mathrm{wrist}}) \cdot T_{\mathrm{ee}}. ]
An iterative damped least-squares IK solver maps this pose to robot joints. Adaptive damping handles near-singular configurations, a null-space shoulder prior favors natural arm postures, and hard clipping enforces joint limits. During physical demonstration collection, Ruckig further limits velocity, acceleration, and jerk before sending arm commands at 200 Hz. The robot’s low-level interface runs at 500 Hz; policy commands are reported at 50 Hz with 18–30 ms delay.
These mechanisms improve control smoothness and the quality of action labels. They do not add physical feedback to the digital operator.
6. Long-Horizon Generation and Re-Anchoring
Pure autoregressive generation accumulates visual drift. The data-generation pipeline divides long videos into 81-frame chunks. The first chunk starts from the true initial frame; later chunks can be re-anchored with an actual robot-camera frame from the corresponding source demonstration.
This detail narrows the strongest “unbounded digital generation” interpretation. Re-anchoring preserves object pose and lighting when a physical source trajectory exists. A fully synthetic, live session starting from only one image cannot access future real frames, so it must tolerate model drift or use another state-correction mechanism.
Experiments: Is the Generated Data Useful?
The downstream evaluation trains Diffusion Policy, (\pi_0), and (\pi_{0.5}) on real and generated trajectories, then runs 35 real-robot trials per task. Adding 300 generated episodes to 300 real episodes generally improves success. The largest reported gain occurs in lid placement:
| Policy | Training data | Lid placement success |
|---|---|---|
| (\pi_{0.5}) | 300 real | 42.86% |
| (\pi_{0.5}) | 300 real + 300 generated | 62.86% |
| (\pi_0) | 300 real | 34.29% |
| (\pi_0) | 300 real + 300 generated | 54.29% |
A (\pi_0) policy trained on 300 generated episodes and zero real episodes reaches 82.86% on block pushing and 77.14% on bimanual lifting. Lid placement remains harder at 28.57%, revealing the limit of generated data for precise contact and alignment.
These results validate RynnWorld-Teleop primarily as a robot-data generator. They do not measure whether a human operator can reliably judge contact forces or complete tasks through the generated visual loop.
Strengths and Limitations
The paper’s strongest idea is the shared action source. A hand-pose stream produces both visual observations and embodiment-specific actions, making generated trajectories immediately compatible with imitation learning. Depth-aware skeletal conditioning preserves fine control, and causal distillation turns a large video model into a 40 Hz interactive generator.
Several boundaries matter in practice:
- The system offers visual feedback without haptic or force feedback.
- Model latency is measured; full operator-loop latency and subjective usability are not evaluated.
- Generated contact follows learned visual priors and can fail on liquids and highly deformable objects.
- Each robot platform currently requires separate embodiment adaptation.
- Causal real-time generation loses image quality relative to the slower teacher.
- Long-horizon re-anchoring uses real source frames when available, limiting a fully synthetic interpretation.
Takeaways
RynnWorld-Teleop’s operator loop can be summarized as
[
\boxed{
\text{tracked gesture}
\rightarrow
\begin{cases}
\text{40 Hz generated robot-view video}
\text{retargeted 54D robot action}
\end{cases}
}.
]
The operator receives fast visual consequences for hand motion, which supports a useful sense of control. Physical hand feel remains absent: contact, load, friction, and slip are seen through generated pixels and never reflected into the operator’s hands. For scalable imitation-learning data, that trade is compelling. For contact-rich telepresence or delicate force-controlled manipulation, an additional bilateral haptic channel and stronger state grounding would still be required.