[Paper Notes] TeleDexter: Towards Human-level Dexterous Teleoperation
Published:
TeleDexter adds a learned low-level “cerebellum” between an operator’s intent and a dexterous robot hand. The operator and a tracked object provide synchronized fingertip-position and object-pose targets. A reinforcement-learning controller then chooses robot joint commands and contact transitions that realize those targets. This changes dexterous teleoperation from direct pose copying into goal-conditioned, contact-aware execution.
The mechanism is also the right boundary for reading the paper’s “human-level” claim. TeleDexter demonstrates in-hand reorientation, finger gaiting, and multi-stage tool use that conventional retargeting baselines rarely complete. It remains an object-specific controller trained from motion-capture data, requires real-time object tracking, observes no tactile signal, and is evaluated on seven fixed tasks with 15 trials each. The result is a strong contact-execution layer and a valuable demonstration generator; general human-level manipulation remains a much larger goal.
Paper Info
“Towards Human-level Dexterous Teleoperation” is by Puhao Li, Zeyuan Chen, Yingying Wu, Pengkun Wei, Yuyang Li, Tianyu Wang, Jiaxiao Shi, Mingrui Yu, Baoxiong Jia, Song-Chun Zhu, Tengyu Liu, and Siyuan Huang. The authors are affiliated with Tsinghua University, the State Key Laboratory of General Artificial Intelligence at BIGAI, and Peking University. These notes refer to arXiv:2607.11481v1, submitted July 13, 2026.
The Missing Layer in Dexterous Teleoperation
Kinematic teleoperation maps tracked human joints or geometric hand features into robot joint targets. It is intuitive and fast, but the mapping carries no model of contact force, friction, object inertia, or actuator response. This becomes fragile as soon as a finger must release, move around the object, and re-establish contact while other fingers keep the object stable.
TeleDexter inserts a learned dynamics prior into this loop. All quantities are expressed in the robot wrist frame. The arm independently follows the operator’s wrist through inverse kinematics, while the hand policy receives a co-tracking goal
[ g_t=\left(\hat{p}^{\mathrm{tip}}_t,\hat{T}^{o}_t\right), ]
where (\hat{p}^{\mathrm{tip}}_t) contains target fingertip positions and (\hat{T}^{o}_t=(\hat{x}^{o}_t,\hat{R}^{o}_t)\in SE(3)) is the target object pose. The policy outputs joint-position commands
[ a_t=\pi_\theta(o_t,g_t). ]
The goal specifies what geometry should be reached. Simulation-trained control supplies how to move contacts and joints to reach it. Tracking the object target matters as much as tracking the fingertips: identical fingertip positions can correspond to a stable regrasp, a slipping object, or a failed contact transition.
At deployment, motion capture measures an operator-side hand-object interaction while the robot-side state provides the current hand and object configuration. The resulting feedback loop is therefore grounded in two errors simultaneously: fingertip target versus robot fingertip state, and desired object motion versus the robot object’s measured motion.
1. Geometry-Aware Reference Motions
The low-level controller is trained from unscripted human hand-object interaction. For each object, the authors record 150 trajectories of 20 seconds each, totaling about 50 minutes. The recordings include in-hand translation, rotation, free play, finger gaiting, and tool-use motions.
Direct human-to-robot retargeting provides a useful initialization but can place fingers inside the object or destroy intended contacts. TeleDexter uses a two-stage pipeline:
- vector-based retargeting aligns human and robot hand geometry;
- object-mesh-aware optimization refines the whole trajectory.
The second stage solves
[ q^*{1:T}=\arg\min{q_{1:T}} \sum_{t=1}^{T} \left( L^t_{\mathrm{vec}} +\lambda_{\mathrm{surf}}L^t_{\mathrm{surf}} +\lambda_{\mathrm{pen}}L^t_{\mathrm{pen}} +\lambda_{\mathrm{col}}L^t_{\mathrm{col}} \right) +\lambda_{\mathrm{smooth}}L_{\mathrm{smooth}}. ]
The surface term attracts near-contact hand points to the object surface; the penetration term pushes hand points out of the object; the collision term prevents fingers from overlapping; and trajectory smoothness suppresses motion-capture jitter. This produces synchronized robot fingertip and object-pose targets with more plausible contact geometry. The later RL rollouts supply physical feasibility by resolving forces and friction that lie outside the geometric optimization.
2. Consecutive Subgoals Give RL Room to Discover Contact
Dense frame-by-frame imitation fixes a target at every instant. For dexterous contact, this can over-constrain exploration: the robot embodiment and dynamics may require a finger sequence that differs from the captured human timing.
TeleDexter samples consecutive co-tracking subgoals from each reference trajectory. For active subgoal (k), it measures per-finger position error, object-position error, and object-rotation error:
[ e^f_{t,k}=\lVert p^{\mathrm{tip}}{t,f}-\hat p^{\mathrm{tip}}{k,f}\rVert_2, \qquad e^{\mathrm{pos}}_{t,k}=\lVert x^o_t-\hat x^o_k\rVert_2, ]
[ e^{\mathrm{rot}}_{t,k} =\left\lVert \operatorname{Log}!\left(\hat R^{o\top}_kR^o_t\right) \right\rVert_2. ]
A subgoal counts as reached after all errors stay below their thresholds for 5–15 consecutive frames. The final tolerances are 3 cm per fingertip, 1 cm object translation, and 10° object rotation. Once the target is reached, the system samples a later subgoal; 10% of transitions switch to a different reference trajectory at the same temporal index. The policy must therefore connect configurations that were never adjacent in a human demonstration.
The reward combines a sparse reach bonus, a small dense tracking term, and a time penalty:
[ r_t= \mathbf{1}{\mathrm{reach}}(t) w{\mathrm{step}}(t)r_{\mathrm{score}}(t) +\alpha_{\mathrm{dense}}r_{\mathrm{dense}}(t) -c_{\mathrm{time}}. ]
The sparse bonus is weighted by subgoal distance, so longer successful transitions earn more. The dense term helps early exploration before the first goal is reached. A curriculum gradually increases gravity, subgoal distance, object-tracking pressure, and action-mask duration. This formulation preserves the reference motion’s geometric intent while allowing RL to discover rolling, regrasping, and finger-gaiting strategies through contact simulation.
The ablation makes this point sharply. On held-out cuboid, hammer, and screwdriver motions, the dense-tracking policy reaches only about 2.6–2.7 sparse subgoals before stalling; TeleDexter reaches 32.6, 186.6, and 178.5, respectively.
3. The Learned Controller and Its Information Boundary
For SharpaWave, the observation has 142 dimensions; LeapHand uses 112. It contains joint positions and their sine/cosine encoding, the current object pose, gravity in the wrist frame, target and error vectors for fingertips and object pose, and the previous low-level command. An LSTM policy produces residual joint-position targets through a soft dead zone that lets the controller explicitly hold its previous command.
Two omissions are important: the controller receives neither contact force nor joint velocity. The authors exclude these signals because they are noisy or unavailable on the hardware. The policy must infer motion and contact consequences from kinematic state history, the previous command, gravity, and object-pose feedback. This makes hardware deployment simpler, while leaving ambiguous contact states unresolved. The appendix identifies tracking stalls caused by exactly this ambiguity: without touch, the controller cannot tell whether a finger is pressing the object or sliding past it.
Training uses SAPG in Isaac Gym with about 62,400 parallel environments on four RTX 5090 GPUs. Each object-specific controller consumes roughly (10^{10}) environment steps and about one day of training. The paper’s “single-stage” description means one RL stage and a shared reward recipe across contact modes. It still requires offline capture, geometry-aware retargeting, large-scale simulation, and a separately trained controller for each object.
4. Random Action Masking Bridges Asynchronous Hardware
Domain randomization covers hand and object dynamics, friction, object scale, external perturbations, observation noise, latency, and wrist initialization. TeleDexter adds random action masking as a stronger action-space regularizer.
With probability 0.15, the simulator selects three joint dimensions and freezes them at their previous commands for 1–10 control steps. Other joints receive the new action normally:
[
\tilde a_t[j]=
\begin{cases}
\tilde a_{t-1}[j], & j\in\mathcal{M}_t,
a_t[j], & \text{otherwise}.
\end{cases}
]
This exposes the policy to stale and desynchronized actuation, approximating motor lag, backlash, compliance, and missed SDK commands. Removing action masking drops real-world success from 66.7% to 33.3% on HammerUse, from 73.3% to 0% on ScrewdriverUse, and from 80.0% to 26.7% on CuboidReorient. Among the reported sim-to-real interventions, it is the most consequential single design.
5. Real-Time Teleoperation Has Two Control Modes
The complete deployment loop runs at 30 Hz on a Franka FR3 equipped with either a 22-DoF SharpaWave or a 16-DoF LeapHand. NOKOV optical motion capture tracks wrist pose, fingertip positions, and 6D object pose.
TeleDexter splits each trial into two phases:
- Reaching and grasping: standard vector retargeting drives the pre-grasp approach and establishes initial contact.
- In-hand manipulation: the operator switches to the learned co-tracking controller for reorientation, regrasping, finger gaiting, and tool use.
This division is practical and narrows the learned controller’s responsibility. TeleDexter handles the contact-rich regime after a usable grasp exists; the free-space reach and initial grasp remain under conventional retargeting.
Experiments: Where the Gain Appears
The main evaluation contains seven tasks, 15 real-world trials per method and task, and two metrics: full-task success rate (SR) and fraction of task stages completed (TP). Three tasks test in-hand reorientation; four test long-horizon tool use.
| Task | DexRT SR | GeoRT SR | DexGen SR | TeleDexter SR / TP |
|---|---|---|---|---|
| CylinderReorient | 6.7% | 0.0% | 0.0% | 80.0% / 86.7% |
| CuboidReorient | 26.7% | 0.0% | 0.0% | 80.0% / 86.7% |
| BunnyReorient | 0.0% | 0.0% | 0.0% | 66.7% / 77.8% |
| HammerUse | 0.0% | 0.0% | 0.0% | 66.7% / 86.7% |
| BrushSweep | 0.0% | 0.0% | 0.0% | 73.3% / 89.5% |
| ScrewdriverUse | 6.7% | 0.0% | 0.0% | 73.3% / 86.7% |
| BulbReplace | 0.0% | 0.0% | 0.0% | 86.7% / 95.6% |
| Average | 5.7% | 0.0% | 0.0% | 75.2% / 87.1% |
The stage-wise analysis is more informative than the average. Baselines often complete pickup and gross motion, then collapse at the first in-hand rotation, functional grasp switch, continuous axial rotation, or sustained tool contact. TeleDexter retains most trials through these stages. For example, 13 of 15 ScrewdriverUse trials complete continuous tightening, and all 15 BulbReplace trials that reach installation also survive both screw-in and unscrew stages.
Cross-embodiment results are promising but narrower. Using the same human references and adapting only geometry-aware retargeting, the LeapHand controller reaches 60.0% on cylinder and 73.3% on cuboid reorientation. An any-to-any target stress test reports 41.1 consecutive targets for the cylinder and 12.1 for the cuboid on average, showing both generalization beyond recorded sequences and sensitivity to object geometry.
From Teleoperation to Autonomous Policies
The authors collect 50 TeleDexter demonstrations per task and train RGB-conditioned Diffusion Policies with third-person and wrist cameras. On 15 trials, the autonomous policies achieve:
- 73.3% on HammerDriver;
- 46.7% on BulbInstall;
- 40.0% on BrushForward.
These results are modest compared with the teleoperation controller, yet they validate the data-engine argument: TeleDexter can collect contact-rich examples that standard teleoperation baselines fail to produce. The failure pattern also exposes perception bottlenecks. BrushForward loses most trials during the thin-handle grasp, while hammering fails mainly during repeated environment contact and bulb installation during precise alignment.
How to Interpret the 75.2% Result
The comparison strongly supports the value of learned hand-object co-tracking over pure kinematic retargeting for the tested tasks. Several qualifications matter when interpreting the headline number:
- Each task uses 15 trials, so a single trial changes SR by 6.7 percentage points.
- Objects are wrapped in medical bandage tape to increase friction.
- Controllers are object-specific and use about 50 minutes of reference capture plus a dedicated training run per object.
- Real-time deployment depends on a NOKOV optical motion-capture setup and marked objects.
- DexGen has no official implementation. The authors reimplement it and substitute TeleDexter’s co-tracking controller for an underspecified rollout-generation stage, which complicates a fully clean method comparison.
- SimToolReal is an autonomous object-centric tool policy included as a reference, with a different interface from the teleoperation baselines.
- The paper reports task success without a user study of operator workload, learnability, latency perception, or subjective control quality.
Failure Modes and Research Opportunities
The appendix gives a useful diagnosis of three real-world failures:
- interaction perturbation: impacts such as hammer strikes shift the in-hand object outside the free-space training distribution;
- contact-transition jam: compliant real fingers wedge against edges or concavities in ways rigid simulation misses;
- tracking stall: the hand keeps the object but cannot find a productive new contact, partly because the policy has no tactile observation.
These failures point directly to useful extensions: impact-aware reference collection and simulation, better compliant-contact models, tactile state estimation, markerless hand-object tracking, and an object-conditioned controller shared across shapes and tools.
Takeaway
TeleDexter’s essential computation is
[ \boxed{ \text{operator wrist, fingertips, and object motion} \rightarrow \text{hand-object geometric subgoals} \rightarrow \text{learned contact execution} }. ]
Its contribution is the middle-to-low-level bridge: human intent remains geometric and easy to specify, while a simulation-trained recurrent policy absorbs the burden of joint coordination, timing, and contact switching. Consecutive subgoals give the policy enough freedom to discover feasible strategies; geometry-aware references keep those strategies aligned with human intent; random action masking makes them survive imperfect hardware.
This is a convincing recipe for collecting rare, high-dexterity robot data and for executing long contact sequences under operator guidance. General human-level dexterity will require broader object generalization, touch, interaction-aware training, lighter perception infrastructure, and stronger evidence across operators and unstructured tasks.