Lixin Xu

[Paper Notes] TeleDexter: Towards Human-level Dexterous Teleoperation

2026-07-21T00:00:00+00:00

TeleDexter adds a learned low-level “cerebellum” between an operator’s intent and a dexterous robot hand. The operator and a tracked object provide synchronized fingertip-position and object-pose targets. A reinforcement-learning controller then chooses robot joint commands and contact transitions that realize those targets. This changes dexterous teleoperation from direct pose copying into goal-conditioned, contact-aware execution.

The mechanism is also the right boundary for reading the paper’s “human-level” claim. TeleDexter demonstrates in-hand reorientation, finger gaiting, and multi-stage tool use that conventional retargeting baselines rarely complete. It remains an object-specific controller trained from motion-capture data, requires real-time object tracking, observes no tactile signal, and is evaluated on seven fixed tasks with 15 trials each. The result is a strong contact-execution layer and a valuable demonstration generator; general human-level manipulation remains a much larger goal.

Paper Info

“Towards Human-level Dexterous Teleoperation” is by Puhao Li, Zeyuan Chen, Yingying Wu, Pengkun Wei, Yuyang Li, Tianyu Wang, Jiaxiao Shi, Mingrui Yu, Baoxiong Jia, Song-Chun Zhu, Tengyu Liu, and Siyuan Huang. The authors are affiliated with Tsinghua University, the State Key Laboratory of General Artificial Intelligence at BIGAI, and Peking University. These notes refer to arXiv:2607.11481v1, submitted July 13, 2026.

The Missing Layer in Dexterous Teleoperation

Kinematic teleoperation maps tracked human joints or geometric hand features into robot joint targets. It is intuitive and fast, but the mapping carries no model of contact force, friction, object inertia, or actuator response. This becomes fragile as soon as a finger must release, move around the object, and re-establish contact while other fingers keep the object stable.

TeleDexter inserts a learned dynamics prior into this loop. All quantities are expressed in the robot wrist frame. The arm independently follows the operator’s wrist through inverse kinematics, while the hand policy receives a co-tracking goal

[ g_t=\left(\hat{p}^{\mathrm{tip}}_t,\hat{T}^{o}_t\right), ]

where (\hat{p}^{\mathrm{tip}}_t) contains target fingertip positions and (\hat{T}^{o}_t=(\hat{x}^{o}_t,\hat{R}^{o}_t)\in SE(3)) is the target object pose. The policy outputs joint-position commands

[ a_t=\pi_\theta(o_t,g_t). ]

The goal specifies what geometry should be reached. Simulation-trained control supplies how to move contacts and joints to reach it. Tracking the object target matters as much as tracking the fingertips: identical fingertip positions can correspond to a stable regrasp, a slipping object, or a failed contact transition.

At deployment, motion capture measures an operator-side hand-object interaction while the robot-side state provides the current hand and object configuration. The resulting feedback loop is therefore grounded in two errors simultaneously: fingertip target versus robot fingertip state, and desired object motion versus the robot object’s measured motion.

1. Geometry-Aware Reference Motions

The low-level controller is trained from unscripted human hand-object interaction. For each object, the authors record 150 trajectories of 20 seconds each, totaling about 50 minutes. The recordings include in-hand translation, rotation, free play, finger gaiting, and tool-use motions.

Direct human-to-robot retargeting provides a useful initialization but can place fingers inside the object or destroy intended contacts. TeleDexter uses a two-stage pipeline:

vector-based retargeting aligns human and robot hand geometry;
object-mesh-aware optimization refines the whole trajectory.

The second stage solves

[ q^*{1:T}=\arg\min{q_{1:T}} \sum_{t=1}^{T} \left( L^t_{\mathrm{vec}} +\lambda_{\mathrm{surf}}L^t_{\mathrm{surf}} +\lambda_{\mathrm{pen}}L^t_{\mathrm{pen}} +\lambda_{\mathrm{col}}L^t_{\mathrm{col}} \right) +\lambda_{\mathrm{smooth}}L_{\mathrm{smooth}}. ]

The surface term attracts near-contact hand points to the object surface; the penetration term pushes hand points out of the object; the collision term prevents fingers from overlapping; and trajectory smoothness suppresses motion-capture jitter. This produces synchronized robot fingertip and object-pose targets with more plausible contact geometry. The later RL rollouts supply physical feasibility by resolving forces and friction that lie outside the geometric optimization.

2. Consecutive Subgoals Give RL Room to Discover Contact

Dense frame-by-frame imitation fixes a target at every instant. For dexterous contact, this can over-constrain exploration: the robot embodiment and dynamics may require a finger sequence that differs from the captured human timing.

TeleDexter samples consecutive co-tracking subgoals from each reference trajectory. For active subgoal (k), it measures per-finger position error, object-position error, and object-rotation error:

[ e^f_{t,k}=\lVert p^{\mathrm{tip}}{t,f}-\hat p^{\mathrm{tip}}{k,f}\rVert_2, \qquad e^{\mathrm{pos}}_{t,k}=\lVert x^o_t-\hat x^o_k\rVert_2, ]

[ e^{\mathrm{rot}}_{t,k} =\left\lVert \operatorname{Log}!\left(\hat R^{o\top}_kR^o_t\right) \right\rVert_2. ]

A subgoal counts as reached after all errors stay below their thresholds for 5–15 consecutive frames. The final tolerances are 3 cm per fingertip, 1 cm object translation, and 10° object rotation. Once the target is reached, the system samples a later subgoal; 10% of transitions switch to a different reference trajectory at the same temporal index. The policy must therefore connect configurations that were never adjacent in a human demonstration.

The reward combines a sparse reach bonus, a small dense tracking term, and a time penalty:

[ r_t= \mathbf{1}{\mathrm{reach}}(t) w{\mathrm{step}}(t)r_{\mathrm{score}}(t) +\alpha_{\mathrm{dense}}r_{\mathrm{dense}}(t) -c_{\mathrm{time}}. ]

The sparse bonus is weighted by subgoal distance, so longer successful transitions earn more. The dense term helps early exploration before the first goal is reached. A curriculum gradually increases gravity, subgoal distance, object-tracking pressure, and action-mask duration. This formulation preserves the reference motion’s geometric intent while allowing RL to discover rolling, regrasping, and finger-gaiting strategies through contact simulation.

The ablation makes this point sharply. On held-out cuboid, hammer, and screwdriver motions, the dense-tracking policy reaches only about 2.6–2.7 sparse subgoals before stalling; TeleDexter reaches 32.6, 186.6, and 178.5, respectively.

3. The Learned Controller and Its Information Boundary

For SharpaWave, the observation has 142 dimensions; LeapHand uses 112. It contains joint positions and their sine/cosine encoding, the current object pose, gravity in the wrist frame, target and error vectors for fingertips and object pose, and the previous low-level command. An LSTM policy produces residual joint-position targets through a soft dead zone that lets the controller explicitly hold its previous command.

Two omissions are important: the controller receives neither contact force nor joint velocity. The authors exclude these signals because they are noisy or unavailable on the hardware. The policy must infer motion and contact consequences from kinematic state history, the previous command, gravity, and object-pose feedback. This makes hardware deployment simpler, while leaving ambiguous contact states unresolved. The appendix identifies tracking stalls caused by exactly this ambiguity: without touch, the controller cannot tell whether a finger is pressing the object or sliding past it.

Training uses SAPG in Isaac Gym with about 62,400 parallel environments on four RTX 5090 GPUs. Each object-specific controller consumes roughly (10^{10}) environment steps and about one day of training. The paper’s “single-stage” description means one RL stage and a shared reward recipe across contact modes. It still requires offline capture, geometry-aware retargeting, large-scale simulation, and a separately trained controller for each object.

4. Random Action Masking Bridges Asynchronous Hardware

Domain randomization covers hand and object dynamics, friction, object scale, external perturbations, observation noise, latency, and wrist initialization. TeleDexter adds random action masking as a stronger action-space regularizer.

With probability 0.15, the simulator selects three joint dimensions and freezes them at their previous commands for 1–10 control steps. Other joints receive the new action normally:

[ \tilde a_t[j]= \begin{cases} \tilde a_{t-1}[j], & j\in\mathcal{M}_t,
a_t[j], & \text{otherwise}. \end{cases} ]

This exposes the policy to stale and desynchronized actuation, approximating motor lag, backlash, compliance, and missed SDK commands. Removing action masking drops real-world success from 66.7% to 33.3% on HammerUse, from 73.3% to 0% on ScrewdriverUse, and from 80.0% to 26.7% on CuboidReorient. Among the reported sim-to-real interventions, it is the most consequential single design.

5. Real-Time Teleoperation Has Two Control Modes

The complete deployment loop runs at 30 Hz on a Franka FR3 equipped with either a 22-DoF SharpaWave or a 16-DoF LeapHand. NOKOV optical motion capture tracks wrist pose, fingertip positions, and 6D object pose.

TeleDexter splits each trial into two phases:

Reaching and grasping: standard vector retargeting drives the pre-grasp approach and establishes initial contact.
In-hand manipulation: the operator switches to the learned co-tracking controller for reorientation, regrasping, finger gaiting, and tool use.

This division is practical and narrows the learned controller’s responsibility. TeleDexter handles the contact-rich regime after a usable grasp exists; the free-space reach and initial grasp remain under conventional retargeting.

Experiments: Where the Gain Appears

The main evaluation contains seven tasks, 15 real-world trials per method and task, and two metrics: full-task success rate (SR) and fraction of task stages completed (TP). Three tasks test in-hand reorientation; four test long-horizon tool use.

Task	DexRT SR	GeoRT SR	DexGen SR	TeleDexter SR / TP
CylinderReorient	6.7%	0.0%	0.0%	80.0% / 86.7%
CuboidReorient	26.7%	0.0%	0.0%	80.0% / 86.7%
BunnyReorient	0.0%	0.0%	0.0%	66.7% / 77.8%
HammerUse	0.0%	0.0%	0.0%	66.7% / 86.7%
BrushSweep	0.0%	0.0%	0.0%	73.3% / 89.5%
ScrewdriverUse	6.7%	0.0%	0.0%	73.3% / 86.7%
BulbReplace	0.0%	0.0%	0.0%	86.7% / 95.6%
Average	5.7%	0.0%	0.0%	75.2% / 87.1%

The stage-wise analysis is more informative than the average. Baselines often complete pickup and gross motion, then collapse at the first in-hand rotation, functional grasp switch, continuous axial rotation, or sustained tool contact. TeleDexter retains most trials through these stages. For example, 13 of 15 ScrewdriverUse trials complete continuous tightening, and all 15 BulbReplace trials that reach installation also survive both screw-in and unscrew stages.

Cross-embodiment results are promising but narrower. Using the same human references and adapting only geometry-aware retargeting, the LeapHand controller reaches 60.0% on cylinder and 73.3% on cuboid reorientation. An any-to-any target stress test reports 41.1 consecutive targets for the cylinder and 12.1 for the cuboid on average, showing both generalization beyond recorded sequences and sensitivity to object geometry.

From Teleoperation to Autonomous Policies

The authors collect 50 TeleDexter demonstrations per task and train RGB-conditioned Diffusion Policies with third-person and wrist cameras. On 15 trials, the autonomous policies achieve:

73.3% on HammerDriver;
46.7% on BulbInstall;
40.0% on BrushForward.

These results are modest compared with the teleoperation controller, yet they validate the data-engine argument: TeleDexter can collect contact-rich examples that standard teleoperation baselines fail to produce. The failure pattern also exposes perception bottlenecks. BrushForward loses most trials during the thin-handle grasp, while hammering fails mainly during repeated environment contact and bulb installation during precise alignment.

How to Interpret the 75.2% Result

The comparison strongly supports the value of learned hand-object co-tracking over pure kinematic retargeting for the tested tasks. Several qualifications matter when interpreting the headline number:

Each task uses 15 trials, so a single trial changes SR by 6.7 percentage points.
Objects are wrapped in medical bandage tape to increase friction.
Controllers are object-specific and use about 50 minutes of reference capture plus a dedicated training run per object.
Real-time deployment depends on a NOKOV optical motion-capture setup and marked objects.
DexGen has no official implementation. The authors reimplement it and substitute TeleDexter’s co-tracking controller for an underspecified rollout-generation stage, which complicates a fully clean method comparison.
SimToolReal is an autonomous object-centric tool policy included as a reference, with a different interface from the teleoperation baselines.
The paper reports task success without a user study of operator workload, learnability, latency perception, or subjective control quality.

Failure Modes and Research Opportunities

The appendix gives a useful diagnosis of three real-world failures:

interaction perturbation: impacts such as hammer strikes shift the in-hand object outside the free-space training distribution;
contact-transition jam: compliant real fingers wedge against edges or concavities in ways rigid simulation misses;
tracking stall: the hand keeps the object but cannot find a productive new contact, partly because the policy has no tactile observation.

These failures point directly to useful extensions: impact-aware reference collection and simulation, better compliant-contact models, tactile state estimation, markerless hand-object tracking, and an object-conditioned controller shared across shapes and tools.

Takeaway

TeleDexter’s essential computation is

[ \boxed{ \text{operator wrist, fingertips, and object motion} \rightarrow \text{hand-object geometric subgoals} \rightarrow \text{learned contact execution} }. ]

Its contribution is the middle-to-low-level bridge: human intent remains geometric and easy to specify, while a simulation-trained recurrent policy absorbs the burden of joint coordination, timing, and contact switching. Consecutive subgoals give the policy enough freedom to discover feasible strategies; geometry-aware references keep those strategies aligned with human intent; random action masking makes them survive imperfect hardware.

This is a convincing recipe for collecting rare, high-dexterity robot data and for executing long contact sequences under operator guidance. General human-level dexterity will require broader object generalization, touch, interaction-aware training, lighter perception infrastructure, and stronger evidence across operators and unstructured tasks.

TeleDexter 在操作者意图与灵巧手之间加入了一个学习得到的低层“类小脑”控制器。 操作者的手和被跟踪物体共同给出同步的指尖位置目标与物体位姿目标；强化学习控制器负责生成机器人关节指令和接触切换，使这些目标在真实动力学下实现。灵巧遥操作由此从直接复制姿态，转变为目标条件、接触感知的低层执行。

这条机制也是理解论文“human-level”表述的准确边界。TeleDexter 展示了传统 retargeting 基线几乎无法完成的手内重定向、finger gaiting 和多阶段工具使用；但它仍然是基于动作捕捉数据训练的物体专用控制器，需要实时物体跟踪，没有触觉观测，并且只在 7 个固定任务上各测试 15 次。它证明了一个很强的接触执行层和数据采集器，还没有构成通用的人类水平操作系统。

论文信息

论文 “Towards Human-level Dexterous Teleoperation” 的作者是 Puhao Li、Zeyuan Chen、Yingying Wu、Pengkun Wei、Yuyang Li、Tianyu Wang、Jiaxiao Shi、Mingrui Yu、Baoxiong Jia、Song-Chun Zhu、Tengyu Liu 和 Siyuan Huang，作者来自 Tsinghua University、BIGAI 的 State Key Laboratory of General Artificial Intelligence 和 Peking University。本文依据 arXiv:2607.11481v1，提交日期为 2026 年 7 月 13 日。

灵巧遥操作缺失的中间层

运动学遥操作把跟踪到的人手关节或几何特征映射成机器人关节目标。这种接口直观、延迟低，却不包含接触力、摩擦、物体惯量和执行器响应。当某根手指需要离开物体、绕行并重新建立接触，同时其他手指还要稳定物体时，纯运动学映射会迅速变得脆弱。

TeleDexter 在控制回路里插入一个学习得到的动力学先验。所有量都表达在机器人腕部坐标系中。机械臂通过逆运动学独立跟踪操作者手腕；灵巧手策略接收 co-tracking 目标

[ g_t=\left(\hat{p}^{\mathrm{tip}}_t,\hat{T}^{o}_t\right), ]

其中 (\hat{p}^{\mathrm{tip}}_t) 是目标指尖位置，(\hat{T}^{o}_t=(\hat{x}^{o}_t,\hat{R}^{o}_t)\in SE(3)) 是目标物体位姿。策略输出关节位置指令

[ a_t=\pi_\theta(o_t,g_t). ]

目标描述应该达到什么几何状态，仿真中训练的策略负责决定怎样移动关节和切换接触才能到达该状态。物体目标与指尖目标同样重要：相同的指尖位置可能对应稳定换抓、物体滑移，也可能对应已经失败的接触切换。

真实部署时，动作捕捉系统测量操作者侧的手—物体交互，机器人侧状态则给出当前灵巧手和物体配置。控制回路同时闭合两类误差：操作者指尖目标与机器人指尖状态之间的误差，以及期望物体运动与机器人侧实测物体运动之间的误差。

1. 带物体几何约束的参考动作

低层控制器从无脚本的人类手—物体交互中学习。每个物体采集 150 段、每段 20 秒的轨迹，总计约 50 分钟，内容覆盖手内平移、旋转、自由操作、finger gaiting 和工具使用。

直接进行人手到机器人手的 retargeting 可以提供良好初值，但可能让手指穿进物体，也可能破坏原动作中的预期接触。TeleDexter 使用两阶段流程：

基于向量的 retargeting 对齐人手与机器人手的几何关系；
引入物体 mesh，对整段轨迹做几何后优化。

第二阶段求解

surface loss 把近接触的手部表面点拉向物体表面；penetration loss 把穿入物体的点推出；collision loss 避免手指互相重叠；轨迹平滑项抑制动作捕捉抖动。输出的机器人指尖目标与物体位姿目标在接触几何上更合理。力和摩擦仍需后续 RL 在物理仿真中解决，几何优化本身不负责动力学可行性。

2. 连续子目标给 RL 留出发现接触策略的空间

逐帧 dense tracking 会在每个时刻固定参考目标。灵巧接触对执行时序非常敏感，机器人形态和动力学往往需要一套不同于人手原始时序的指法，逐帧约束容易限制探索。

TeleDexter 从每条参考轨迹中采样连续 co-tracking 子目标。对于当前子目标 (k)，系统计算每根手指的位置误差、物体位置误差和物体旋转误差：

[ e^f_{t,k}=\lVert p^{\mathrm{tip}}{t,f}-\hat p^{\mathrm{tip}}{k,f}\rVert_2, \qquad e^{\mathrm{pos}}_{t,k}=\lVert x^o_t-\hat x^o_k\rVert_2, ]

[ e^{\mathrm{rot}}_{t,k} =\left\lVert \operatorname{Log}!\left(\hat R^{o\top}_kR^o_t\right) \right\rVert_2. ]

所有误差连续 5–15 帧保持在阈值内，才算到达子目标。训练结束时的阈值为：每个指尖 3 cm、物体平移 1 cm、物体旋转 10°。到达后再采样更后面的子目标；其中 10% 的转移会保持时间索引不变、切换到另一条参考轨迹。因此，策略还要连接人类示范中从未相邻出现过的两个配置。

奖励由稀疏到达奖励、小权重 dense tracking 奖励和时间惩罚组成：

[ r_t= \mathbf{1}{\mathrm{reach}}(t) w{\mathrm{step}}(t)r_{\mathrm{score}}(t) +\alpha_{\mathrm{dense}}r_{\mathrm{dense}}(t) -c_{\mathrm{time}}. ]

子目标跨度越大，成功后的稀疏奖励越高；dense 项帮助策略在尚未到达第一个目标时形成早期探索信号。课程学习逐渐增加重力、子目标间距、物体跟踪权重和 action mask 时长。这套设计保留了参考动作的几何意图，同时允许 RL 在接触仿真中发现滚动、换抓和 finger gaiting 策略。

消融结果非常直接。在 held-out 的 cuboid、hammer 和 screwdriver 动作上，dense tracking 策略在停滞前只能到达约 2.6–2.7 个稀疏子目标；TeleDexter 分别达到 32.6、186.6 和 178.5 个。

3. 控制器观测与信息边界

SharpaWave 的观测维度为 142，LeapHand 为 112。观测包含关节位置及其正余弦编码、当前物体位姿、腕部坐标系中的重力方向、指尖与物体位姿的目标和误差，以及上一时刻的低层指令。LSTM 策略输出残差式关节位置目标，并使用 soft dead zone，让控制器可以明确选择保持上一条指令。

两个缺失信号尤其重要：策略既不接收接触力，也不接收关节速度。作者认为这些信号在硬件上噪声较大或不可得，因此让策略通过运动学状态历史、上一条指令、重力和物体位姿反馈来推断动作与接触结果。这样更容易部署到真实硬件，也留下了无法区分的接触状态。附录中的 tracking stall 正是由这个边界引起：没有触觉时，策略无法判断手指正在压住物体，还是已经从物体旁边滑过。

训练在 Isaac Gym 中使用 SAPG，约 62,400 个并行环境分布在 4 张 RTX 5090 上。每个物体专用控制器需要约 (10^{10}) 个环境步，训练时间约一天。论文所说的“single-stage”指单个 RL 阶段，以及所有接触模式共用一套奖励配方；完整流程仍包含离线采集、带几何约束的 retargeting、大规模仿真和逐物体训练。

4. Random Action Masking 处理硬件异步响应

Domain randomization 覆盖灵巧手与物体动力学、摩擦、物体尺度、外部扰动、观测噪声、延迟和腕部初始姿态。TeleDexter 进一步加入 random action masking，在动作空间中施加强正则。

每个仿真步以 0.15 概率选中 3 个关节维度，并让它们在 1–10 个控制步内保持上一条指令；其余关节正常执行新动作：

[ \tilde a_t[j]= \begin{cases} \tilde a_{t-1}[j], & j\in\mathcal{M}_t,
a_t[j], & \text{其他情况}. \end{cases} ]

这种训练主动制造过期、不同步的执行器响应，用来近似电机延迟、backlash、被动柔顺和 SDK 丢指令。移除 action masking 后，HammerUse 的真实成功率从 66.7% 降到 33.3%，ScrewdriverUse 从 73.3% 降到 0%，CuboidReorient 从 80.0% 降到 26.7%。在论文报告的 sim-to-real 设计中，这是影响最大的单项机制。

5. 真实遥操作包含两种控制模式

完整系统以 30 Hz 运行，平台是 Franka FR3，加装 22-DoF SharpaWave 或 16-DoF LeapHand。NOKOV 光学动作捕捉系统实时跟踪手腕、指尖和物体 6D 位姿。

每次操作分为两个阶段：

接近与抓取：标准 vector retargeting 负责 pre-grasp 接近和建立初始接触。
手内操作：建立稳定抓取后，操作者切换到学习得到的 co-tracking 控制器，完成重定向、换抓、finger gaiting 和工具使用。

这种分工缩小了学习控制器的责任范围。TeleDexter 聚焦于形成可用抓取之后的接触密集阶段，没有训练一个从自由空间接近一直覆盖到任务结束的端到端策略。

实验：优势出现在哪些阶段

主实验包含 7 个任务，每个方法、每个任务进行 15 次真实实验。指标包括完整任务成功率（SR）与完成阶段比例（TP）。3 个任务测试手内重定向，4 个任务测试长时序工具使用。

任务	DexRT SR	GeoRT SR	DexGen SR	TeleDexter SR / TP
CylinderReorient	6.7%	0.0%	0.0%	80.0% / 86.7%
CuboidReorient	26.7%	0.0%	0.0%	80.0% / 86.7%
BunnyReorient	0.0%	0.0%	0.0%	66.7% / 77.8%
HammerUse	0.0%	0.0%	0.0%	66.7% / 86.7%
BrushSweep	0.0%	0.0%	0.0%	73.3% / 89.5%
ScrewdriverUse	6.7%	0.0%	0.0%	73.3% / 86.7%
BulbReplace	0.0%	0.0%	0.0%	86.7% / 95.6%
平均	5.7%	0.0%	0.0%	75.2% / 87.1%

分阶段结果比平均值更能说明机制差异。基线通常可以完成抓取和大范围移动，进入第一次手内旋转、功能性换抓、连续轴向旋转或持续工具接触后迅速失败。TeleDexter 在这些阶段仍保留了大部分实验。例如，15 次 ScrewdriverUse 中有 13 次完成连续拧紧；BulbReplace 中所有进入安装环节的 15 次实验都通过了拧入与拧出两个阶段。

跨形态结果很有潜力，但覆盖面更窄。使用相同的人类参考动作，仅调整 geometry-aware retargeting，LeapHand 控制器在 cylinder 和 cuboid 重定向上分别达到 60.0% 与 73.3%。any-to-any 目标压力测试中，cylinder 平均连续到达 41.1 个目标，cuboid 为 12.1 个，说明策略可以超出原始参考序列，同时也对物体几何高度敏感。

从遥操作数据到自主策略

作者为每个任务采集 50 条 TeleDexter 示范，使用第三人称相机和腕部相机 RGB 训练 Diffusion Policy。每项测试 15 次，自主策略的结果为：

HammerDriver：73.3%；
BulbInstall：46.7%；
BrushForward：40.0%。

这些结果低于遥操作控制器本身，但支持了 data engine 的论点：TeleDexter 可以采集传统遥操作基线难以产生的接触密集示范。失败分布也揭示了感知瓶颈。BrushForward 的主要损失发生在细而不规则的刷柄抓取阶段；hammer 的失败集中于反复环境接触；bulb installation 的瓶颈则是精确对齐。

75.2% 这个结果能说明什么

对比实验有力支持了学习式 hand-object co-tracking 在这些任务上优于纯运动学 retargeting。解读 headline number 时还需要保留以下限定：

每项任务只有 15 次实验，一次实验对应 6.7 个百分点。
测试物体包裹了医用绷带胶带，以增加表面摩擦。
控制器按物体训练，每个物体约需 50 分钟参考采集和一次独立训练。
真实部署依赖 NOKOV 光学动作捕捉系统与带 marker 的物体。
DexGen 没有官方实现。作者自行复现，并因原论文 rollout 生成阶段描述不足，使用 TeleDexter 的 co-tracking controller 代替其中间策略，因此方法比较无法完全排除实现差异。
SimToolReal 是自主式 object-centric 工具策略，只作为参考系统加入；它与遥操作基线的接口不同。
论文报告了任务成功率，没有测试操作者负担、学习成本、延迟感知或主观控制体验。

失败模式与后续方向

附录把真实失败归纳为三类：

interaction perturbation：锤击等冲击会让手中物体瞬间偏离只含自由空间交互的训练分布；
contact-transition jam：真实手指的柔顺性会让手指卡在边缘或凹面，刚体仿真没有充分覆盖这种现象；
tracking stall：手没有丢掉物体，却找不到能继续推进的接触；缺少触觉观测使策略更难判断当前接触状态。

这些问题直接指向几条扩展路线：加入冲击接触数据与仿真、提升柔顺接触建模、引入触觉状态估计、使用 markerless 手—物体跟踪，以及训练跨形状和工具共享的 object-conditioned controller。

总结

TeleDexter 的核心计算可以写成

[ \boxed{ \text{操作者手腕、指尖和物体运动} \rightarrow \text{手—物体几何子目标} \rightarrow \text{学习式接触执行} }. ]

它补上的正是中低层桥梁：人类仍然用容易表达的几何目标给出意图，仿真中训练的循环策略承担关节协调、接触时序和接触切换。连续子目标给策略留下发现可行动作的空间；geometry-aware references 让动作维持人类意图；random action masking 让策略可以适应不完美的硬件响应。

这套方法非常适合采集稀缺的高灵巧度机器人数据，也能在操作者指导下执行长接触序列。走向更广义的人类水平灵巧操作，还需要跨物体泛化、触觉、包含环境冲击的训练、更轻量的感知基础设施，以及覆盖更多操作者和开放任务的证据。

[Paper Notes] FIGAROH: a Python toolbox for dynamic identification and geometric calibration of robots and humans

2026-07-16T00:00:00+00:00

FIGAROH is a Python toolbox for two closely related model-correction problems: geometric calibration, which estimates errors in robot kinematics, and dynamic identification, which estimates inertial, friction, actuator, and sensor parameters. Its most useful systems idea is the complete experimental loop: construct the identification model from URDF, determine which parameter combinations are observable, design informative postures or motions, estimate parameters, validate the result, and update the robot model.

This note reads the original paper together with the current figaroh-plus codebase. The distinction matters. The paper describes the unified scientific workflow demonstrated on robots and a human subject. The current v0.4.5 repository has grown into a more modular library with robust calibration, additional solvers, physical-consistency projection, URDF export, multi-backend interfaces, and reporting/verification tools.

Paper Info

The paper is “FIGAROH: a Python toolbox for dynamic identification and geometric calibration of robots and humans” by Dinh Vinh Thanh Nguyen, Vincent Bonnet, Maxime Sabbah, Maxime Gautier, Pierre Fernbach, and Florent Lamiraux. The provided PDF is the HAL version of the paper published at the 2023 IEEE-RAS International Conference on Humanoid Robots, DOI 10.1109/Humanoids57100.2023.10375232. Despite the local filename FIGAROH_IROS_2022-5.pdf, it is a Humanoids 2023 paper.

The current GitHub repository is a maintained fork of the original LAAS GitLab project. It publishes the figaroh Python package, while runnable robot examples live in a separate figaroh-examples repository.

Two Problems in One Toolbox

Geometric and dynamic identification use similar mathematics and experimental logic, although they correct different parts of the model.

Geometric calibration

A URDF gives each joint a nominal transform. Real assembly introduces small translation and rotation errors. FIGAROH parameterizes the local variation of joint (j) as

[ \Phi_{g,j} = [\delta p_x,\delta p_y,\delta p_z, \delta\phi_x,\delta\phi_y,\delta\phi_z]^T. ]

The forward-kinematics residual is locally linearized:

[ \Delta P=R_g(q,p)\Phi_g, ]

where (R_g) is a geometric regressor or kinematic Jacobian connecting parameter errors to measured pose errors. Measurements can be full poses, positions, orientations, camera observations, motion-capture markers, or geometric constraints such as contact with a plane.

The six-parameter representation does not mean that all six parameters of every joint can be independently recovered. The paper notes that a revolute joint has at most four independently identifiable geometric parameters and a prismatic joint at most two; collinear axes introduce additional dependencies. FIGAROH uses rank analysis to keep the observable parameter combinations.

Dynamic identification

Rigid-body inverse dynamics is linear in inertial and several extended mechanical parameters:

[ D=R_d(q,\dot q,\ddot q)\Phi_d. ]

Here (D) is a stack of measured torques, motor currents, or external wrenches; (R_d) is the dynamic regressor; and (\Phi_d) can include:

ten inertial parameters per body: mass, first moment, and six inertia-tensor terms;
joint viscous and Coulomb friction;
torque-sensor offsets;
actuator rotor inertia and drive-chain friction;
current-to-torque or sensor gains.

The identifiable set depends on available measurements. Joint torques can identify inertial and joint parameters. Combining joint torque with motor current exposes drive-chain terms. External wrench measurements identify a different subset. FIGAROH builds the regressor according to this measurement contract.

Rank Deficiency and Base Parameters

Robot regressors are generally rank deficient. Some physical parameters always appear together in the equations, so data cannot separate them regardless of sample count. FIGAROH numerically applies QR decomposition with column pivoting to rewrite the original regressor in terms of full-rank base parameters:

[ Y=R\Phi=R_b\Phi_b, \qquad \operatorname{rank}(R_b)=\dim(\Phi_b). ]

This step is central. A least-squares solver can produce numbers for an underdetermined parameterization, but those numbers are not uniquely supported by the measurements. Base-parameter reduction makes the estimation problem honest: the solver estimates only independent combinations that the experiment can observe.

The paper emphasizes the interpretability of numerical QR. It can expose which original columns combine into each base parameter, and it can produce different static and dynamic base sets. For example, static data removes velocity and acceleration terms, leaving mass and center-of-mass combinations.

Parameter Estimation

Once the base regressor is full rank, the simplest estimate is ordinary least squares:

[ \hat\Phi_b = (R_b^TR_b)^{-1}R_b^TY. ]

Weighted least squares compensates for measurement channels with different noise levels or units. The paper also covers total least squares for experiments with loaded and unloaded configurations, iterative least squares and Levenberg-Marquardt for nonlinear geometric calibration, and constrained optimization for physically plausible inertial parameters.

The current code retains the regressor/QR workflow and broadens solver support. BaseIdentification.solve() runs the standard identification path, while solve_with_custom_solver() can invoke least squares, QR/SVD-style methods, Ridge, Lasso, Elastic Net, robust regression, and constrained variants through figaroh.tools.LinearSolver.

Physical consistency deserves separate treatment. An unconstrained fit can yield negative mass or an impossible inertia tensor while still minimizing torque error. The paper uses quadratic constrained programming to recover individually meaningful parameters. The current code additionally provides optional pseudo-inertia SDP/LMI projection and full-parameter reconstruction from base parameters, with CAD-derived mass, center-of-mass, and symmetry constraints. These paths require an SDP solver and are disabled by default.

Optimal Experiment Design

FIGAROH treats data collection as part of identification. Random postures may leave the regressor poorly conditioned; random motions can waste robot time and become unsafe for humanoids. The toolbox therefore designs Optimal Exciting Postures (OEP) for calibration and static identification, and Optimal Exciting Motions (OEM) for dynamic identification.

The paper uses singular-value criteria derived from the regressor, including determinant-like information volume, condition number, and minimum singular value. Informally, a useful experiment makes the regressor columns distinguishable:

[ \text{good excitation} \Longrightarrow \sigma_{\min}(R)\text{ increases and } \kappa(R)=\frac{\sigma_{\max}(R)}{\sigma_{\min}(R)} \text{ decreases}. ]

Selecting calibration postures

Candidate postures receive information matrices (\Sigma_i) and continuous weights (\omega_i). FIGAROH solves

[ \max_{\omega} \Psi!\left(\sum_i\omega_i\Sigma_i\right), \qquad \sum_i\omega_i=1, ]

then ranks candidates by weight. The excitation score eventually reaches a peak or plateau, which gives a data-driven stopping point for the number of postures. The paper illustrates a case where the criterion peaks near 40 postures.

Postures are optimized under joint limits, torque limits, workspace bounds, and self-collision constraints. The paper uses Pinocchio/hpp-fcl geometry and IPOPT-based nonlinear optimization.

Generating exciting motions

OEM generation connects consecutive OEP using cubic splines. The optimizer adjusts spline waypoints while respecting endpoint positions, zero endpoint velocity, joint/velocity/torque limits, workspace constraints, and collision avoidance. The objective improves the accumulated dynamic base regressor over the trajectory.

This design closes a loop that many system-identification scripts leave manual:

URDF + measurement setup
        -> identifiable regressor
        -> optimal postures/motions
        -> robot data collection
        -> parameter estimation
        -> cross-validation and model update

What the Paper Demonstrates

The paper evaluates the workflow across human, humanoid, mobile-manipulator, and serial-arm cases.

For a 43-DoF human model, motion capture and force-plate data are used for geometry and dynamics. Compared with anthropometric-table parameters, identified parameters reduce average external-wrench RMSE from 19.47 N to 16.84 N for force and from 33.22 N·m to 16.77 N·m for moment.

For TALOS and TIAGo torso-arm geometric calibration, end-effector position RMSE drops from 14.1 mm to 0.3 mm on TALOS and from 16.9 mm to 1.5 mm on TIAGo. TALOS whole-body calibration with three-point planar contact and 31 automatically generated OEP reduces RMSE from 10.4 mm to 3.3 mm.

TIAGo dynamic identification includes drive-chain parameters and its differential wrist transmission; the identified model predicts measured motor current within 1%. The toolbox also reproduces published Stäubli TX40 inertial identification results and obtains 1.76 mm end-effector position RMSE for UR10 eye-in-hand calibration.

These examples show the toolbox’s breadth. They are heterogeneous case studies with different sensors and metrics, so they do not form one standardized benchmark.

Reading the Current `figaroh-plus` Code

The current source follows three practical layers.

Workflow layer

figaroh.calibration.BaseCalibration orchestrates configuration loading, measurement validation, QR-based parameter selection, nonlinear optimization, iterative outlier removal, metrics, plots, reports, and verification. Robot-specific behavior can be provided through a subclass cost function. The implementation uses SE(3) log-map residuals for pose calibration and can apply separate weights to position and orientation channels.

figaroh.identification.BaseIdentification loads trajectory data, computes and reduces the dynamic regressor, performs filtering/decimation, solves for base parameters, reconstructs predictions, evaluates RMSE/correlation/condition number, and optionally applies physical-consistency or full-parameter reconstruction.

BaseOptimalCalibration and BaseOptimalTrajectory provide the optimization framework for posture selection and spline trajectory design. IPOPT support relies on cyipopt, which is intentionally omitted from the simple pip dependency set; the repository recommends its Conda environment for this path.

Tools layer

The reusable tools include RegressorBuilder, QRDecomposer, LinearSolver, collision utilities, IPOPT wrappers, URDF export, visualization, result provenance, and report generation. The current library can emit self-contained HTML diagnostics, machine-readable pass/fail verification reports, and static comparisons between two runs.

This V&V layer is a meaningful extension beyond the paper. Calibration quality is more than a fitted RMSE: the code reports per-DoF residuals, held-out before/after validation, parameter uncertainty, strongly correlated parameters, and regressor conditioning. verify() converts configured thresholds into a CI-friendly verdict.

Backend layer

The source currently contains real PinocchioBackend and MuJoCoBackend implementations. Pinocchio remains the default and provides the most complete URDF/regressor path. MuJoCo implements mass, bias, kinematics, and contact-related computations, but its analytical parameter regressor delegates to a lazily loaded Pinocchio model because MuJoCo does not expose the same runtime inertial-parameter regressor.

The architecture document also discusses Genesis and Isaac Sim backends, marked as future work. There are no genesis.py or isaacsim.py implementations in the checked v0.4.5 source tree, so users should currently treat backend support as Pinocchio plus optional MuJoCo.

Paper Versus Current Repository

The scientific spine remains stable:

URDF-based rigid multi-body modeling;
geometric and dynamic regressors;
numerical base-parameter extraction;
least-squares or constrained estimation;
OEP/OEM experiment design;
statistical validation and model export.

The plus repository modernizes the engineering surface. It uses a src/ package layout, abstract workflow classes, unified YAML parsing with legacy conversion, Python packaging on PyPI, broader regression solvers, optional physically consistent reconstruction, model-export validation, richer reports, and an explicit backend interface.

Several practical boundaries are visible in the code:

The package has no registered command-line entry point; users work through Python classes and project scripts.
Robot examples and some datasets are maintained in the separate examples repository.
BaseCalibration is extensible by design and may require a task-specific subclass or cost function.
Advanced OEP/OEM and SDP workflows bring native optimization dependencies that are more involved than pip install figaroh.
Documentation describes a broader backend roadmap than the checked source currently implements.

This is a research toolbox that has accumulated production-oriented checks, not a black-box auto-calibration service. Users still need to define the measurement model, frame conventions, active parameters, excitation constraints, and validation protocol correctly.

Strengths

FIGAROH’s strongest quality is end-to-end consistency. The same robot model drives regressor construction, observability analysis, experiment design, estimation, collision checking, and final model update. This reduces the chance that each stage uses subtly different joint ordering, frames, or parameter conventions.

The second strength is support for tree-structured and floating-base systems. The workflow covers industrial manipulators, mobile manipulators, humanoids, and human biomechanical models. Optimal excitation is especially valuable for these systems because manually selected configurations become unsafe or uninformative as kinematic complexity grows.

The current code’s validation tooling is also well chosen. A parameter fit should be judged on held-out measurements and physical plausibility, with uncertainty and conditioning made visible. HTML reports and machine-readable verdicts make that discipline easier to repeat.

Limitations and Cautions

Identification quality remains bounded by the assumed model. Unmodeled flexibility, backlash, temperature-dependent friction, cable effects, timing errors, and sensor bias can leak into the estimated rigid-body parameters. Numerical differentiation of positions to obtain velocity and acceleration amplifies noise, making filtering and excitation design essential.

Geometric calibration starts from local linearization, while the nonlinear solve still depends on a reasonable initial model and correct frame conventions. Base parameters are identifiable combinations; recovering every URDF inertial term requires priors or physical constraints and may remain ambiguous.

The original paper leaves joint geometric-dynamic identification and nonlinear flexibility as future directions. The current repository improves physical reconstruction and calibration validation, but it still separates the primary calibration and identification workflows. Documentation and APIs are evolving quickly, and v0.4.5 is classified as beta software.

Takeaway

FIGAROH is best understood as an experiment-design and model-identification framework, not simply a least-squares implementation. Its key chain is:

[ \text{measurement contract} \rightarrow \text{regressor and observability} \rightarrow \text{optimal excitation} \rightarrow \text{parameter estimation} \rightarrow \text{held-out validation} \rightarrow \text{updated model}. ]

The paper establishes that this chain can serve very different robots and humans. The current figaroh-plus code turns it into a cleaner library with stronger verification and export support. For serious calibration work, the value is precisely this integration: the toolbox helps decide what can be identified, which experiment will identify it, and whether the resulting model actually improved.

FIGAROH 是一个同时解决两类模型修正问题的 Python toolbox：geometric calibration 用来估计机器人运动学误差，dynamic identification 用来估计惯性、摩擦、执行器和传感器参数。它最有价值的系统设计是一条完整实验闭环：从 URDF 构建 identification model，判断哪些参数组合可观测，设计有信息量的姿态或运动，估计参数，验证结果，再更新机器人模型。

这篇笔记同时阅读原始论文和当前 figaroh-plus 代码。两者需要分开理解。论文描述统一 scientific workflow，并在机器人和人体数据上验证；当前 v0.4.5 仓库已经发展为更模块化的 library，增加 robust calibration、更多 solvers、physical-consistency projection、URDF export、multi-backend interface 以及 reporting/verification tools。

论文信息

论文题目是 “FIGAROH: a Python toolbox for dynamic identification and geometric calibration of robots and humans”，作者为 Dinh Vinh Thanh Nguyen, Vincent Bonnet, Maxime Sabbah, Maxime Gautier, Pierre Fernbach, and Florent Lamiraux。用户提供的 PDF 是发表于 2023 IEEE-RAS International Conference on Humanoid Robots 的 HAL 版本，DOI 为 10.1109/Humanoids57100.2023.10375232。虽然本地文件名是 FIGAROH_IROS_2022-5.pdf，论文实际来自 Humanoids 2023。

当前 GitHub 仓库是原 LAAS GitLab 项目的持续维护 fork，发布 figaroh Python package；可运行的机器人例程放在独立的 figaroh-examples 仓库中。

一个工具箱里的两个问题

Geometric calibration 和 dynamic identification 修正模型的不同部分，但它们具有相近的数学形式和实验逻辑。

Geometric calibration

URDF 为每个 joint 提供 nominal transform，真实装配会产生细小平移和旋转误差。FIGAROH 把 joint (j) 的局部变化写成

[ \Phi_{g,j} = [\delta p_x,\delta p_y,\delta p_z, \delta\phi_x,\delta\phi_y,\delta\phi_z]^T. ]

Forward-kinematics residual 在 nominal model 附近线性化：

[ \Delta P=R_g(q,p)\Phi_g, ]

其中 (R_g) 是 geometric regressor 或 kinematic Jacobian，把 parameter error 映射到 measured pose error。Measurement 可以来自 full pose、position、orientation、camera、motion-capture markers，或者接触平面这样的 geometric constraints。

六参数表示不等于每个 joint 的六个参数都能被独立恢复。论文指出 revolute joint 最多有四个独立 identifiable geometric parameters，prismatic joint 最多两个；collinear axes 还会引入更多依赖。FIGAROH 通过 rank analysis 保留实际可观测的参数组合。

Dynamic identification

Rigid-body inverse dynamics 对惯性参数和多种扩展机械参数是线性的：

[ D=R_d(q,\dot q,\ddot q)\Phi_d. ]

这里 (D) 是 measured torques、motor currents 或 external wrenches 的堆叠，(R_d) 是 dynamic regressor，(\Phi_d) 可以包含：

每个 body 的十个 inertial parameters：mass、first moment 和六个 inertia-tensor terms；
joint viscous/Coulomb friction；
torque-sensor offsets；
actuator rotor inertia 和 drive-chain friction；
current-to-torque 或 sensor gains。

可辨识参数集合由 available measurements 决定。Joint torques 可以识别 inertial 和 joint parameters；加入 motor current 后可以暴露 drive-chain terms；external wrench 对应另一组 identifiable subset。FIGAROH 根据 measurement contract 构造相应 regressor。

Rank Deficiency 与 Base Parameters

Robot regressor 通常 rank deficient。某些物理参数在方程中始终组合出现，无论采集多少样本，数据都无法将它们拆开。FIGAROH 使用带 column pivoting 的数值 QR decomposition，把原 regressor 改写成 full-rank base parameters：

[ Y=R\Phi=R_b\Phi_b, \qquad \operatorname{rank}(R_b)=\dim(\Phi_b). ]

这是整个方法的核心步骤。Least-squares solver 可以给欠定参数化输出一组数值，但这些数值并没有得到 measurement 的唯一支持。Base-parameter reduction 让 estimation problem 保持诚实：solver 只估计当前实验真正能够观测的独立组合。

论文强调 numerical QR 的可解释性：它能显示哪些 original columns 被组合成一个 base parameter，也可以分别得到 static 和 dynamic base sets。例如 static data 中 velocity 和 acceleration terms 消失，只剩 mass 与 center-of-mass combinations。

Parameter Estimation

Base regressor 变成 full rank 后，最简单的估计是 ordinary least squares：

[ \hat\Phi_b = (R_b^TR_b)^{-1}R_b^TY. ]

Weighted least squares 用来处理噪声水平或单位不同的 measurement channels。论文还包括 loaded/unloaded experiments 的 total least squares、nonlinear geometric calibration 的 iterative least squares 与 Levenberg-Marquardt，以及得到 physically plausible inertial parameters 的 constrained optimization。

当前代码保留 regressor/QR workflow，并扩展 solver support。BaseIdentification.solve() 运行标准 identification path；solve_with_custom_solver() 可以通过 figaroh.tools.LinearSolver 调用 least squares、QR/SVD 类方法、Ridge、Lasso、Elastic Net、robust regression 和 constrained variants。

Physical consistency 需要单独关注。Unconstrained fit 即使 torque error 很低，也可能产生 negative mass 或不可能的 inertia tensor。论文使用 quadratic constrained programming 恢复具有物理意义的 individual parameters。当前代码进一步提供可选的 pseudo-inertia SDP/LMI projection，以及从 base parameters 重建 full parameters 的功能，并可加入 CAD-derived mass、center-of-mass 和 symmetry constraints。这些路径需要 SDP solver，默认关闭。

Optimal Experiment Design

FIGAROH 把 data collection 视作 identification 的一部分。Random postures 可能让 regressor 严重 ill-conditioned，random motions 也会浪费机器人时间，并给 humanoid 带来安全风险。因此工具箱为 calibration/static identification 设计 Optimal Exciting Postures (OEP)，为 dynamic identification 设计 Optimal Exciting Motions (OEM)。

论文使用 regressor singular values 构造多种 criterion，包括类似 determinant 的 information volume、condition number 和 minimum singular value。直观上，好实验会让 regressor columns 更容易区分：

[ \text{good excitation} \Longrightarrow \sigma_{\min}(R)\text{ increases and } \kappa(R)=\frac{\sigma_{\max}(R)}{\sigma_{\min}(R)} \text{ decreases}. ]

选择 calibration postures

每个 candidate posture 对应 information matrix (\Sigma_i) 和连续权重 (\omega_i)。FIGAROH 求解

[ \max_{\omega} \Psi!\left(\sum_i\omega_i\Sigma_i\right), \qquad \sum_i\omega_i=1, ]

再根据 weight 对 candidate 排序。随着姿态数量增加，excitation score 会达到峰值或 plateau，从而自动决定需要采集多少姿态。论文示例中的 criterion 在约 40 个 postures 达到峰值。

Posture optimization 同时考虑 joint limits、torque limits、workspace bounds 和 self-collision constraints。论文使用 Pinocchio/hpp-fcl geometry 和基于 IPOPT 的 nonlinear optimization。

生成 exciting motions

OEM generation 用 cubic splines 连接相邻 OEP。Optimizer 调整 spline waypoints，同时满足 endpoint positions、zero endpoint velocity、joint/velocity/torque limits、workspace constraints 和 collision avoidance。Objective 用来改善整段轨迹累计的 dynamic base regressor。

这套设计补上了许多 system-identification scripts 仍需手工完成的闭环：

URDF + measurement setup
        -> identifiable regressor
        -> optimal postures/motions
        -> robot data collection
        -> parameter estimation
        -> cross-validation and model update

论文做了哪些验证？

论文在 human、humanoid、mobile manipulator 和 serial arm 上验证整个 workflow。

对于 43-DoF human model，系统使用 motion capture 和 force-plate data 完成 geometry/dynamics identification。相比 anthropometric-table parameters，identified parameters 把 external-wrench average RMSE 从 force 19.47 N 降到 16.84 N，moment 从 33.22 N·m 降到 16.77 N·m。

对于 TALOS 和 TIAGo torso-arm geometric calibration，TALOS end-effector position RMSE 从 14.1 mm 降到 0.3 mm，TIAGo 从 16.9 mm 降到 1.5 mm。TALOS whole-body calibration 使用 three-point planar contact 和 31 个自动生成 OEP，把 RMSE 从 10.4 mm 降到 3.3 mm。

TIAGo dynamic identification 覆盖 drive-chain parameters 和 differential wrist transmission，identified model 预测 measured motor current 的误差约 1%。工具箱还复现了 Stäubli TX40 的公开 inertial identification 结果，并在 UR10 eye-in-hand calibration 中达到 1.76 mm end-effector position RMSE。

这些案例展示了 toolbox 的适用范围。它们使用不同 sensors 和 metrics，属于异构 case studies，不是一个统一 benchmark。

阅读当前 `figaroh-plus` 代码

当前源码分成三个实用层次。

Workflow layer

figaroh.calibration.BaseCalibration 负责 configuration loading、measurement validation、QR-based parameter selection、nonlinear optimization、iterative outlier removal、metrics、plots、reports 和 verification。Robot-specific behavior 可以通过 subclass cost function 提供。实现使用 SE(3) log-map residuals 处理 pose calibration，并可对 position/orientation channels 施加不同权重。

figaroh.identification.BaseIdentification 负责加载 trajectory data、计算并约简 dynamic regressor、filtering/decimation、求解 base parameters、重建 prediction、评估 RMSE/correlation/condition number，并可选择 physical-consistency 或 full-parameter reconstruction。

BaseOptimalCalibration 和 BaseOptimalTrajectory 提供 posture selection 与 spline trajectory design 的 optimization framework。IPOPT support 依赖 cyipopt；它没有放进简单 pip dependency set，仓库建议这类任务使用 Conda environment。

Tools layer

复用工具包括 RegressorBuilder、QRDecomposer、LinearSolver、collision utilities、IPOPT wrappers、URDF export、visualization、result provenance 和 report generation。当前 library 可以输出 self-contained HTML diagnostics、machine-readable pass/fail verification reports，以及两个运行结果的静态对比页面。

这套 V&V layer 是论文之后很有意义的扩展。Calibration quality 不应只看 fitted RMSE；代码还报告 per-DoF residuals、held-out before/after validation、parameter uncertainty、strongly correlated parameters 和 regressor conditioning。verify() 可以按照 configurable thresholds 输出适合 CI 使用的 verdict。

Backend layer

当前源码实际包含 PinocchioBackend 和 MuJoCoBackend。Pinocchio 仍是默认后端，提供最完整的 URDF/regressor path。MuJoCo 实现 mass、bias、kinematics 和 contact-related computations，但 analytical parameter regressor 会委托给 lazy-loaded Pinocchio model，因为 MuJoCo 没有暴露同等形式的 runtime inertial-parameter regressor。

Architecture document 还讨论 Genesis 和 Isaac Sim backends，并标记为 future work。检查 v0.4.5 source tree 后，没有发现 genesis.py 或 isaacsim.py 实现，因此当前 backend support 应理解为 Pinocchio + optional MuJoCo。

论文与当前仓库的关系

Scientific spine 保持稳定：

基于 URDF 的 rigid multi-body modeling；
geometric/dynamic regressors；
numerical base-parameter extraction；
least-squares 或 constrained estimation；
OEP/OEM experiment design；
statistical validation 和 model export。

plus 仓库主要现代化了 engineering surface：采用 src/ package layout、abstract workflow classes、带 legacy conversion 的 unified YAML parsing、PyPI packaging、更广泛的 regression solvers、可选 physically consistent reconstruction、model-export validation、丰富 reports，以及显式 backend interface。

代码中也能看到若干实际边界：

Package 没有注册 command-line entry point，用户通过 Python classes 和 project scripts 工作。
Robot examples 和部分 datasets 位于独立 examples repository。
BaseCalibration 面向扩展，实际项目可能需要 task-specific subclass 或 cost function。
Advanced OEP/OEM 和 SDP workflow 需要更复杂的 native optimization dependencies。
Documentation 描述的 backend roadmap 比当前源码实现更广。

这是一个增加了 production-oriented checks 的 research toolbox，不是 black-box auto-calibration service。用户仍然需要正确设置 measurement model、frame conventions、active parameters、excitation constraints 和 validation protocol。

优点

FIGAROH 最大的优点是 end-to-end consistency。同一个 robot model 贯穿 regressor construction、observability analysis、experiment design、estimation、collision checking 和最终 model update，降低不同阶段使用错误 joint ordering、frames 或 parameter conventions 的风险。

第二个优点是支持 tree-structured 和 floating-base systems。Workflow 覆盖 industrial manipulators、mobile manipulators、humanoids 和 human biomechanical models。随着 kinematic complexity 增长，手工选择 configurations 更容易不安全或缺少信息，因此 optimal excitation 对这些系统尤其有价值。

当前代码的 validation tooling 也很合理。Parameter fit 应该在 held-out measurements 上验证，同时检查 physical plausibility，并展示 uncertainty 与 conditioning。HTML reports 和 machine-readable verdicts 让这套流程更容易重复执行。

局限与注意事项

Identification quality 仍然受 assumed model 限制。Unmodeled flexibility、backlash、temperature-dependent friction、cable effects、timing errors 和 sensor bias 都可能泄漏进 estimated rigid-body parameters。通过 position 数值微分得到 velocity/acceleration 会放大噪声，因此 filtering 和 excitation design 非常关键。

Geometric calibration 从 local linearization 出发，nonlinear solve 仍依赖合理 initial model 和正确 frame conventions。Base parameters 是 identifiable combinations；恢复每个 URDF inertial term 需要 priors 或 physical constraints，而且仍可能存在歧义。

原论文把 joint geometric-dynamic identification 和 nonlinear flexibility 留作未来工作。当前仓库增强了 physical reconstruction 与 calibration validation，但主要 calibration/identification workflows 仍然分开。Documentation 和 APIs 还在快速演进，v0.4.5 的软件状态是 beta。

总结

FIGAROH 最适合被理解成 experiment-design and model-identification framework，它的价值不局限于 least-squares implementation。核心链路是：

论文证明这条链路可以服务差异很大的 robots 和 humans；当前 figaroh-plus 代码把它整理成更清晰的 library，并增强 verification 与 export support。对于严肃的 calibration 工作，真正重要的正是这种整合：工具箱帮助用户判断什么能被识别、应该进行什么实验，以及最终模型是否真的改善。

[Paper Notes] OmniContact: Chaining Meta-Skills via Contact Flow for Generalizable Humanoid Loco-Manipulation

2026-07-16T00:00:00+00:00

OmniContact proposes a compact interface for long-horizon humanoid loco-manipulation. Instead of asking a planner to generate dense whole-body motion, it represents a future skill using sparse body targets and binary contact schedules. A learned controller executes this contact flow, while a rule-based planner regenerates it online as the object state changes.

My read: the central contribution is the boundary between planning and control. CF-Gen converts object-level goals into contact-aware references through hand-designed phase templates, geometry, IK, and interpolation. CF-Track is a unified reinforcement-learning policy that turns those references into low-level humanoid actions. Contact flow is the shared language that makes carrying, pushing, kicking, recovery, and skill chaining fit into one system.

Paper Info

The paper is “OmniContact: Chaining Meta-Skills via Contact Flow for Generalizable Humanoid Loco-Manipulation” by Runyi Yu, Xiaoyi Lin, Ji Ma, Yinhuai Wang, Koukou Luo, Jiahao Ji, Huayi Wang, Wenjia Wang, Runhan Zhang, Ping Tan, Ting Wu, Ruoli Dai, Qifeng Chen, and Lei Han. The paper, project page, official MuJoCo execution repository, and OmniContact dataset are public.

The Problem: What Should a High-Level Planner Command?

Long-horizon loco-manipulation has two coupled problems. The robot needs a robust controller for individual interactions, and it needs a mechanism for composing those interactions while reacting to failed contacts or displaced objects.

Dense human-object trajectories precisely specify behavior, but they are expensive to synthesize and brittle under online changes. A symbolic command such as carry(box, target) is easy to plan, yet gives the low-level controller little information about where and when contact should happen. Latent skill embeddings are compact, although their semantics are difficult for an external planner to inspect and modify.

OmniContact chooses an intermediate representation: preserve the small set of future body targets required to express motion intent, and explicitly attach the contact timing that distinguishes manipulation from ordinary locomotion.

Contact Flow

At time (t), contact flow is defined as

[ F_t={(b_{t+k},c_{t+k})}_{k\in\mathcal{T}}, ]

with non-uniform future offsets

[ \mathcal{T}={0,1,2,3,4,8,12,16,24,32,50}. ]

Here (b_{t+k}) contains sparse future body targets, primarily torso, wrists, and ankles. The contact vector

[ c_{t+k}\in{0,1}^{4} ]

specifies binary contact states for four robot end effectors. Non-uniform sampling allocates more targets to the immediate future and fewer to longer-term intent.

This representation carries two kinds of information:

Where the body should move: sparse future torso and end-effector targets.
When interaction should occur: explicit contact/no-contact state for each end effector.

Contact flow does not contain contact force, pressure, or a dense physics trajectory. It describes intended contact mode and sparse kinematic goals. The low-level policy retains freedom to choose the remaining whole-body motion needed for balance and compliance.

The ablation strongly supports this design. Tracking torso and end effectors without contact reaches only 11.5% success on the CF-Gen Carry Box evaluation. Adding binary contact raises success to 98.7%. Adding dense object, joint-DoF, or full-body targets then hurts generalization because the controller becomes overconstrained. The successful interface is the compact set torso + end effectors + contact.

CF-Track: The Learned Low-Level Executor

CF-Track is one unified policy for locomotion, carrying, pushing, sliding, relocating, kicking, and their transitions. Its input combines the target contact flow with a five-step observation history:

[ x_t=[F_t,H_t], \qquad H_t=[o_t,\ldots,o_{t-4}]. ]

Each observation includes robot proprioception, base orientation, end-effector positions, the previous action, and the object’s relative 6D pose and bounding box. The policy outputs low-level motor actions at 50 Hz. The actor and critic use Transformer networks.

CF-Track is trained with reinforcement learning using three reward families:

[ r_t=lambda_{\text{track}}r_t^{\text{track}} +\lambda_{\text{amp}}r_t^{\text{amp}} +\lambda_{\text{reg}}r_t^{\text{reg}}. ]

The tracking term follows sparse body, object, and contact-flow targets. The AMP adversarial motion prior encourages human-like whole-body motion. The regularization term penalizes abrupt action changes. The selected balance places most weight on tracking while retaining enough motion prior to smooth the artifacts of rule-generated references.

Pure tracking achieves relatively accurate motion but reduces disturbance stability. Giving the motion prior too much weight produces stable-looking behavior with almost no task success. This is a genuine control trade-off: the policy must obey contact intent strongly enough to finish the task while preserving enough behavioral flexibility to absorb imperfect plans and perturbations.

CF-Gen: Rule-Based Contact-Flow Synthesis

CF-Gen is described as a lightweight, heuristic planner. It is not a learned trajectory generator. Each meta-skill has a hand-designed sequence of phases. Carrying, for example, is decomposed into approach, move to a hand-contact pose, lift, walk while maintaining contact, and release.

For each phase, CF-Gen performs four operations.

1. Select object-centric contact anchors

The planner uses object pose and dimensions to choose suitable contact faces and approach directions. It adapts wrist, torso, and ankle targets to object geometry, enabling the same carry or push template to handle different object sizes and initial poses.

2. Generate keyframes with selective IK

Locomotion phases mainly specify ankle goals and a nominal posture. Contact phases specify ankle and wrist targets, then solve constrained IK over pelvis height, pelvis pitch, and robot joints. The solver is limited to 20 iterations for real-time synthesis.

3. Interpolate a dense kinematic reference

Positions and joint values use linear interpolation, while pelvis orientation uses SLERP. This creates a continuous reference between phase keyframes.

4. Convert the reference into sparse contact flow

CF-Gen samples the interpolated trajectory at the non-uniform future offsets and transforms the targets into the current torso-yaw frame. CF-Track receives only the resulting sparse body targets and binary contact schedule.

This sequence is important: CF-Gen temporarily creates a dense reference internally, but the controller does not directly track the full joint trajectory. Contact flow discards unnecessary constraints before execution.

Closed-Loop Chaining and Recovery

CF-Gen maintains a finite stage state containing the active object, selected meta-skill, phase, and target. When a phase finishes, it generates the next contact-flow segment. Multi-stage tasks are built by switching objects and meta-skills at stage boundaries.

During execution, CF-Gen monitors the observed and predicted object state at 50 Hz:

[ \delta_t=d(x^{\text{obj}}{t,\text{obs}},x^{\text{obj}}{t,\text{pred}}). ]

If (\delta_t) crosses a threshold, the planner aborts the current reference and replans from the observed state. A dropped or displaced box can therefore trigger a new approach and contact sequence.

Online replanning raises Push Suitcase from 82.5% to 94.5%, Stack Boxes from 56.6% to 80.5%, and Push-Stack Boxes from 76.5% to 84.5%. The paper also notes an important nuance: most of this gain comes from refreshing future segments with updated object states. Full emergency recovery is triggered relatively rarely.

Under deliberately injected failures, recovery reaches 92.5% after dropping a carried box, 97.5% after offsetting its pose, and 89.5% after offsetting a pushed suitcase, with roughly 1.5-1.8 replans per trial.

OmniContact Dataset

The authors collect a synchronized MoCap human-object interaction dataset tailored to long-horizon loco-manipulation. It contains:

1,274 valid sequences and 22.29 hours of motion;
7.22 million object frames captured at 90 Hz;
synchronized BVH human motion and rigid-object 6D trajectories;
carry, push, relocate, slide, and kick primitives;
primitive-level contact modes and task-level language descriptions.

The average sequence lasts 62.98 seconds and moves the object 19.76 meters, emphasizing long-range transport. The dataset has fewer objects and sequences than OMOMO, but longer clips, higher temporal resolution, and explicit loco-manipulation structure. Scaling CF-Track training data from 2.2 to 22.3 hours improves both success and object accuracy.

The dataset does not provide measured contact forces. Contact supervision is constructed from synchronized body/object trajectories, binary contact modes, and dynamic/static phase annotations.

Main Results

In the randomized simulation benchmark, OmniContact reaches:

98.7% on Carry Box;
82.5% on Push Suitcase;
56.5% final-stage success on stacking three boxes;
76.5% final-stage success on Push-Stack Boxes.

The baselines perform much worse on interaction and chaining. PhysHSI reaches 87% on Carry Box but fails at the final stacking stage. LessMimic reaches 34% on Carry Box and 12.5% on Push Suitcase, with 0% final success on the chained tasks. OmniContact’s main empirical advantage is therefore composition under object-pose variation, not basic locomotion; several baselines already perform well on locomotion-only evaluation.

The additional tasks include Relocate Ball, Slide Box, Kick Ball, Relocate-then-Kick, and Push-then-Relocate. Long-duration experiments show perfect 40-minute survival when repeatedly resampling goals for one object under the easiest protocol. Survival drops to 29.5% at 40 minutes when both initial and final object goals are repeatedly resampled, and becomes harder with multiple interacting objects. These extended results give a more calibrated picture than the headline 40-minute demonstration.

Real-World Deployment

The learned policy is deployed on a Unitree G1. The controller runs at 50 Hz, and a Noitom motion-capture system tracks the robot pelvis and manipulated object at 100 Hz to provide global poses. Before hardware deployment, the policy is checked through Isaac Lab-to-MuJoCo sim-to-sim transfer.

The real-world demonstrations show that the controller can execute the behaviors on hardware, but the main quantitative tables are simulation results. The deployment also relies on external MoCap for robot/object pose; this is not yet a vision-only autonomous loco-manipulation system.

The public repository currently focuses on MuJoCo execution. It includes an ONNX policy, CF-Gen references, scripted skill and skill-chaining runners, and an interactive joystick-based state-switching path that mirrors sim-to-real deployment.

VLM Integration

A VLM can receive a rendered scene and language instruction, then output an ordered list of object-level goals and available meta-skills. For example, it can decompose “arrange the boxes into a heart” into repeated carry/place subgoals. CF-Gen grounds each subgoal into contact anchors and contact flow; CF-Track executes it.

The VLM never predicts joint trajectories, contact timing, or low-level actions. Its role is semantic decomposition. The paper presents this integration qualitatively, including heart-shaped layouts and multi-object arrangements. These examples demonstrate interface compatibility; an end-to-end trained vision-language-control model remains outside the paper’s scope.

What Is Actually New?

OmniContact combines several familiar components: MoCap imitation, AMP-style reinforcement learning, hand-designed state machines, object-centric geometry, IK, interpolation, and online replanning. Its main design contribution is how these components communicate.

Contact flow is compact enough for a planner to synthesize and edit online, explicit enough to represent contact timing, and sparse enough to leave the RL controller freedom over whole-body execution. The system separates:

What interaction should happen: meta-skill, object goal, contact anchors, and binary contact schedule.
How the humanoid physically realizes it: balance, gait, compliant adaptation, and low-level motor actions.

The strongest ablation is the jump from 11.5% to 98.7% after adding explicit contact state to torso and end-effector targets. That result supports contact flow as a useful planning-control interface more directly than the headline VLM examples do.

Limitations

CF-Gen depends on manually designed phase templates, contact-anchor rules, thresholds, and task-specific geometry logic. The authors report that it struggles in highly dynamic settings and propose learning contact-anchor generation in future work. The G1’s underactuated grippers restrict fine manipulation, and contact flow has not yet been extended to dexterous hands.

The approach assumes access to accurate object pose and dimensions. Real deployment uses external MoCap, while VLM demonstrations operate on rendered scenes and qualitative task decomposition. Broader visual perception, uncertain geometry, force-aware contact, and unstructured real-world recovery remain open. The baseline comparison also requires task-specific adaptations because several methods do not natively support autonomous approach, release, or chaining.

Takeaway

OmniContact is best understood as a hierarchical humanoid control interface. It does not ask one model to invent long-horizon motion from language. A semantic planner selects object-level goals; CF-Gen turns them into sparse future body targets and contact schedules; CF-Track converts those targets into stable whole-body control; observed object-state errors close the loop through replanning.

The useful idea is simple: for loco-manipulation, sparse pose targets are too ambiguous, while dense whole-body trajectories are too restrictive. Adding an explicit contact timeline creates a middle representation that can be planned, composed, tracked, and repaired online.

OmniContact 为长时程 humanoid loco-manipulation 提出了一种紧凑的中间接口。高层规划器无需生成 dense whole-body motion，只要输出稀疏身体目标和二值接触时序。学习到的控制器负责执行这段 contact flow；当物体状态变化时，规则规划器在线重新生成 contact flow。

我的理解是：论文最核心的贡献在规划与控制的边界设计。CF-Gen 通过手工 phase templates、物体几何、IK 和 interpolation，把 object-level goal 转换成 contact-aware reference。CF-Track 是统一的强化学习策略，把 reference 转成 humanoid low-level actions。Contact flow 是两者之间的共同语言，让 carry、push、kick、recovery 和 skill chaining 可以进入同一套系统。

论文信息

论文题目是 “OmniContact: Chaining Meta-Skills via Contact Flow for Generalizable Humanoid Loco-Manipulation”，作者为 Runyi Yu, Xiaoyi Lin, Ji Ma, Yinhuai Wang, Koukou Luo, Jiahao Ji, Huayi Wang, Wenjia Wang, Runhan Zhang, Ping Tan, Ting Wu, Ruoli Dai, Qifeng Chen, and Lei Han。目前论文、项目主页、官方 MuJoCo 执行仓库和 OmniContact dataset均已公开。

问题：高层规划器应该输出什么？

长时程 loco-manipulation 包含两个耦合问题：机器人要稳定执行单个 interaction，还要能够组合多个 interaction，并在接触失败或物体被移动后继续完成任务。

Dense human-object trajectory 能精确描述动作，但在线合成成本高，对环境变化也比较脆弱。carry(box, target) 这样的 symbolic command 容易规划，却没有告诉底层控制器应该在哪里、何时建立接触。Latent skill embedding 很紧凑，其语义难以被外部 planner 检查和修改。

OmniContact 选择中间层表示：保留表达运动意图所需的少量未来 body targets，再显式加入 manipulation 区别于普通 locomotion 的 contact timing。

Contact Flow

在时间 (t)，contact flow 定义为

[ F_t={(b_{t+k},c_{t+k})}_{k\in\mathcal{T}}, ]

其中非均匀 future offsets 为

[ \mathcal{T}={0,1,2,3,4,8,12,16,24,32,50}. ]

(b_{t+k}) 表示稀疏 future body targets，主要包含 torso、wrists 和 ankles。Contact vector

[ c_{t+k}\in{0,1}^{4} ]

表示机器人四个 end effectors 的二值 contact states。Non-uniform sampling 在短期未来放置更多 targets，同时用少量远期目标表达长期意图。

这个表示携带两类信息：

身体应该去哪里：未来 torso 和 end-effector 的稀疏目标。
什么时候发生 interaction：每个 end effector 的 contact/no-contact 状态。

Contact flow 不包含 contact force、pressure 或 dense physics trajectory。它描述预期 contact mode 和稀疏 kinematic goals，剩余 whole-body motion 由底层策略根据平衡和 compliance 自行决定。

消融实验很有说服力。只跟踪 torso 和 end effectors 时，CF-Gen Carry Box evaluation 的成功率只有 11.5%；加入 binary contact 后上升到 98.7%。继续增加 dense object、joint-DoF 或 full-body targets 会过度约束控制器，反而削弱 generalization。最终有效的接口是 torso + end effectors + contact。

CF-Track：学习到的底层执行器

CF-Track 是一个统一策略，覆盖 locomotion、carry、push、slide、relocate、kick 及其切换。输入由目标 contact flow 和五步 observation history 组成：

[ x_t=[F_t,H_t], \qquad H_t=[o_t,\ldots,o_{t-4}]. ]

每个 observation 包含机器人 proprioception、base orientation、end-effector positions、previous action，以及物体的 relative 6D pose 和 bounding box。策略以 50 Hz 输出 low-level motor actions，actor 和 critic 使用 Transformer networks。

CF-Track 使用 reinforcement learning 训练，包含三类奖励：

[ r_t=lambda_{\text{track}}r_t^{\text{track}} +\lambda_{\text{amp}}r_t^{\text{amp}} +\lambda_{\text{reg}}r_t^{\text{reg}}. ]

Tracking term 跟踪稀疏身体、物体和 contact-flow targets；AMP adversarial motion prior 维持接近人类的 whole-body motion；regularization term 惩罚突然变化的 action。最终配置让 tracking 占主要权重，同时保留足够的 motion prior，用来平滑规则生成 reference 中的动作瑕疵。

Pure tracking 可以得到比较准确的 motion，却会降低扰动下的稳定性。Motion prior 权重过高则会产生稳定、自然的动作，但任务几乎无法成功。这里存在真实的控制权衡：策略既要强力服从 contact intent 来完成任务，也要保留足够自由度来吸收 imperfect plan 和 perturbation。

CF-Gen：规则式 Contact-Flow Synthesis

论文把 CF-Gen 定义为 lightweight heuristic planner，它并不是学习到的 trajectory generator。每个 meta-skill 都有手工设计的 phase sequence。例如 carry 会被分解为 approach、移动到 hand-contact pose、lift、保持接触并行走、release。

每个 phase 中，CF-Gen 依次完成四项操作。

1. 选择 object-centric contact anchors

Planner 根据 object pose 和 dimensions 选择适合的接触面及 approach direction，并调整 wrist、torso 和 ankle targets，使同一 carry 或 push template 能适应不同物体尺寸和初始位置。

2. 使用 selective IK 生成 keyframes

Locomotion phases 主要指定 ankle goals 和 nominal posture。Contact phases 会指定 ankle 和 wrist targets，再对 pelvis height、pelvis pitch 和 robot joints 求 constrained IK。为满足实时性，solver 最多迭代 20 次。

3. 插值得到 dense kinematic reference

Position 和 joint values 使用 linear interpolation，pelvis orientation 使用 SLERP，在 phase keyframes 之间形成连续 reference。

4. 把 reference 转换成 sparse contact flow

CF-Gen 按非均匀 future offsets 对 interpolated trajectory 采样，并把 targets 转换到当前 torso-yaw frame。CF-Track 最终只接收稀疏 body targets 和 binary contact schedule。

这里有一个重要细节：CF-Gen 内部会暂时产生 dense reference，但 controller 不直接跟踪完整 joint trajectory。Contact flow 会在执行前丢弃多余约束。

Closed-Loop Chaining 与 Recovery

CF-Gen 维护一个 finite stage state，记录 active object、selected meta-skill、phase 和 target。一个 phase 结束后，它生成下一段 contact flow。Multi-stage task 通过在 stage boundary 切换 object 和 meta-skill 完成。

执行时，CF-Gen 以 50 Hz 比较 observed 和 predicted object state：

[ \delta_t=d(x^{\text{obj}}{t,\text{obs}},x^{\text{obj}}{t,\text{pred}}). ]

如果 (\delta_t) 超过阈值，planner 会中止当前 reference，并从 observed state 重新规划。箱子掉落或被移动后，系统可以重新生成 approach 和 contact sequence。

Online replanning 把 Push Suitcase 从 82.5% 提高到 94.5%，Stack Boxes 从 56.6% 提高到 80.5%，Push-Stack Boxes 从 76.5% 提高到 84.5%。论文同时指出：大部分提升来自根据新 object state 刷新后续 segments，完整 emergency recovery 实际触发次数较少。

在主动注入失败的实验中，carry box 掉落后的恢复成功率为 92.5%，box pose offset 后为 97.5%，push suitcase pose offset 后为 89.5%，平均每次需要约 1.5-1.8 次 replans。

OmniContact Dataset

作者采集了一个面向长时程 loco-manipulation 的同步 MoCap human-object interaction dataset，包含：

1,274 条 valid sequences，总计 22.29 小时；
以 90 Hz 捕捉的 7.22 million object frames；
同步 BVH human motion 和 rigid-object 6D trajectories；
carry、push、relocate、slide 和 kick primitives；
primitive-level contact modes 和 task-level language descriptions。

平均每条 sequence 长 62.98 秒，物体平均移动 19.76 米，数据重点是 long-range transport。相比 OMOMO，它的 objects 和 sequences 更少，但 clips 更长、时间分辨率更高，而且包含显式 loco-manipulation structure。把 CF-Track training data 从 2.2 小时扩展到 22.3 小时后，成功率和 object accuracy 都继续提高。

数据集没有 measured contact forces。Contact supervision 来自同步 body/object trajectories、binary contact modes 和 dynamic/static phase annotations。

主要结果

在 randomized simulation benchmark 中，OmniContact 达到：

Carry Box：98.7%；
Push Suitcase：82.5%；
三个箱子 stacking 的 final-stage success：56.5%；
Push-Stack Boxes final-stage success：76.5%。

Baselines 在 interaction 和 chaining 上明显更弱。PhysHSI 的 Carry Box 为 87%，但 stacking 最终阶段失败；LessMimic 在 Carry Box 和 Push Suitcase 上分别为 34% 和 12.5%，chained tasks 最终成功率为 0%。OmniContact 的主要优势是 object-pose variation 下的 composition；它并不以 basic locomotion 为主要贡献，多种 baseline 在 locomotion-only evaluation 中已经表现很好。

Additional tasks 包含 Relocate Ball、Slide Box、Kick Ball、Relocate-then-Kick 和 Push-then-Relocate。长时间实验中，最简单 protocol 对同一物体不断重采样目标，40 分钟 survival 为 100%。如果反复重采样 object initial/final goals，40 分钟 survival 会降到 29.5%；多个物体互相干扰时更困难。这组 extended results 比单独的“运行约 40 分钟”更完整。

真实机器人部署

学习到的策略部署在 Unitree G1 上。Controller 以 50 Hz 运行，Noitom motion-capture system 以 100 Hz 跟踪 robot pelvis 和 manipulated object，提供 global poses。硬件部署前还进行了 Isaac Lab 到 MuJoCo 的 sim-to-sim 检查。

真实演示说明 controller 能在硬件上执行这些动作，但主要 quantitative tables 来自 simulation。部署还依赖 external MoCap 获取 robot/object pose，因此目前还不是 vision-only autonomous loco-manipulation system。

公开仓库当前主要提供 MuJoCo execution，包括 ONNX policy、CF-Gen references、scripted skill/skill-chaining runners，以及模拟 sim-to-real switching 的 joystick interactive execution path。

VLM 集成

VLM 接收 rendered scene 和 language instruction，输出有序 object-level goals 与可用 meta-skills。例如，它可以把“把箱子排列成心形”分解成多次 carry/place subgoals。CF-Gen 把每个 subgoal grounding 成 contact anchors 和 contact flow，CF-Track 再负责执行。

VLM 不预测 joint trajectories、contact timing 或 low-level actions，它只负责 semantic decomposition。论文以 heart-shaped layout 和多物体排列进行定性展示。这些例子验证了接口兼容性；端到端训练的 vision-language-control model 不在本文范围内。

真正的新设计是什么？

OmniContact 组合了多种已有组件：MoCap imitation、AMP-style reinforcement learning、hand-designed state machines、object-centric geometry、IK、interpolation 和 online replanning。它最主要的设计贡献是这些组件之间如何通信。

Contact flow 足够紧凑，可以由 planner 在线生成和修改；接触语义显式可读；稀疏 targets 又给 RL controller 留出了 whole-body execution 自由度。整个系统拆分了：

应该发生什么 interaction：meta-skill、object goal、contact anchors 和 binary contact schedule。
Humanoid 如何在物理上实现它：balance、gait、compliant adaptation 和 low-level motor actions。

最有说服力的消融是加入 explicit contact state 后，成功率从 11.5% 跳到 98.7%。相比吸引眼球的 VLM examples，这个结果更直接地支持 contact flow 作为 planning-control interface 的价值。

局限

CF-Gen 依赖人工设计的 phase templates、contact-anchor rules、thresholds 和 task-specific geometry logic。作者也承认它在 highly dynamic settings 中表现受限，未来希望学习 contact-anchor generation。G1 的 underactuated grippers 限制 fine manipulation，contact flow 也尚未扩展到 dexterous hands。

方法假设能够获得准确 object pose 和 dimensions。真实部署依赖 external MoCap；VLM demonstrations 使用 rendered scenes，主要验证 qualitative task decomposition。更广泛的 visual perception、uncertain geometry、force-aware contact 和 unstructured real-world recovery 仍未解决。Baseline comparison 还需要针对各方法做任务适配，因为部分方法原本不支持 autonomous approach、release 或 chaining。

总结

OmniContact 最适合被理解成一种 hierarchical humanoid control interface。系统没有让一个模型直接从语言生成完整 long-horizon motion。Semantic planner 选择 object-level goals，CF-Gen 把它们转换成稀疏 future body targets 和 contact schedules，CF-Track 将这些目标变成稳定 whole-body control，object-state error 再通过 replanning 闭合反馈回路。

核心想法很清楚：对于 loco-manipulation，只有 sparse pose targets 会产生歧义，dense whole-body trajectories 又会限制控制器。显式加入 contact timeline 后，就得到一个可以规划、组合、跟踪并在线修复的中间表示。

[Paper Notes] RynnWorld-Teleop: An Action-Conditioned World Model for Digital Teleoperation

2026-07-16T00:00:00+00:00

RynnWorld-Teleop replaces the physical robot in a teleoperation session with an action-conditioned video world model. An operator’s hand-pose stream simultaneously drives a robot-view video generator and a human-to-robot retargeting pipeline, producing synchronized RGB observations and 54-dimensional robot actions for imitation learning.

My main takeaway concerns the operator’s “feel.” The system provides a responsive visual control loop: a distilled causal generator produces robot-centric video at 40 FPS with about 25 ms model latency. It does not provide haptic or force feedback. The Manus gloves capture pose; they do not return contact force, weight, friction, or slip to the hands. RynnWorld-Teleop is therefore best understood as a scalable data engine with visual telepresence, not a bilateral force-reflecting teleoperation system.

Paper Info

“RynnWorld-Teleop: An Action-Conditioned World Model for Digital Teleoperation” is by Haoyu Zhao, Xingyue Zhao, Hangyu Li, Biao Gong, Kehan Li, Siteng Huang, Xin Li, Deli Zhao, and Zhongyu Li, from DAMO Academy, Alibaba Group, the Hong Kong Embodied AI Lab, CUHK, Hupan Lab, and Ant Group. The notes below refer to arXiv:2607.06558v2, dated July 12, 2026.

Digital Teleoperation as Data Generation

Physical teleoperation couples every demonstration to robot availability, a prepared workspace, real objects, and manual resets. RynnWorld-Teleop moves the interaction into a learned visual world. Given a reference image (I_{\mathrm{ref}}) and a hand-gesture sequence

[ P={p_1,\ldots,p_T}, ]

the model generates a robot-centric egocentric video

[ V={v_1,\ldots,v_T}. ]

The same gesture stream is retargeted into robot joint positions. The output trajectory is therefore

[ \tau={(v_t,a_t)}_{t=1}^{T}, \qquad a_t\in\mathbb{R}^{54}, ]

where each action concatenates two 7-DoF arms and two 20-DoF dexterous hands. The generated observation and robot action share the same human control signal, which gives the system its action-grounded property.

The paper defines three requirements for useful digital teleoperation:

robot-centric: the generated view depicts the target robot embodiment;
action-grounded: each frame remains tied to recoverable joint-level actions;
real-time: the operator stays in the visual loop and can compose longer skills.

What the Operator Actually Feels

The operator wears five HTC Vive trackers—chest, wrists, and upper arms—and Manus data gloves. In the physical data-collection setup, the trackers provide wrist-to-chest transforms at 100–120 Hz. The gloves are converted into a 21-point MediaPipe hand skeleton and then retargeted to each 20-DoF WUJI hand, with an exponential moving average for smoothing.

These devices are motion sensors. The paper describes no actuators in the gloves, fingertip force display, vibrotactile feedback, exoskeleton resistance, or other haptic channel. The feedback available during digital teleoperation is the generated first-person robot video.

Feedback or control channel	Present in the paper?	Role
Hand and arm pose tracking	Yes	Captures operator intent
Real-time robot-view video	Yes	Visual closed-loop feedback
Motion smoothing and IK constraints	Yes	Stabilizes retargeted action labels
Contact-force reflection	No	No physical reaction force reaches the operator
Weight or inertia sensation	No	Inferred only from generated visual motion
Friction and slip sensation	No	No tactile slip feedback
Operator user study	No	No measured comfort, presence, or subjective hand-feel score

“Fluid and responsive” in the paper is supported by throughput and latency measurements. It should not be interpreted as measured haptic realism.

1. Depth-Aware Skeletons Preserve Fine Motion

A standard 2D skeleton loses the hand’s distance from the camera, which is crucial when reaching behind, around, or into contact with an object. RynnWorld-Teleop renders a 21-joint skeletal video whose color and joint/bone diameter vary with camera-space depth. The result remains a 2D image-like signal while carrying explicit 3D cues.

A pretrained VAE maps this skeleton video to a control latent

[ c\in\mathbb{R}^{C\times T\times H\times W}, ]

spatially and temporally aligned with the target video latent. This representation gives the video model fine-grained articulation and depth information without requiring full meshes or robot simulation assets.

2. Pose-Conditioned Video DiT

The world model builds on Wan2.2-TI2V-5B. It introduces a control patch-embedding branch alongside the original video branch. Because skeleton and video latents have different statistics, the control latent is normalized into the video latent distribution before fusion:

[ \tilde c = \frac{c-\mu_c}{\sigma_c}\sigma_z+\mu_z, ]

[ x = \operatorname{PatchEmbed}_z(z_t) +\alpha\operatorname{PatchEmbed}_c(\tilde c). ]

The control branch is zero-initialized and (\alpha) starts small, allowing pose control to enter gradually while preserving the pretrained video prior. Direct concatenation disrupts that distribution: the reported FVD rises from 585 to 1191 in the LoRA ablation.

This conditioning mechanism is the link between an operator gesture and the visual consequence. It improves controllability, although it cannot guarantee that every generated contact obeys true rigid-body dynamics.

3. Human Priors First, Robot Embodiment Second

Training proceeds across two domains. Stage 1 uses large-scale egocentric human video to learn hand-object interaction priors:

VITRA: 30.7 million frames;
EgoDex: 74.0 million frames.

Stage 2 fine-tunes on 1,800 paired human-robot teleoperation episodes, totaling 0.43 million robot frames. The four tasks are dual picking, block pushing, bimanual lifting, and lid placement. Human gestures are paired with their robot executions through IK, teaching the generator how a human intention should appear through a specific robot embodiment.

The human-video stage supplies object permanence, contact patterns, and visual “common sense.” Removing it causes a severe collapse: FVD increases to 2598, with ghosting, blurred structures, and disappearing end effectors or objects. These learned priors make contact look plausible to the operator and make generated trajectories useful for policy training. They remain learned statistical dynamics, without explicit online force or contact solving.

4. Streaming Distillation Creates the Visual Loop

A bidirectional diffusion teacher is too slow for interactive control. The authors convert it into a causal student in two steps.

First, causal flow-matching warm-up teaches frame-causal generation. At time (t), attention can use only frames (1,\ldots,t). A fixed-size KV cache supports streaming, while the reference-image embedding persists as a sink token to preserve the original scene.

Second, Distribution Matching Distillation (DMD) transfers the teacher’s output distribution into a four-step generator. Training backpropagates through the persistent KV cache across successive chunks, reducing boundary artifacts and improving long-rollout consistency.

On one H100 at (480\times832), the causal model reports:

40.0 FPS;
approximately 25 ms per generated frame;
about 5% of latency in skeletal encoding;
about 72% in four-step causal DiT denoising;
about 23% in VAE decoding.

This rate matches or exceeds a typical 30 Hz robot camera stream and is the main basis for the paper’s claim of a responsive operator experience. The paper reports model inference latency, not complete motion-to-photon latency including trackers, communication, display, and buffering.

There is also a quality-speed trade-off. On EgoDex-Test, the non-causal SFT model obtains FVD 550 at 2.8 FPS; the causal model reaches 40 FPS with FVD 1226. Real-time interaction comes with reduced visual fidelity.

5. Retargeting Produces Smooth Robot Actions

The operator’s wrist pose is expressed relative to the chest, scaled into the robot workspace, and transformed into the robot base frame:

[ T_{\mathrm{target}} = T_{\mathrm{base}} \cdot \operatorname{Scale}(T_{\mathrm{chest}}^{-1}T_{\mathrm{wrist}}) \cdot T_{\mathrm{ee}}. ]

An iterative damped least-squares IK solver maps this pose to robot joints. Adaptive damping handles near-singular configurations, a null-space shoulder prior favors natural arm postures, and hard clipping enforces joint limits. During physical demonstration collection, Ruckig further limits velocity, acceleration, and jerk before sending arm commands at 200 Hz. The robot’s low-level interface runs at 500 Hz; policy commands are reported at 50 Hz with 18–30 ms delay.

These mechanisms improve control smoothness and the quality of action labels. They do not add physical feedback to the digital operator.

6. Long-Horizon Generation and Re-Anchoring

Pure autoregressive generation accumulates visual drift. The data-generation pipeline divides long videos into 81-frame chunks. The first chunk starts from the true initial frame; later chunks can be re-anchored with an actual robot-camera frame from the corresponding source demonstration.

This detail narrows the strongest “unbounded digital generation” interpretation. Re-anchoring preserves object pose and lighting when a physical source trajectory exists. A fully synthetic, live session starting from only one image cannot access future real frames, so it must tolerate model drift or use another state-correction mechanism.

Experiments: Is the Generated Data Useful?

The downstream evaluation trains Diffusion Policy, (\pi_0), and (\pi_{0.5}) on real and generated trajectories, then runs 35 real-robot trials per task. Adding 300 generated episodes to 300 real episodes generally improves success. The largest reported gain occurs in lid placement:

Policy	Training data	Lid placement success
(\pi_{0.5})	300 real	42.86%
(\pi_{0.5})	300 real + 300 generated	62.86%
(\pi_0)	300 real	34.29%
(\pi_0)	300 real + 300 generated	54.29%

A (\pi_0) policy trained on 300 generated episodes and zero real episodes reaches 82.86% on block pushing and 77.14% on bimanual lifting. Lid placement remains harder at 28.57%, revealing the limit of generated data for precise contact and alignment.

These results validate RynnWorld-Teleop primarily as a robot-data generator. They do not measure whether a human operator can reliably judge contact forces or complete tasks through the generated visual loop.

Strengths and Limitations

The paper’s strongest idea is the shared action source. A hand-pose stream produces both visual observations and embodiment-specific actions, making generated trajectories immediately compatible with imitation learning. Depth-aware skeletal conditioning preserves fine control, and causal distillation turns a large video model into a 40 Hz interactive generator.

Several boundaries matter in practice:

The system offers visual feedback without haptic or force feedback.
Model latency is measured; full operator-loop latency and subjective usability are not evaluated.
Generated contact follows learned visual priors and can fail on liquids and highly deformable objects.
Each robot platform currently requires separate embodiment adaptation.
Causal real-time generation loses image quality relative to the slower teacher.
Long-horizon re-anchoring uses real source frames when available, limiting a fully synthetic interpretation.

Takeaways

RynnWorld-Teleop’s operator loop can be summarized as

[ \boxed{ \text{tracked gesture} \rightarrow \begin{cases} \text{40 Hz generated robot-view video}
\text{retargeted 54D robot action} \end{cases} }. ]

The operator receives fast visual consequences for hand motion, which supports a useful sense of control. Physical hand feel remains absent: contact, load, friction, and slip are seen through generated pixels and never reflected into the operator’s hands. For scalable imitation-learning data, that trade is compelling. For contact-rich telepresence or delicate force-controlled manipulation, an additional bilateral haptic channel and stronger state grounding would still be required.

RynnWorld-Teleop 在遥操过程中用动作条件视频世界模型替代真实机器人。遥操员的手部姿态流同时驱动机器人视角视频生成器和人到机器人的 retargeting 流程，最终产生时间同步的 RGB 观测与 54 维机器人动作，用于模仿学习。

我最关注的是遥操员的“手感”。系统提供的是低延迟视觉控制闭环：蒸馏后的因果生成器以 40 FPS 输出机器人第一视角视频，模型延迟约 25 ms。系统没有力反馈或触觉反馈。Manus 手套负责采集姿态，不会把接触力、重量、摩擦或滑移反馈到人的手上。因此，RynnWorld-Teleop 更适合被理解为带视觉临场感的可扩展数据引擎，而非双向力反射遥操作系统。

论文信息

论文 “RynnWorld-Teleop: An Action-Conditioned World Model for Digital Teleoperation” 的作者是 Haoyu Zhao、Xingyue Zhao、Hangyu Li、Biao Gong、Kehan Li、Siteng Huang、Xin Li、Deli Zhao 和 Zhongyu Li，来自 DAMO Academy、Alibaba Group、Hong Kong Embodied AI Lab、CUHK、Hupan Lab 和 Ant Group。本文依据 arXiv:2607.06558v2，日期为 2026 年 7 月 12 日。

把数字遥操定义成数据生成

物理遥操让每条示范都依赖机器人可用时间、整理好的工作空间、真实物体与人工复位。RynnWorld-Teleop 把交互移入学习得到的视觉世界。给定参考图像 (I_{\mathrm{ref}}) 与手势序列

[ P={p_1,\ldots,p_T}, ]

模型生成机器人第一视角视频

[ V={v_1,\ldots,v_T}. ]

同一手势流还会被 retarget 成机器人关节位置，最终轨迹为

[ \tau={(v_t,a_t)}_{t=1}^{T}, \qquad a_t\in\mathbb{R}^{54}, ]

其中每个动作拼接了两条 7-DoF 手臂和两只 20-DoF 灵巧手。生成观测和机器人动作共享同一个人体控制信号，这构成了系统的 action-grounded 属性。

论文为可用的数字遥操提出三个条件：

robot-centric：生成画面呈现目标机器人 embodiment；
action-grounded：每一帧都与可恢复的关节级动作绑定；
real-time：操作者能够留在视觉闭环中，并连续组合较长技能。

遥操员实际感受到什么

遥操员佩戴五个 HTC Vive tracker，位置分别是胸部、双手腕和双上臂，同时佩戴 Manus 数据手套。在物理数据采集系统中，tracker 以 100–120 Hz 提供手腕相对胸部的位姿。手套信号被转换成 21 点 MediaPipe 手骨架，再 retarget 到每只 20-DoF WUJI Hand，并通过指数移动平均进行平滑。

这些设备属于运动传感器。论文没有描述手套执行器、指尖力显示、振动触觉、外骨骼阻力或其他触觉通道。数字遥操过程中，操作者获得的反馈是生成的机器人第一视角视频。

反馈或控制通道	论文是否提供	作用
手部和手臂姿态跟踪	有	捕获操作者意图
实时机器人视角视频	有	提供视觉闭环反馈
运动平滑与 IK 约束	有	稳定 retarget 后的动作标签
接触力反射	无	操作者接收不到物理反作用力
重量或惯性感受	无	只能从生成画面推测
摩擦和滑移感受	无	没有触觉滑移反馈
操作者用户研究	无	没有舒适度、临场感或主观手感评分

论文中的 “fluid and responsive” 由吞吐率和延迟测量支持，不能等同于经过测量的触觉真实性。

1. Depth-Aware Skeleton 保留精细动作

普通二维骨架会丢失手相对相机的深度，这个信息对绕过物体、伸向物体后方和建立接触都很重要。RynnWorld-Teleop 渲染 21 关节点骨架视频，颜色以及关节/骨骼直径会随相机坐标系深度变化。最终信号保持二维图像形式，同时携带显式三维线索。

预训练 VAE 将骨架视频映射为控制 latent：

[ c\in\mathbb{R}^{C\times T\times H\times W}, ]

它在空间和时间上与目标视频 latent 对齐。世界模型由此获得精细手指关节和深度信息，同时不依赖完整 mesh 或机器人仿真资产。

2. 由姿态控制的 Video DiT

世界模型基于 Wan2.2-TI2V-5B，在原始视频分支旁增加控制 patch-embedding 分支。骨架 latent 和视频 latent 的统计分布不同，因此控制信号会先对齐到视频 latent 分布：

[ \tilde c = \frac{c-\mu_c}{\sigma_c}\sigma_z+\mu_z, ]

[ x = \operatorname{PatchEmbed}_z(z_t) +\alpha\operatorname{PatchEmbed}_c(\tilde c). ]

控制分支使用零初始化，(\alpha) 的初值也很小，让姿态控制逐步进入网络，同时保留预训练视频先验。直接拼接会破坏原有分布：LoRA 消融中的 FVD 从 585 上升到 1191。

这个条件机制把操作者手势连接到视觉结果，提高了可控性，但无法保证每次生成接触都严格满足真实刚体动力学。

3. 先学习人类先验，再适配机器人 Embodiment

训练横跨两个域。Stage 1 使用大规模第一视角人类视频学习手—物交互先验：

VITRA：3070 万帧；
EgoDex：7400 万帧。

Stage 2 使用 1800 条人—机器人配对遥操 episode 继续训练，共计 43 万机器人视频帧。四项任务分别是双手拾取、推方块、双臂抬升和盒盖放置。人体手势通过 IK 与机器人执行配对，使生成器学会同一人类意图在特定机器人 embodiment 上应该呈现什么样的视觉结果。

人类视频预训练提供物体持续性、接触模式和视觉“常识”。去掉这一阶段后，FVD 恶化到 2598，并出现重影、模糊以及末端执行器或物体消失。学习得到的先验让接触在操作者眼中更合理，也使生成轨迹能够训练策略。它依然属于统计动力学，没有在线显式求解接触力。

4. Streaming Distillation 建立视觉闭环

双向扩散 teacher 太慢，无法交互式控制。作者用两步方法将其转换为因果 student。

第一步是 causal flow-matching warm-up，让模型学习逐帧因果生成。在时刻 (t)，attention 只能使用第 (1,\ldots,t) 帧。固定大小 KV cache 支持流式推理，参考图像 embedding 则作为持续存在的 sink token，维持初始场景身份。

第二步使用 Distribution Matching Distillation（DMD），把 teacher 的输出分布转移到四步生成器。训练会跨连续 chunk 反向传播通过持续 KV cache，减少 chunk 边界伪影并提高长序列一致性。

因果模型在一张 H100、(480\times832) 分辨率下报告：

40.0 FPS；
每个生成帧约 25 ms；
骨架编码约占 5% 延迟；
四步因果 DiT 去噪约占 72%；
VAE 解码约占 23%。

这个速度达到或超过常见的 30 Hz 机器人相机，是论文声称操作者体验流畅的主要依据。论文测量的是模型推理延迟，没有覆盖 tracker、通信、显示器和缓冲在内的完整 motion-to-photon 延迟。

实时性也带来质量交换。在 EgoDex-Test 上，非因果 SFT 模型以 2.8 FPS 获得 FVD 550；因果模型达到 40 FPS 时 FVD 为 1226。交互速度提升伴随着视觉质量下降。

5. Retargeting 生成平滑机器人动作

系统先计算操作者手腕相对胸部的位姿，将平移缩放到机器人工作空间，再变换到机器人基座坐标系：

[ T_{\mathrm{target}} = T_{\mathrm{base}} \cdot \operatorname{Scale}(T_{\mathrm{chest}}^{-1}T_{\mathrm{wrist}}) \cdot T_{\mathrm{ee}}. ]

迭代式 damped least-squares IK 将目标位姿映射到机器人关节。自适应 damping 处理接近奇异位形的情况，null-space shoulder prior 引导自然手臂姿态，hard clipping 则执行关节限制。在物理示范采集期间，Ruckig 继续限制速度、加速度和 jerk，再以 200 Hz 发送手臂命令。机器人底层接口运行在 500 Hz，策略命令为 50 Hz，报告延迟为 18–30 ms。

这些机制提高控制平滑性和动作标签质量，不会为数字遥操员增加物理反馈。

6. 长序列生成与 Re-Anchoring

纯自回归生成会积累视觉漂移。数据生成流程把长视频分成 81 帧 chunk。第一个 chunk 使用真实初始帧，后续 chunk 可以使用来源示范中对应时刻的真实机器人相机帧重新锚定。

这个细节限定了最强意义上的“无限数字生成”。存在物理来源轨迹时，re-anchoring 可以保持物体位置与光照。只从一张图像开始的完全合成在线遥操无法取得未来真实帧，因此需要承受模型漂移，或增加其他状态修正机制。

实验：生成数据是否有用

下游实验分别训练 Diffusion Policy、(\pi_0) 和 (\pi_{0.5})，再对每项任务进行 35 次真实机器人测试。向 300 条真实 episode 中加入 300 条生成 episode 后，成功率总体提高。盒盖放置的提升最明显：

策略	训练数据	盒盖放置成功率
(\pi_{0.5})	300 条真实数据	42.86%
(\pi_{0.5})	300 真实 + 300 生成	62.86%
(\pi_0)	300 条真实数据	34.29%
(\pi_0)	300 真实 + 300 生成	54.29%

只用 300 条生成数据、完全不使用真实数据训练的 (\pi_0)，在推方块任务达到 82.86%，双臂抬升达到 77.14%。盒盖放置只有 28.57%，显示生成数据在精细接触和对准任务上的局限。

这些结果主要验证 RynnWorld-Teleop 的机器人数据生成能力，没有测量人类操作者能否通过生成视觉闭环可靠判断接触力或完成任务。

优点与局限

论文最强的想法是共享动作来源：手部姿态流同时产生视觉观测和 embodiment-specific 动作，生成轨迹可以直接进入模仿学习。Depth-aware skeletal conditioning 保留精细控制，因果蒸馏则把大型视频模型转化为 40 Hz 交互生成器。

实际使用时需要关注以下边界：

系统提供视觉反馈，没有触觉或力反馈。
论文测量了模型延迟，没有评测完整操作者闭环延迟和主观可用性。
生成接触依赖学习得到的视觉先验，在液体和高度可变形物体上可能失败。
当前每种机器人平台都需要单独进行 embodiment adaptation。
因果实时生成的画质低于较慢的 teacher。
长序列 re-anchoring 会在可用时使用真实来源帧，限制了完全合成的解释。

Takeaways

RynnWorld-Teleop 的操作者闭环可以概括为

[ \boxed{ \text{人体手势跟踪} \rightarrow \begin{cases} \text{40 Hz 生成机器人视角视频}
\text{retarget 后的 54D 机器人动作} \end{cases} }. ]

操作者能够快速看到手部动作产生的视觉结果，由此获得可用的控制感。物理手感仍然缺失：接触、负载、摩擦和滑移只呈现在生成像素中，不会回传到操作者手上。对于扩展模仿学习数据，这个取舍很有吸引力；对于接触丰富的临场遥操作或精细力控任务，系统仍需要双向触觉通道和更强的状态锚定。

[Paper Notes] Tactile Genesis: Exploring Tactile Sensors at Scale for Learning Dexterous Tasks

2026-07-16T00:00:00+00:00

Tactile Genesis builds a GPU-parallel tactile simulation platform and uses it to ask a practical hardware-design question: for dexterous manipulation, what tactile signal should a robot hand measure, where should it measure it, and how dense should the sensors be?

The short answer from the paper is unusually concrete. Whole-hand coverage matters more than installing a sophisticated sensor only at the fingertips. Around 200 taxels distributed across the hand are enough for the three studied tasks, and per-taxel force/torque is the strongest general-purpose representation. The policy architecture is mainly an experimental instrument for making this comparison; the central contribution is the simulator and the controlled sensor study.

Paper Info

The paper is “Tactile Genesis: Exploring Tactile Sensors at Scale for Learning Dexterous Tasks” by Trinity Chung, Kashu Yamazaki, Dhruv Patel, Alexis Duburcq, Yiling Qiao, Katerina Fragkiadaki, and Aran Nayebi. It is listed as a CoRL 2026 paper on the project page. The paper and a self-contained code snapshot are publicly available.

What Does the Paper Actually Do?

Existing tactile hardware bundles together several design choices: sensing principle, location, spatial resolution, physical response, and noise. Changing the sensor often changes the robot hand itself, which makes an apples-to-apples comparison almost impossible on real hardware.

Tactile Genesis turns these choices into independent simulation variables. It integrates multiple tactile abstractions into the Genesis physics engine behind a common interface, allows arbitrary placement and resolution on robot surfaces, and adds configurable imperfections such as latency, white noise, bias, random-walk drift, hysteresis, dead taxels, gain variation, and spatial crosstalk. This lets the authors hold the task, hand, policy, and training procedure fixed while changing only the tactile observation.

The work therefore has two layers:

A scalable tactile simulator that implements several sensor families on arbitrary robot geometry.
A controlled policy study that varies signal type, placement, resolution, and noise on three dexterous tasks.

The Simulated Tactile Modalities

The common interface covers seven broad sensing abstractions.

Binary contact: whether each taxel is touching an object.
Contact depth: penetration or distance-to-surface measurements from ray casting or a signed distance field.
Kinematic force/torque: a six-axis estimate at each taxel derived from contact depth, surface normal, and relative linear/angular velocity.
Elastomer displacement: simulated marker motion on a compliant tactile surface, similar to the intermediate signal used by GelSight-style sensors.
Geometry-aware proximity: a pre-contact response obtained by querying tracked object points near each taxel.
Temperature: a voxelized thermal field with diffusion, contact conduction, internal heat generation, radiation, convection, and sensor response lag.
Contact audio: synthesized impact, rolling, sliding, and actuation sounds conditioned on contact physics and material properties.

These models intentionally live at different abstraction levels. KinematicTaxel estimates useful local force without solving full deformable mechanics. ElastomerTaxel models spatially coupled marker displacement. TemperatureGrid solves a coarse heat-transfer field. Contact audio is a parametric signal generator layered over rigid-body contacts. The platform aims to expose policy-relevant sensor outputs at training scale, with parameters that can be calibrated toward real devices.

The implementation reaches more than 20,000 parallel environments and supports more than 1,000 taxels per hand on one GPU. The authors also benchmark regimes above 10,000 taxels. Depending on the sensor and baseline, they report approximately 3x to 20x higher throughput than previous tactile simulators, including TacSL, Tacmap, and HydroShear comparisons.

How the Policy Comparison Works

For each task-hand pair, the authors first train a privileged teacher with PPO. The teacher sees full object state, privileged contact information, proprioception, and the task goal. Random Network Distillation is added to encourage exploration.

They then train a tactile student through DAgger-style behavioral cloning. At deployment, the student receives joint positions, joint velocities, the previous action, one tactile representation, and the goal for reorientation tasks. Small auxiliary heads reconstruct privileged object quantities from the student’s hidden state during training:

[ \mathcal{L}{\text{student}} = \mathcal{L}{\text{BC}} + \mathcal{L}_{\text{aux}}. ]

The auxiliary decoders are discarded after training. Their role is to encourage the tactile encoder to preserve task-relevant object information. Keeping the student architecture and teacher behavior fixed makes performance differences easier to attribute to tactile sensing.

The comparison includes eight downstream tactile observations: per-taxel binary contact, link-aggregated contact, contact depth, link-aggregated force, per-taxel force, per-taxel force/torque, elastomer displacement, and proximity. A proprioception-only student provides the baseline.

Three Dexterous Tasks

The experiments use three XHand1 tasks with different contact regimes.

In-palm rotate: the thumb must locate and capture an object moving on the palm. Pre-contact location information can help.
In-hand repose: the hand continuously manipulates an object toward a target orientation. Slip and grip force are central.
Screwdriver: the fingers perform fast gaiting to keep a screwdriver spinning. Contacts are brief and rapidly changing.

Across these tasks, the study sweeps fingertip, finger, and whole-hand placement; low, medium, and high spatial resolution; clean and noisy sensing; and the tactile representations above.

Main Findings

1. Proprioception alone is insufficient

Every tactile student beats the proprioception-only baseline on all three tasks, including students that receive only binary contact. Joint state and the previous action cannot reliably reveal object slip, decoupled object motion, or hidden contact events.

2. Coverage matters more than sensor sophistication

On in-palm rotation, sensing only at the fingertips leaves a large gap to the privileged teacher. Adding the palm and proximal/middle finger surfaces closes most of that gap, even when the added taxels carry a relatively simple signal. The marginal value of covering a new hand region exceeds the value of upgrading an already instrumented fingertip.

3. Roughly 200 well-placed taxels are enough here

The medium-resolution XHand1 configuration contains 199 taxels across the whole hand. It performs close to the higher-resolution 667-taxel configuration across the tested tasks. The results suggest that coarse spatial contact distribution carries most of the useful information in this setting.

This is a task-scoped result. Fine texture recognition, tiny object geometry, or very local slip estimation could still benefit from high-resolution sensors; those capabilities are outside this experiment.

4. Per-taxel force/torque is the best default

The strongest signal depends on the task. Force/torque is best for in-hand reposing, where incipient slip and grip strength matter. Proximity is strongest for in-palm rotation because it detects an approaching object and lets the thumb pre-shape before contact. On screwdriver spinning, all tactile types behave similarly and remain below the teacher, suggesting that temporal integration or vision may be the missing ingredient.

Aggregated across tasks, per-taxel force/torque matches or exceeds the alternatives most consistently. The paper recommends it as the default tactile abstraction when the hardware supports it.

5. Elastomer marker motion is not automatically the best control signal

Elastomer displacement performs below force/torque on the two sustained-contact tasks. Marker motion at one location depends on deformation at neighboring locations, so the signal is spatially coupled. That coupling is useful for reconstructing local shape; it makes the local force vector less direct for the policy. This result concerns the chosen representation and control tasks, and does not imply that vision-based tactile sensors are generally weak.

Temperature and Audio Experiments

The temperature module is more than a rendering effect. The simulator models diffusion inside a voxel grid, heat exchange through contact, radiation, convection, and a first-order sensor lag. In a separate task, a hand must find one hot ball among eight geometrically identical balls using only proprioception and finger temperature readings.

The policy succeeds only under highly conductive, sufficiently strong thermal configurations. Parameters matching the sensitivity of current robotic temperature hardware fail. The experiment is valuable as a sensor-specification study: simulation can estimate how responsive a future thermal skin would need to be before building it.

The paper also demonstrates contact audio generated from impact, rolling, sliding, material, and actuator state. Audio is part of the platform, but it is not included in the main three-task tactile-policy ablation, so the paper does not yet establish how much it improves dexterous control.

Sim-to-Real Result

The authors deploy the in-palm rotation student on a real XHand1 using its aggregate fingertip force readings. The hand completes one to two consecutive rotations before dropping the object, which approximately matches the corresponding low-information simulated student.

This is useful evidence that the simulated observation can predict the real sensor’s performance level. It is still a modest validation: one hand, one task, aggregate fingertip force, and a low absolute success count. The paper supports simulator plausibility more strongly than broad real-world policy transfer.

What Is New, and What Is Conventional?

The teacher-student learning recipe is fairly standard: privileged PPO teacher, DAgger/behavioral-cloning student, recurrent tactile encoding, and auxiliary state reconstruction. The novelty sits elsewhere:

a unified, configurable set of tactile sensor models inside a fast GPU simulator;
arbitrary sensor placement and resolution on many robot geometries;
realistic and independently controllable sensor imperfections;
controlled comparison of tactile type, coverage, density, and noise;
temperature sensing and contact audio support at robot-learning scale.

So this is best read as a tactile simulation and sensor-design paper, with robot learning used to evaluate which sensor information is actionable.

Limitations

The student inherits the privileged teacher’s strategy and cannot discover behaviors outside that strategy. The main comparison uses three tasks and is centered on XHand1, without visual observations. The real-world test is narrow, and several sensor models are policy-oriented approximations of physical devices. The conclusions about coverage and 200-taxel resolution are compelling design hypotheses, but they need broader task, hand, and hardware studies before becoming universal rules.

Takeaway

The most useful message is a hardware-design priority:

Instrument the palm and proximal finger links first; then improve signal richness and resolution.

For the tasks studied, broad low-to-medium-resolution coverage gives a policy more useful state information than a small high-resolution fingertip patch. Per-taxel force/torque is the safest general signal, proximity helps with pre-contact capture, and richer substrate models should be chosen when the task actually needs their information.

Tactile Genesis makes these claims testable at scale. Its lasting value may be less about the three trained policies and more about giving researchers a common experimental surface for deciding what future tactile hardware should measure.

Tactile Genesis 搭建了一个 GPU 并行的触觉仿真平台，并借助这个平台回答一个很实际的硬件设计问题：灵巧操作到底需要测量哪种触觉信号、传感器应该铺在哪里、空间分辨率需要多高？

论文给出的答案很具体。全手覆盖比只在指尖安装复杂传感器更重要；在本文三个任务中，把大约 200 个 taxels 分布到整只手上已经足够；per-taxel force/torque 是最稳健的通用表示。策略网络在这里主要充当受控实验工具，论文的中心贡献是仿真器和传感器设计研究。

论文信息

论文题目是 “Tactile Genesis: Exploring Tactile Sensors at Scale for Learning Dexterous Tasks”，作者为 Trinity Chung, Kashu Yamazaki, Dhruv Patel, Alexis Duburcq, Yiling Qiao, Katerina Fragkiadaki, and Aran Nayebi。其项目主页将论文列为 CoRL 2026。目前论文和可复现实验的代码快照都已公开。

这篇论文究竟做了什么？

现有触觉硬件通常把多个设计变量绑定在一起：传感原理、安装区域、空间分辨率、材料响应和噪声特性。更换传感器往往等于更换整只机器人手，因此很难在真实硬件上进行严格的同条件比较。

Tactile Genesis 把这些因素变成可以独立控制的仿真变量。它把多种触觉抽象接入 Genesis physics engine，通过统一接口配置机器人表面上的任意 placement 和 resolution，同时模拟 latency、white noise、bias、random-walk drift、hysteresis、dead taxels、gain variation 和 spatial crosstalk 等传感器缺陷。研究者可以固定任务、手、策略和训练流程，只改变 tactile observation。

整项工作可以分成两层：

可扩展的触觉仿真器：在任意机器人几何表面上实现多类 tactile sensors。
受控策略实验：在三个灵巧任务上系统改变信号类型、安装位置、分辨率和噪声。

仿真的触觉模态

统一接口覆盖七类主要 sensing abstractions。

Binary contact：每个 taxel 是否接触物体。
Contact depth：通过 ray casting 或 signed distance field 得到穿透深度或表面距离。
Kinematic force/torque：根据接触深度、表面法向和相对线/角速度，估计每个 taxel 的六轴力与力矩。
Elastomer displacement：模拟柔性触觉表面的 marker motion，对应 GelSight 类传感器中的中间信号。
Geometry-aware proximity：查询 taxel 邻域内的物体点云，在直接接触前产生响应。
Temperature：使用 voxelized thermal field 模拟扩散、接触导热、内部热源、辐射、对流和传感器响应延迟。
Contact audio：依据接触物理和材料属性合成碰撞、滚动、滑动及执行器声音。

这些模型处于不同抽象层级。KinematicTaxel 无需完整求解软体力学，直接估计控制所需的局部力；ElastomerTaxel 建模空间耦合的 marker displacement；TemperatureGrid 求解粗粒度热传导场；contact audio 则在刚体接触结果上生成参数化声音。平台追求的是训练规模下可用、并且能够朝真实设备标定的 policy-relevant sensor output。

在单张 GPU 上，系统可以运行超过 20,000 个并行环境，每只手支持超过 1,000 个 taxels；作者还测试了超过 10,000 taxels 的配置。根据传感器和对比系统不同，其吞吐量相对 TacSL、Tacmap、HydroShear 等已有方法提高约 3 到 20 倍。

策略比较如何进行？

对于每个 task-hand 组合，作者先用 PPO 训练 privileged teacher。teacher 可以看到完整 object state、privileged contact information、proprioception 和任务 goal，并加入 Random Network Distillation 促进探索。

随后通过 DAgger-style behavioral cloning 训练 tactile student。部署时，student 只能看到 joint positions、joint velocities、previous action、一种 tactile representation，以及 reorientation tasks 的 goal。训练阶段还使用小型 auxiliary heads，从 student hidden state 重建 privileged object quantities：

[ \mathcal{L}{\text{student}} = \mathcal{L}{\text{BC}} + \mathcal{L}_{\text{aux}}. ]

部署时会移除 auxiliary decoders。它们的作用是促使 tactile encoder 保留与任务有关的物体状态。整个实验固定 student architecture 和 teacher behavior，使性能差异能够更直接地归因于 tactile sensing。

对比的下游触觉观测一共有八种：per-taxel binary contact、link-aggregated contact、contact depth、link-aggregated force、per-taxel force、per-taxel force/torque、elastomer displacement 和 proximity；另外设置 proprioception-only student 作为 baseline。

三个灵巧操作任务

实验使用三种具有不同接触特性的 XHand1 任务。

In-palm rotate：拇指要定位并捕捉在手掌上运动的物体，pre-contact 位置信息很有帮助。
In-hand repose：手持续控制物体到达目标朝向，slip 和 grip force 是关键信号。
Screwdriver：手指快速 gaiting，使螺丝刀持续旋转；接触短暂且变化很快。

实验系统地改变 fingertip、finger 和 whole-hand placement，低、中、高三档 spatial resolution，clean/noisy sensing，以及前述各种 tactile representations。

主要发现

1. 只有 proprioception 不够

在三个任务中，所有 tactile students 都超过 proprioception-only baseline，甚至只有 binary contact 的 student 也更好。Joint state 和 previous action 无法可靠揭示物体滑动、物体与手解耦运动以及隐藏接触事件。

2. 覆盖范围比传感器复杂度更重要

在 in-palm rotation 中，只覆盖 fingertips 会与 privileged teacher 留下明显差距。增加 palm 以及手指近端/中段表面后，即使新增 taxels 只提供简单信号，也能弥补大部分差距。覆盖新的手部区域带来的边际收益，高于继续升级已经布置传感器的指尖。

3. 本文任务中约 200 个合理分布的 taxels 已经足够

XHand1 的中等分辨率配置在整只手上有 199 个 taxels，其表现已经接近 667-taxel 高分辨率配置。这个结果说明，在当前任务中，粗粒度的接触空间分布包含了大部分有用信息。

这个结论有明确的任务边界。精细纹理识别、微小几何感知或高度局部的 slip estimation 仍可能需要高分辨率传感器，本文实验没有覆盖这些能力。

4. Per-taxel force/torque 是最稳妥的默认选择

每个任务的最佳信号并不完全相同。In-hand reposing 中，incipient slip 和 grip strength 很重要，因此 force/torque 最强。In-palm rotation 中，proximity 可以提前发现靠近的物体，让拇指在接触前完成 pre-shape，所以表现最好。Screwdriver spinning 中，各种触觉信号接近，而且都没有达到 teacher；这暗示缺失的信息可能来自 temporal integration 或 vision。

综合三个任务，per-taxel force/torque 最稳定地达到或超过其他表示。论文建议硬件允许时将其作为默认 tactile abstraction。

5. Elastomer marker motion 不一定是最直接的控制信号

在两个持续接触任务中，elastomer displacement 低于 force/torque。某个位置的 marker motion 会受到周围区域形变影响，因此信号具有空间耦合。它适合重建局部形状，却不方便策略直接读出局部力向量。这个发现针对本文的表示方式和控制任务，并不说明 vision-based tactile sensors 整体能力较弱。

Temperature 与 Audio 实验

Temperature module 包含真实的物理近似：voxel grid 内部扩散、接触导热、辐射、对流，以及一阶 sensor lag。单独的温度任务要求机器人只使用 proprioception 和手指温度，在八个几何完全相同的球中找到一个热球。

策略只在高导热、热信号足够强的配置下成功。按照当前机器人温度传感硬件的灵敏度设置参数时，任务失败。这个实验的价值在于 sensor specification：研究者可以先在仿真中估计未来 thermal skin 需要达到多高的响应能力，再投入硬件开发。

论文还演示了根据碰撞、滚动、滑动、材料和 actuator state 生成的 contact audio。Audio 已经进入平台，但没有参与三个主任务的 tactile-policy ablation，因此论文暂时没有证明它能把灵巧控制提升多少。

Sim-to-Real 结果

作者把 in-palm rotation student 部署到真实 XHand1，并使用手上的 aggregate fingertip force readings。机器人在物体掉落前完成 连续一到两次旋转，与仿真中对应的低信息量 student 大致一致。

这个结果说明 simulated observation 可以预测真实传感器的大致性能水平。验证范围仍然有限：只有一只手、一个任务、aggregate fingertip force，而且绝对成功次数较低。因此它更有力地支持 simulator plausibility，对广泛 real-world policy transfer 的支撑还比较初步。

哪些部分新，哪些部分比较常规？

Teacher-student learning recipe 相对常规：privileged PPO teacher、DAgger/behavioral-cloning student、recurrent tactile encoding 和 auxiliary state reconstruction。创新重点集中在以下方面：

在高速 GPU simulator 中提供统一且可配置的多种 tactile sensor models；
支持多种机器人几何表面上的任意 sensor placement 和 resolution；
独立控制较真实的传感器缺陷；
受控比较 tactile type、coverage、density 和 noise；
在 robot-learning scale 下支持 temperature sensing 和 contact audio。

因此，这篇论文更适合归类为一项 tactile simulation 与 sensor-design 工作，robot learning 用来评估哪些传感信息真正能够转化为控制性能。

局限

Student 继承 privileged teacher 的策略，无法发现 teacher 行为之外的新方案。主要比较只有三个任务，核心硬件是 XHand1，而且没有视觉输入。真实世界验证较窄，部分 sensor models 也是面向策略学习的物理近似。Coverage 和 200-taxel resolution 是很有价值的设计假设，在成为普遍规律之前仍需扩展到更多任务、手和真实传感硬件。

总结

这篇论文最实用的信息是一条硬件设计优先级：

先覆盖手掌和手指近端，再提高信号丰富度与分辨率。

在本文任务中，广泛的中低分辨率覆盖，比一小块高分辨率指尖传感器向策略提供了更多有效状态信息。Per-taxel force/torque 是最稳妥的通用信号，proximity 适合接触前捕捉；只有任务确实需要基底形变信息时，复杂 substrate model 才更有价值。

Tactile Genesis 让这些设计判断可以在大规模实验中被检验。它的长期价值很可能主要来自一个共同实验平台：未来的机器人触觉硬件应该测什么，可以先在这里系统地回答。

[Paper Notes] Grasp Multiple Objects with One Hand

2026-07-15T00:00:00+00:00

MultiGrasp enables a Shadow Hand to grasp and lift multiple independent objects concurrently. Its decisive choice is to make every stage object-aware: the pre-grasp objective asks for force closure on each object, the diffusion model keeps object identities separate, and the lifting reward is controlled by the lowest object. A grasp that secures only one of two targets cannot score well.

My read: the paper succeeds by dividing the problem at the boundary between static contact geometry and dynamic execution. An augmented differentiable force-closure optimizer and a diffusion model construct a multi-contact hand pose; motion planning and goal-conditioned reinforcement learning then reach that pose and preserve all contacts during lift. This split is especially important because moving one finger to help one object can destabilize the other.

Paper Info

“Grasp Multiple Objects with One Hand” is by Yuyang Li, Bo Liu, Yiran Geng, Puhao Li, Yaodong Yang, Yixin Zhu, Tengyu Liu, and Siyuan Huang. It appeared in IEEE Robotics and Automation Letters (RA-L), Vol. 9, No. 5, 2024.

Why Two Objects Change the Problem

A single-object dexterous grasp can often envelop the target and close all fingers toward it. With two independent objects, improving contact on one target may release the other. The hand must use its full workspace—including finger lengths and side surfaces—and maintain a separate force-closure condition for every object.

MultiGrasp studies tabletop arrangements

[ O={O_j}_{j=1}^{N_o}, ]

where each object is represented by a surface point cloud. The output is a sequence of hand actions that simultaneously grasps and lifts all targets. The objects must be close enough and small enough to fit in one hand; this is a geometric assumption of the task, not a capability learned around arbitrary placements.

The complete pipeline has two high-level stages:

propose and refine a pre-grasp pose that encloses all targets;
plan a collision-aware reach, then use a learned policy to lift every object.

1. Independent Force Closure for Every Object

The synthetic grasp generator extends Differentiable Force Closure (DFC). For hand configuration (H) and objects (O), it minimizes

[ E(H,O) = \sum_{j=1}^{N_o} \min_{x_j\subset S(H)} E_{\mathrm{FC}}(x_j,O_j) +\lambda_p E_p(H,O) +\lambda_{sp}E_{sp}(H) +\lambda_qE_q(H). ]

The first term is the key multi-object design. For each object (O_j), the optimizer selects hand contacts (x_j) and independently evaluates force-closure error. Stability cannot come solely from treating the pair as one combined shape. The remaining terms penalize hand-object penetration, hand self-collision, and joint-limit violations. Gradient optimization with Metropolis-Adjusted Langevin Algorithm sampling explores multiple local solutions, followed by filtering.

This objective encourages different parts of the dexterous hand to support different objects. The allocation is produced by contact optimization instead of a fixed assignment such as “thumb and index finger for object A.”

2. A Diffusion Model That Preserves Object Identity

DFC synthesis is effective and slow—roughly 1000 seconds in the reported batched setting. MultiGrasp therefore learns a SceneDiffuser-style DDPM that maps the object point clouds to a pre-grasp pose:

[ p(H^{(0)}\mid O) = \prod_{t=1}^{T}p(H^{(t-1)}\mid H^{(t)},O). ]

PointNet++ extracts features independently from each object. The denoising network uses the noisy hand representation as queries and object features as keys and values in cross-attention. A learned object embedding is attached to every point feature: points from the same object share an embedding, while different objects receive different embeddings. The network can therefore reason about two independent targets instead of seeing one merged point set.

The hand pose is represented by 31 Cartesian keypoints distributed over the palm and finger links. An optimization-based inverse-kinematics solver converts these points to joint angles. Cartesian keypoints expose local finger-link/object relations directly. In the ablation, this representation reaches 40.20% execution success, compared with 19.31% when the model directly generates joint angles.

The training set, Grasp’Em, contains about 90,000 synthetic pre-grasps:

16.4k single-object grasps;
73.7k dual-object grasps;
8 objects and 36 single/dual combinations from YCB and ContactDB;
randomized stable tabletop positions and orientations.

The objects are rescaled so several can fit in the hand. Training also normalizes the palm’s projected direction around the tabletop’s vertical axis. At inference, rotating the scene into this canonical orientation and rotating the predicted hand back allows the system to control palm direction and avoid particularly difficult approaches.

Diffusion samples can contain shallow penetration or leave useful fingers floating near a surface. A coarse-to-fine refinement step minimizes penetration while attracting nearby potential hand contacts toward object surfaces. Its distance threshold decreases from 2 mm to 1 mm.

This small-looking stage has a large effect. Removing refinement increases maximum penetration from 1.67 mm to 7.68 mm and reduces success from 40.20% to 16.24%. The comparison shows that a visually plausible hand pose is insufficient; multi-object lifting needs contacts that survive dynamics.

4. Reach First, Then Learn to Lift

Execution is split into reaching and lifting. The reaching stage linearly interpolates from a flat hand to the pre-grasp, then optimizes the trajectory for smoothness and hand-object collision reduction. Small remaining collisions may displace an object, so the lifting policy is trained to recover from the resulting state.

The PPO lifting policy controls 18 actuated joint-angle targets and a 6D hand-base wrench. Its observations include proprioception, fingertip wrenches, current and goal hand states, object states, and point-cloud features. The policy is goal-conditioned on the pre-grasp configuration instead of discarding it after reaching.

The central reward is

[ r = \omega_{\mathrm{lift}}r_{\mathrm{lift}} +\omega_{\mathrm{succ}}\mathbf{1}{\mathrm{succ}} +\omega_r r_r +\omega_q r_q +\omega{\mathrm{obj}}r_{\mathrm{obj}}, \qquad r_{\mathrm{lift}}=\min_j h_j. ]

Here (h_j) is the height of object (j). Taking the minimum makes the lowest object the bottleneck: lifting one target high cannot compensate for leaving or dropping another. A success bonus is activated only when all objects exceed the training threshold. Additional terms maintain the goal hand orientation, joint configuration, and each object’s position relative to the hand.

This reward encodes simultaneous grasping in the dynamic stage. The ablation is direct: the complete training setting obtains 45.25%, while removing RL leaves only 1.37% success. Removing goal observations and rewards gives 16.79%.

5. Specialists, Vision Distillation, and Error Adaptation

Object shape, pair identity, and placement relative to the palm produce different lifting dynamics. The authors cluster training grasps by object combination and by the direction connecting object centers in the palm frame. A specialist policy is trained for each cluster. Their demonstrations are distilled with DAgger into a vision-based generalist that consumes a fused scene point cloud from three RGB-D cameras.

The final policy also needs to tolerate imperfect pre-grasps. MultiGrasp uses a three-phase curriculum:

train on high-quality synthetic pre-grasps;
mix synthetic samples with generated poses and states disturbed during reaching;
emphasize generated and displaced states.

Without adaptation to imperfect poses, success drops to 25.05%; removing the curriculum yields 24.88%. This training recipe closes the gap between a static target pose and the actual state encountered after physical reaching.

What the Experiments Establish

Success is defined as lifting all objects above 10 cm. Evaluation uses 512 unique poses with five trials per pose. For unseen object placements using synthetic pre-grasps, state-based specialists achieve 68.34%, while the distilled vision generalist achieves the paper’s headline 44.13%. With diffusion-generated pre-grasps on unseen placements, the corresponding results are 40.20% / 30.24%.

The number 44.13% therefore has a precise scope: it is the vision-based generalist’s simulated dual-object success with synthetic pre-grasps on unseen placements. It is not an unrestricted real-world success rate.

The pre-grasp ablations isolate the representation choices:

Pre-grasp model	Q1 ↑	Penetration (mm) ↓	Success ↑
Full model	0.29	1.67	40.20%
Joint-angle representation	0.18	5.50	19.31%
Without object embedding	0.27	1.38	37.21%
Without refinement	0.29	7.68	16.24%

The method also demonstrates three to five small cylinders. As object count grows, inter-object contact becomes increasingly important; the five-object case uses an inverted, scooping hand pose. These tests use synthesized grasps and case-specific state-based execution policies, so they demonstrate scalability of the formulation more than a single general policy across arbitrary object counts.

On the physical system, a Shadow Hand mounted on a UR10e lifts two objects. The authors precompute execution trajectories in simulation and replay them on the robot. This experiment establishes physical feasibility while leaving online perception and closed-loop sim-to-real robustness open.

Strengths and Limitations

The strongest aspect is alignment across stages. Static synthesis, learned generation, execution reward, and evaluation all require success on every object. The method also preserves the distinction between two independent objects through explicit embeddings and per-object state goals.

Its main boundaries are equally clear. Objects must be close and fit in one hand. The data contains only eight mostly convex training objects, and performance falls on unseen combinations and geometries. Vision suffers from severe occlusion around a closed hand, and the physical demonstration replays trajectories prepared in simulation. Typical failures come from missing force closure, penetration, dropping an object during lift, or failing to raise every target. The paper identifies tactile sensing and improved sim-to-real modeling as natural next steps.

Takeaways

MultiGrasp’s recipe for concurrent dual-object grasping can be summarized as

[ \boxed{ \text{per-object force closure} +\text{object-aware diffusion} +\text{Cartesian hand keypoints} +\text{contact refinement} +\min_j h_j\text{-driven lifting} }. ]

The first and last terms are the most fundamental. Per-object force closure defines a static grasp that supports both targets; the minimum-height reward prevents the controller from sacrificing one target during execution. The keypoint generator, refinement, specialist distillation, and curriculum make that objective computationally practical and robust to imperfect reaching.

MultiGrasp 让 Shadow Hand 用一只手同时抓起并抬升多个相互独立的物体。它最关键的选择是让整个流程都显式感知每个物体：pre-grasp 目标要求每个物体分别达到 force closure，扩散模型保留不同物体的身份，抬升奖励则由高度最低的物体决定。双物体中只抓稳一个，策略无法获得高回报。

我的理解是：论文沿着静态接触几何与动态执行的边界拆解问题。扩展后的可微 force-closure 优化器和扩散模型负责构造多接触手型，运动规划与目标条件强化学习负责到达该手型，并在抬升过程中维持所有接触。双物体抓取尤其需要这种分工，因为调整一根手指帮助一个物体时，另一个物体可能随之失稳。

论文信息

论文 “Grasp Multiple Objects with One Hand” 的作者是 Yuyang Li、Bo Liu、Yiran Geng、Puhao Li、Yaodong Yang、Yixin Zhu、Tengyu Liu 和 Siyuan Huang，发表于 IEEE Robotics and Automation Letters（RA-L），Vol. 9, No. 5, 2024。

为什么两个物体会改变问题

单物体灵巧抓取通常可以包住目标，再让所有手指朝它闭合。面对两个彼此独立的物体，改善一个目标的接触可能释放另一个目标。整只手需要充分利用工作空间，包括手指长度和侧面，并为每个物体维持独立的 force-closure 条件。

MultiGrasp 研究桌面场景中的物体集合

[ O={O_j}_{j=1}^{N_o}, ]

每个物体由表面点云表示，系统输出同时抓取并抬升所有目标的手部动作序列。物体需要彼此足够靠近，尺寸也要适合装入一只手；这是任务的几何假设，系统并不能处理任意远距离或任意尺寸的摆放。

完整流程包含两个高层阶段：

生成并优化一个包围所有目标的 pre-grasp；
规划避碰的 reaching 轨迹，再通过学习策略抬起全部物体。

1. 为每个物体分别建立 Force Closure

合成抓姿生成器扩展了 Differentiable Force Closure（DFC）。给定手部构型 (H) 和物体集合 (O)，其优化目标为

[ E(H,O) = \sum_{j=1}^{N_o} \min_{x_j\subset S(H)} E_{\mathrm{FC}}(x_j,O_j) +\lambda_p E_p(H,O) +\lambda_{sp}E_{sp}(H) +\lambda_qE_q(H). ]

第一项是最核心的多物体设计。对于每个物体 (O_j)，优化器选择对应的手部接触点 (x_j)，并单独计算 force-closure error。稳定性不能只来自把两个物体当作一个组合形状来包裹。其余三项分别惩罚手—物体穿透、手部自碰撞和关节越界。算法使用梯度优化与 Metropolis-Adjusted Langevin Algorithm 采样探索多个局部解，最后过滤不合格结果。

这个目标会促使灵巧手的不同区域分别支撑不同物体。接触分配由优化产生，并未硬编码成“拇指和食指负责物体 A”之类的固定规则。

2. 保留物体身份的扩散模型

DFC 合成有效但很慢，论文报告的批量设置约需 1000 秒。MultiGrasp 因此训练了一个基于 SceneDiffuser 的 DDPM，把多个物体的点云映射为 pre-grasp：

[ p(H^{(0)}\mid O) = \prod_{t=1}^{T}p(H^{(t-1)}\mid H^{(t)},O). ]

PointNet++ 分别提取每个物体的特征。去噪网络以带噪手部表示为 query，以物体特征为 key 和 value 进行 cross-attention。每个点特征还附带一个可学习的 object embedding：同一物体的点共享 embedding，不同物体使用不同 embedding。网络由此能够推理两个独立目标，而不会把它们看成一个合并点集。

手部姿态使用分布在手掌和手指 link 上的 31 个笛卡尔关键点表示，再由优化式 IK 恢复关节角。笛卡尔关键点直接表达局部的手指 link—物体关系。消融实验中，这种表示达到 40.20% 执行成功率，直接生成关节角只有 19.31%。

训练数据集 Grasp’Em 包含约 9 万个合成 pre-grasp：

16.4k 个单物体抓姿；
73.7k 个双物体抓姿；
来自 YCB 和 ContactDB 的 8 种物体，共 36 种单双物体组合；
随机生成的稳定桌面位置和朝向。

物体会被缩放到多个目标能够装入一只手。训练还会围绕桌面竖直轴归一化手掌投影方向。推理时先把场景旋转到规范朝向，生成后再把手型旋回原场景，从而控制手掌接近方向，并避开特别困难的抓取朝向。

扩散模型生成的手型可能有轻微穿透，也可能让有用的手指悬在物体表面附近。论文加入由粗到细的 refinement：在降低穿透的同时，把附近的潜在手部接触点拉向物体表面，距离阈值从 2 mm 逐步缩小到 1 mm。

这个看似很小的步骤影响很大。去掉 refinement 后，最大穿透从 1.67 mm 上升到 7.68 mm，成功率从 40.20% 降到 16.24%。外观合理的手型还不够，多物体抬升需要能够经受动力学过程的真实接触。

4. 先到达抓姿，再学习抬升

执行过程分成 reaching 和 lifting。Reaching 首先从张开的手线性插值到 pre-grasp，随后优化整段轨迹的平滑性，并减少手—物体碰撞。少量残余碰撞仍可能推动物体，因此 lifting 策略专门训练了对这种状态偏移的恢复能力。

PPO 抬升策略控制 18 个主动关节角目标和一个 6D 手掌基座 wrench。观测包含本体状态、指尖 wrench、当前与目标手部状态、物体状态和点云特征。策略始终以 pre-grasp 为目标条件，在 reaching 结束后继续使用这份目标信息。

最重要的奖励是

其中 (h_j) 是第 (j) 个物体的高度。取最小值会让最低的物体成为瓶颈：把一个目标抬得很高，无法补偿另一个目标留在桌面上或中途掉落。只有所有物体超过训练阈值时才得到成功奖励。其他奖励项分别维持目标手掌朝向、关节构型和各物体相对手掌的位置。

这个奖励在动态执行阶段直接编码了“同时抓住”。消融结果很清楚：完整训练达到 45.25%，去掉 RL 后只剩 1.37%；去掉目标观测和目标奖励后为 16.79%。

5. Specialist、视觉蒸馏与误差适应

物体形状、物体组合和二者相对手掌的摆放都会改变抬升动力学。作者按照物体组合，以及两个物体中心连线在手掌坐标系中的方向，对训练抓姿进行聚类。每个 cluster 训练一个 specialist，再通过 DAgger 将示范蒸馏到视觉 generalist。最终策略使用三台 RGB-D 相机融合得到的场景点云。

策略还需要容忍不准确的 pre-grasp。MultiGrasp 使用三阶段 curriculum：

先训练高质量合成 pre-grasp；
混合合成样本、扩散生成的手型，以及 reaching 过程中发生位移的状态；
后期以生成样本和受扰动状态为主。

去掉对不准确抓姿的适应训练后，成功率降到 25.05%；保留适应但去掉 curriculum 后为 24.88%。这套训练方法连接了静态目标手型与真实 reaching 结束后遇到的物理状态。

实验到底证明了什么

实验把成功定义为所有物体都被抬到 10 cm 以上。评估包含 512 个独立抓姿，每个运行五次。对于未见过的物体摆放和合成 pre-grasp，基于状态的 specialist 达到 68.34%，蒸馏后的视觉 generalist 达到论文摘要中的 44.13%。换成扩散模型生成的 pre-grasp 后，对应结果为 40.20% / 30.24%。

因此，44.13% 的准确含义是：在仿真中，面对未见过的双物体摆放，视觉 generalist 使用合成 pre-grasp 得到的成功率。它不是无约束真实场景中的成功率。

Pre-grasp 消融进一步分离了各项表示设计：

Pre-grasp 模型	Q1 ↑	穿透（mm）↓	成功率 ↑
完整模型	0.29	1.67	40.20%
关节角表示	0.18	5.50	19.31%
无 object embedding	0.27	1.38	37.21%
无 refinement	0.29	7.68	16.24%

论文还展示了抓取三到五个小圆柱。物体数量增加后，物体间接触越来越重要；五物体场景使用了翻转手掌的“铲取”姿态。这些实验采用合成抓姿，并为不同物体数量训练专门的状态策略，因此它们主要证明框架形式可以扩展，还没有得到覆盖任意物体数量的单一通用策略。

真实系统由 UR10e 搭载 Shadow Hand，能够从桌面上抓起两个物体。作者先在仿真中预计算执行轨迹，再在机器人上复现。这个实验验证了物理可行性，在线感知和闭环 sim-to-real 鲁棒性仍有待解决。

优点与局限

论文最强的地方是各阶段目标一致。静态合成、学习式生成、执行奖励和评测都要求每个物体成功。显式 object embedding 与逐物体状态目标也始终保留两个独立实体之间的区别。

方法边界同样明确。物体必须足够接近并能放入一只手，训练集只有 8 种以凸形为主的物体，遇到新组合和新几何时性能明显下降。手部闭合后会造成严重视觉遮挡，真实机器人演示也使用了仿真预计算轨迹。典型失败包括缺少 force closure、手—物体穿透、抬升途中掉落一个物体，以及部分目标始终没有离开桌面。论文把触觉感知和更准确的 sim-to-real 建模列为后续方向。

Takeaways

MultiGrasp 的并发双物体抓取方案可以概括为

[ \boxed{ \text{逐物体 force closure} +\text{物体感知扩散模型} +\text{笛卡尔手部关键点} +\text{接触 refinement} +\min_j h_j\text{ 驱动的抬升策略} }. ]

其中第一项和最后一项最根本。逐物体 force closure 定义了同时支撑两个目标的静态手型，最小高度奖励防止控制器在动态执行中牺牲任何一个目标。关键点生成、接触 refinement、specialist 蒸馏和 curriculum 则让这个目标具备可计算性，并能容忍 reaching 产生的误差。

[Paper Notes] InterMimic: Towards Universal Whole-Body Control for Physics-Based Human-Object Interactions

2026-07-14T00:00:00+00:00

InterMimic learns one physics-based controller for diverse whole-body human-object interactions (HOIs) from imperfect motion-capture data. Its central recipe is “perfect first, then scale up”: small teacher policies first retarget and physically repair the captured interactions; a Transformer student then absorbs the teachers’ corrected references and actions, followed by reinforcement-learning fine-tuning.

My read: the most important idea is that the teachers are more than action-label generators. Their simulated rollouts become a cleaner training distribution for the student. This converts teacher-student learning into two coupled operations—reference distillation cleans what the student should track, while policy distillation teaches how to track it. PPO then resolves conflicts left by supervised aggregation.

Paper Info

The paper is “InterMimic: Towards Universal Whole-Body Control for Physics-Based Human-Object Interactions” by Sirui Xu, Hung Yu Ling, Yu-Xiong Wang, and Liang-Yan Gui, from the University of Illinois Urbana-Champaign and Electronic Arts. It was presented at CVPR 2025 as a Highlight paper.

The Problem: Large HOI Data Is Useful and Physically Messy

Physics-based motion imitation usually asks a simulated character to follow a reference trajectory. Human-object interaction raises the difficulty because errors couple across the body and the object. A floating hand contact can make the object fall; an inaccurate symmetric-object rotation can imply sliding; shape differences between captured subjects and the simulated body can destroy a valid interaction after retargeting. Many HOI datasets also omit detailed finger motion.

The target trajectory contains human and object states:

[ q_t = {q_t^h,q_t^o}, \qquad q_t^h={\theta_t^h,p_t^h}, \qquad q_t^o={\theta_t^o,p_t^o}. ]

The human model has 52 joints: 22 body joints and 30 hand joints. Fifty-one joints are actuated, and the action

[ a_t\in\mathbb{R}^{51\times 3} ]

specifies exponential-map joint targets that a PD controller converts into torques. The objective is therefore richer than pose matching: the controller must preserve the interaction outcome while repairing reference errors and accommodating a new embodiment.

The Two-Stage Curriculum

Stage	Model	Training signal	Main job
1. Perfect	Subject-specific MLP teachers	PPO with tracking, contact, and energy rewards	Retarget and physically refine small data subsets
2. Scale	One Transformer student	Refined references, teacher actions, DAgger, then PPO	Integrate many subjects, objects, and interaction skills

The decomposition is a space-time trade-off. Seventeen OMOMO teacher policies can be trained on subject-sized subsets in parallel. Their experience then bootstraps one large student, avoiding the sample cost of discovering every skill through PPO from scratch.

Policy Inputs: Geometry, Contact, and Future Goals

The policy state has a physical-state component (s_t^s) and a goal component (s_t^g):

[ s_t={s_t^s,s_t^g}. ]

The physical state contains human and object poses and velocities, plus two interaction descriptors:

(d_t): vectors from human joints to the nearest points on each object surface;
(c_t): body-part contact markers derived from applied forces.

The goal provides relative and absolute reference information at several future keyframes. Teachers use offsets (K={1,16}); the student uses (K={1,2,4,16}). The longer temporal context helps the student distinguish skills that look locally similar but require different future actions.

Most MoCap datasets have no reliable contact labels. InterMimic infers reference contact using object acceleration as evidence of human-applied force, then combines it with distance thresholds. Contacts fall into three zones: promote, neutral, or penalize. The neutral buffer is important because a noisy reference distance should not become a rigid physical constraint.

Stage 1: Imitation as Retargeting and Perfecting

Each teacher uses the same canonical human model while learning data from one captured subject. This embeds retargeting inside RL. The paper separates the reward into two kinds of evidence.

Embodiment-aware terms loosely preserve the captured kinematics. Joint-position tracking receives more weight near an object, where spatial precision controls contact; joint-rotation tracking dominates farther away. Joint-to-object distance tracking preserves the interaction layout.

Embodiment-agnostic terms preserve the interaction dynamics through object position and rotation tracking, promoted or penalized contacts, and energy penalties. The authors’ assumption is that body shape may change the exact kinematic trajectory while the human-object dynamics should remain consistent.

For a tracking cost (E), each corresponding reward takes the exponential form

[ R_E=\exp(-\lambda E). ]

The complete scalar reward combines human pose, interaction distance, object pose, contact, and energy terms. The contact-energy penalty discourages abrupt forces and jitter.

Recovering Missing Hand Interaction

OMOMO and BEHAVE often contain averaged or flattened hand poses. When a fingertip or palm lies close to an object, InterMimic activates a hand contact target and lets PPO discover a workable grasp under hand range-of-motion constraints. This recovers functional contact for the paper’s relatively low-dexterity tasks; it does not reconstruct precise captured finger articulation.

Physical State Initialization (PSI)

Reference State Initialization (RSI) starts an episode from a random MoCap frame. With faulty contacts, that frame may already be dynamically unrecoverable: the object drops immediately, or a floating hand cannot reach it in time.

PSI maintains an initialization buffer containing raw reference states and successful simulation states from earlier rollouts. After a rollout, high-return trajectory states enter the FIFO buffer. Future episodes can therefore begin from states that are close to the intended phase and physically achievable. PSI preserves access to later motion phases without repeatedly initializing the simulator inside broken reference configurations.

Interaction Early Termination (IET)

Standard motion-imitation termination checks character deviation and unwanted ground contacts. IET adds HOI-specific failure tests. An episode ends when:

mean object-point deviation exceeds 0.5 m;
the weighted joint-to-object distance differs from the reference by more than 0.5 m; or
a required body-object contact is absent for over 10 consecutive frames.

This keeps PPO from spending rollout budget after the interaction has already failed.

Stage 2: Two Forms of Distillation

After teacher training, the teachers are frozen and rolled out online. Their trajectories provide both states (s^{(T)}) and actions (a^{(T)}).

Reference Distillation

Teacher simulation states replace noisy MoCap states when constructing the student’s goal and reward. This transfers corrected contacts, feasible hand placement, a canonical embodiment, and physically consistent object motion. The student learns from references that its simulator can actually realize.

Policy Distillation and RL Fine-Tuning

The teacher actions supervise the student through behavior cloning and DAgger. Training gradually shifts toward PPO:

[ \mathcal{L}t = w_t\mathcal{L}{\mathrm{PPO}} + (1-w_t)\lVert a^{(S)}-a^{(T)}\rVert, \qquad w_t\uparrow 1. ]

Early supervision provides efficient skill acquisition. Later PPO updates optimize the actual interaction reward and resolve ambiguous supervision, such as different teachers choosing different controls for similar observed states. The critic is trained throughout this transition.

The teachers are three-hidden-layer MLPs with widths 1024, 1024, and 512. The student uses a three-layer Transformer encoder with four heads, hidden size 256, and feed-forward size 512. All controllers run at 30 Hz in Isaac Gym.

Experiments

InterMimic draws on OMOMO, BEHAVE, HODome, IMHD, and HIMO. OMOMO is the main large-scale benchmark, with 15 objects and about 10 hours of motion. The authors train 17 subject-specific teachers, reserve subject 14 as the test subject, and discard a small amount of severely corrupted data that the teachers cannot repair.

The metrics measure success rate, imitation duration before IET, human joint-position error (E_h), and object point-position error (E_o).

Teacher-Level Correction

On one BEHAVE subject interacting with a yoga mat, the full teacher substantially improves over SkillMimic:

Method	Duration ↑	(E_h) (cm) ↓	(E_o) (cm) ↓
SkillMimic	12.2	7.2	13.4
InterMimic without IET	40.3	6.7	9.9
InterMimic without PSI	36.1	6.6	10.2
InterMimic	42.6	6.4	9.2

Both PSI and IET contribute. The qualitative examples show the controller correcting floating contacts, misplaced hands, and implausible rotations of symmetric objects.

Student Scaling and Generalization

Direct PPO on raw MoCap reaches only 9.6% success on the held-out OMOMO subject. The complete Transformer student reaches 98.1%, with 176.5 seconds of tracked duration, 5.9 cm human error, and 11.3 cm object error. The full MLP student scores 95.5% on the same split, while the Transformer has the best test success and improves results on the InterDiff-generated references.

The broader table adds useful nuance. A ten-times-heavier-object test remains hard: the Transformer student’s success falls to 56.8%. Text-conditioned HOI-Diff references are harder still, with 12.5% success. On future interactions generated by InterDiff, success reaches 76.7%. These experiments support zero-shot compatibility with new references and geometries, while also showing that “universal” does not mean uniformly solved.

What the Ablations Say

The ablations give each training component a distinct role:

PSI makes later and contact-sensitive motion phases reachable during training.
IET removes rollouts whose human-object relationship has already broken.
Reference distillation produces a large gain on the held-out body shape because the teachers have already retargeted all references to one embodiment.
Policy distillation supplies a scalable supervised starting point.
PPO fine-tuning resolves teacher conflicts and improves beyond demonstration averaging.
The Transformer benefits from denser future keyframes and temporal modeling, especially on held-out and generated references.

This separation is a major strength of the paper: data repair, optimization efficiency, and policy capacity are handled by different mechanisms with corresponding measurements.

Limitations

The supplementary material identifies several concrete boundaries:

Teachers fail when a reference contains too many severe errors, such as a flipped hand. Those clips are filtered before student training.
The simulator can produce unnatural object support through penetration. Contact-energy penalties reduce the problem but do not eliminate it.
Hand recovery works for the evaluated tasks and may be insufficient for dexterous manipulation requiring detailed finger motion.
Soft-body interactions, such as carrying a bag by its strap, are excluded because the simulator lacks suitable support.
Generalization still depends on coverage. More diverse objects and motions should improve performance, and the heavy-object and HOI-Diff results leave substantial room for progress.

The real-humanoid demonstrations are best read as evidence that the formulation can transfer to a Unitree G1 embodiment. The paper’s main quantitative claims concern physics simulation, so broad real-world robustness remains open.

Takeaways

InterMimic offers a reusable recipe for learning control from imperfect motion data:

use small experts to turn noisy kinematic demonstrations into feasible physical trajectories;
distill both the corrected targets and the expert actions;
transition from imitation to reward optimization as the student becomes competent;
encode geometry and contact explicitly so one controller can span object shapes and interaction modes.

The conceptual shift is subtle and valuable. The MoCap sequence becomes a task specification with uncertain measurements. Physics-based RL is allowed to modify the path while preserving the interaction, and the corrected simulations become the dataset used for scale.

InterMimic 从不完美的动作捕捉数据中学习一个统一的物理控制器，覆盖多种全身人-物交互（Human-Object Interaction, HOI）。它的核心路线是 “先完善，再扩展”（perfect first, then scale up）：小规模教师策略先对交互动作进行重定向和物理修正，Transformer 学生再吸收教师生成的修正版参考轨迹与控制动作，最后通过强化学习继续优化。

我的理解是：教师的价值远超动作标签生成。教师在仿真器中的 rollout 为学生构造了一个更干净的训练分布。教师-学生学习因此包含两个相互配合的过程：参考蒸馏负责修正“学生应该跟踪什么”，策略蒸馏负责传递“应该怎样跟踪”；PPO 随后处理监督聚合遗留的策略冲突。

论文信息

论文题目为 “InterMimic: Towards Universal Whole-Body Control for Physics-Based Human-Object Interactions”，作者是 Sirui Xu、Hung Yu Ling、Yu-Xiong Wang 和 Liang-Yan Gui，来自伊利诺伊大学厄巴纳-香槟分校与 Electronic Arts。论文发表于 CVPR 2025，并入选 Highlight。

问题：大规模 HOI 数据很有价值，也存在物理缺陷

物理运动模仿通常要求仿真角色跟踪参考轨迹。进入人-物交互后，误差会沿人体与物体的耦合关系传播：悬空的手部接触可能让物体掉落；对称物体的错误旋转可能表现为贴地滑动；被采集者和仿真人体之间的体型差异也会让原本有效的交互在重定向后失效。许多 HOI 数据集还缺少细致的手指动作。

目标轨迹同时包含人体和物体状态：

[ q_t = {q_t^h,q_t^o}, \qquad q_t^h={\theta_t^h,p_t^h}, \qquad q_t^o={\theta_t^o,p_t^o}. ]

人体模型共有 52 个关节，包括 22 个身体关节和 30 个手部关节。其中 51 个关节可驱动，动作

[ a_t\in\mathbb{R}^{51\times 3} ]

表示指数映射形式的关节目标，随后由 PD 控制器转换为关节力矩。因此，学习目标需要同时完成姿态跟踪、参考误差修复、接触结果保持和跨体型适配。

两阶段课程

阶段	模型	训练信号	主要任务
1. 完善	按被试划分的 MLP 教师	PPO、跟踪奖励、接触奖励、能量奖励	对小数据子集进行重定向和物理修正
2. 扩展	单一 Transformer 学生	修正版参考、教师动作、DAgger、PPO	整合多被试、多物体和多类交互技能

这种拆分形成了空间换时间的训练方式。17 个 OMOMO 教师可以分别在被试级数据子集上并行学习，再由一个大规模学生吸收它们的经验，减少学生从零开始通过 PPO 探索全部技能的样本成本。

策略输入：几何、接触与未来目标

策略状态由物理状态 (s_t^s) 和目标状态 (s_t^g) 组成：

[ s_t={s_t^s,s_t^g}. ]

物理状态包含人体和物体的姿态与速度，还包含两类交互描述：

(d_t)：从人体各关节指向物体表面最近点的向量；
(c_t)：依据受力判断的身体部位接触标记。

目标状态在多个未来关键帧上提供相对和绝对参考信息。教师使用 (K={1,16})，学生使用 (K={1,2,4,16})。学生获得更密集的时间上下文后，可以区分局部状态相似、后续动作不同的交互技能。

大多数 MoCap 数据没有可靠的接触标签。InterMimic 把物体加速度作为人体施力的证据，再与距离阈值结合，推断参考接触。接触标签分为促进、缓冲和惩罚三个区域。缓冲区可以避免把有噪声的参考距离强行变成严格的物理约束。

第一阶段：把模仿作为重定向与完善

每个教师都使用相同的规范人体模型，同时学习一个被试的数据，重定向过程由此直接进入 RL 训练。论文将奖励中的证据分为两类。

与 embodiment 相关的奖励对采集运动学进行宽松保持。人体靠近物体时，关节位置跟踪的权重更高，因为空间精度决定能否建立接触；远离物体时，关节旋转跟踪占据更高权重。关节到物体的距离奖励用于保持交互布局。

与 embodiment 无关的奖励通过物体位置、物体旋转、接触促进或惩罚以及能量约束来保持交互动力学。作者的基本假设是：体型变化会改变具体的运动学轨迹，而人体与物体之间的动力学关系应保持一致。

对任意跟踪代价 (E)，对应奖励采用指数形式：

[ R_E=\exp(-\lambda E). ]

最终标量奖励综合人体姿态、交互距离、物体姿态、接触和能量等部分。接触能量惩罚用于抑制突发大力和抖动。

恢复缺失的手部交互

OMOMO 和 BEHAVE 中的手部姿态经常经过平均化或被压平。只要指尖或手掌靠近物体，InterMimic 就激活手部接触目标，让 PPO 在手部运动范围约束下搜索可用的抓持方式。该方法可以为论文中的低灵巧度任务恢复功能性接触，但无法还原精确的被采集手指关节运动。

Physical State Initialization（PSI）

Reference State Initialization（RSI）从随机 MoCap 帧启动 episode。当这一帧存在错误接触时，初始状态在动力学上可能已经无法挽回，例如物体立即掉落，或悬空的手无法及时到达物体。

PSI 维护一个初始化缓冲区，其中同时保存原始参考状态和此前 rollout 中成功的仿真状态。一次 rollout 结束后，高回报轨迹中的状态进入 FIFO 缓冲区。后续 episode 因而能够从接近目标动作阶段、同时满足物理可达性的状态启动。这样可以继续学习轨迹后段和接触敏感阶段，也减少了从破损参考状态重复初始化的次数。

Interaction Early Termination（IET）

常规运动模仿会在角色偏离过大或发生非计划地面接触时终止。IET 增加三项 HOI 失败条件：

物体表面点的平均偏差超过 0.5 m；
加权关节-物体距离相对参考值的偏差超过 0.5 m；
参考要求的身体-物体接触连续缺失超过 10 帧。

当交互关系已经失效时，IET 会停止 episode，避免继续消耗 PPO rollout 预算。

第二阶段：两种蒸馏

教师训练完成后参数被冻结，并在线生成 rollout。教师轨迹同时提供状态 (s^{(T)}) 和动作 (a^{(T)})。

参考蒸馏

构造学生的目标与奖励时，教师仿真状态替代带噪声的 MoCap 状态。学生得到的参考已经包含修正后的接触、可行的手部位置、统一的人体 embodiment 和物理一致的物体运动。这些目标在学生所处的仿真器中具有实际可实现性。

策略蒸馏与 RL 微调

教师动作通过行为克隆和 DAgger 监督学生，训练过程逐渐转向 PPO：

[ \mathcal{L}t = w_t\mathcal{L}{\mathrm{PPO}} + (1-w_t)\lVert a^{(S)}-a^{(T)}\rVert, \qquad w_t\uparrow 1. ]

早期监督让学生高效获得技能，后期 PPO 直接优化交互奖励，并处理模糊监督。例如，当多个教师对相似状态输出不同控制动作时，PPO 可以帮助学生选择更高回报的解。Critic 在整个过渡过程中持续训练。

教师是三层隐藏层 MLP，宽度分别为 1024、1024 和 512。学生使用三层 Transformer Encoder，包含 4 个注意力头、256 维隐藏层和 512 维前馈层。所有控制器均在 Isaac Gym 中以 30 Hz 运行。

实验

InterMimic 使用 OMOMO、BEHAVE、HODome、IMHD 和 HIMO。OMOMO 是主要的大规模评测数据集，包含 15 种物体和约 10 小时运动。作者训练 17 个按被试划分的教师，将 14 号被试留作测试，并丢弃少量教师也无法修复的严重损坏数据。

评测指标包括成功率、触发 IET 前的持续模仿时间、人体关节位置误差 (E_h) 和物体表面点位置误差 (E_o)。

教师级修正能力

在 BEHAVE 中单一被试与瑜伽垫交互的实验上，完整教师相对 SkillMimic 有明显提升：

方法	持续时间 ↑	(E_h)（cm）↓	(E_o)（cm）↓
SkillMimic	12.2	7.2	13.4
InterMimic 去掉 IET	40.3	6.7	9.9
InterMimic 去掉 PSI	36.1	6.6	10.2
InterMimic	42.6	6.4	9.2

PSI 和 IET 都有独立贡献。定性结果还展示了控制器对悬空接触、错误手部位置和对称物体不合理旋转的修正。

学生的扩展能力与泛化

直接在原始 MoCap 上运行 PPO，在 OMOMO 留出被试上只有 9.6% 成功率。完整 Transformer 学生达到 98.1%，持续跟踪时间为 176.5 秒，人体误差为 5.9 cm，物体误差为 11.3 cm。完整 MLP 学生在相同划分上的成功率为 95.5%；Transformer 在测试成功率上更高，并改善了对 InterDiff 生成参考的执行效果。

完整表格也揭示了边界。在物体质量放大十倍的测试中，Transformer 学生成功率下降到 56.8%。文本条件 HOI-Diff 生成的参考更难，成功率为 12.5%。对 InterDiff 预测的未来交互，成功率达到 76.7%。这些结果支持策略对新参考和新几何的零样本兼容性，同时说明 “universal” 仍然对应一个尚未均匀解决的目标。

消融实验揭示了什么

各组件在消融实验中承担不同职责：

PSI 让轨迹后段和接触敏感阶段在训练中真正可达；
IET 及时停止人体-物体关系已经破坏的 rollout；
参考蒸馏先将所有参考重定向到统一 embodiment，因此在留出体型上带来显著提升；
策略蒸馏提供可扩展的监督学习起点；
PPO 微调处理教师冲突，并突破动作平均化带来的性能上限；
Transformer利用更密集的未来关键帧和时序建模，在留出参考与生成参考上更有优势。

这种职责分解是论文的一项重要优点：数据修复、优化效率与策略容量分别由不同机制处理，并得到对应实验验证。

局限性

补充材料明确给出了几项边界：

当参考中包含过多严重错误时，教师也会失败，例如手掌方向完全翻转。这类片段会在学生训练前被过滤。
仿真器有时会通过物体穿透产生不自然的支撑。接触能量惩罚可以缓解，但无法彻底消除。
手部恢复适用于论文评测的任务，对需要精细手指运动的灵巧操作可能不足。
由于仿真器缺少合适的软体支持，论文排除了通过包带提包等交互。
泛化能力仍受数据覆盖范围影响。增加物体和动作多样性有望继续提升性能；重物体和 HOI-Diff 实验仍有很大改进空间。

真实人形机器人演示说明该建模方式可以迁移到 Unitree G1 embodiment。论文的核心定量结论来自物理仿真，因此大范围真实环境鲁棒性仍是开放问题。

总结

InterMimic 给出了一套可复用的不完美运动数据学习方案：

用小型专家把带噪声的运动学示范转换为物理可行轨迹；
同时蒸馏修正后的目标和专家的动作；
随着学生能力提升，从动作模仿逐步转向奖励优化；
显式编码物体几何与接触，让单一控制器覆盖多种物体形状和交互模式。

其中的观念转变很有价值：MoCap 序列被视为带有测量不确定性的任务规格。物理 RL 可以在保持交互结果的前提下修改执行路径，修正后的仿真轨迹再成为规模化学习使用的数据集。

[Paper Notes] UniDexTok: A Unified Dexterous Hand Tokenizer from Real Data

2026-07-14T00:00:00+00:00

UniDexTok studies a representation problem that sits before dexterous manipulation policy learning: how can data from different human and robot hands be put into one shared state space, then converted into discrete tokens that future VLA or world-action models can use?

My read: the paper’s core contribution is a retargeting-free state tokenizer. UniDexTok avoids converting every hand trajectory into a target robot hand through MANO-to-robot retargeting. It first standardizes human and robot hand states into a 22-DoF semantic interface called UDHM, then trains one conditional tokenizer with a shared encoder, shared codebook, and shared decoder across embodiments. The important shift is from “make every hand imitate one robot” to “make every hand speak the same joint-semantic language.”

Paper Info

The paper is “UniDexTok: A Unified Dexterous Hand Tokenizer from Real Data” by Dong Fang, Youjun Wu, Yuanxin Zhong, Rui Zhang, Yunlong Wang, Xiaosong Jia, and Yu-Gang Jiang from Fudan University, Rimbot, Hefei University of Technology, and Beijing University of Posts and Telecommunications. It is available as arXiv:2606.10683.

Parallel grippers are easy to represent: opening width, gripper pose, maybe a binary open-close state. Dexterous hands are messier. Different hands have different numbers of fingers, joint names, joint orders, coupled joints, control ranges, and units. Even datasets collected with the same family of hands can log joint states in different conventions, including radians, degrees, or discretized bins.

This makes cross-embodiment learning awkward. If every robot hand has a separate tokenizer, then each codebook learns a private latent space. The model may reconstruct each hand, but the discrete tokens do not have a stable meaning across hardware. If the pipeline relies on retargeting human motion into every robot hand, the representation inherits geometric mismatch from retargeting and loses the native state statistics of real robot data.

UniDexTok’s answer is to standardize the state space first, then tokenize.

UDHM: A 22-DoF Semantic Interface

The Unified Dexterous Hand Model, or UDHM, is the paper’s shared coordinate system. It assumes a human-hand-like kinematic structure and maps MANO-style human keypoints and robot hand joints into a 22-dimensional active joint interface.

For the index, middle, and ring fingers, UDHM uses four active coordinates per finger: MCP abduction/adduction, MCP flexion/extension, PIP flexion/extension, and DIP flexion/extension. The pinky adds one twist degree of freedom to cover larger lateral variation. The thumb uses five coordinates: CMC flexion, CMC spread, MCP flexion, MCP abduction, and IP flexion. Together these choices give 22 active coordinates.

The model fits a palm plane from the wrist and MCP keypoints, uses the palm normal and local lateral axes to define motion planes, and reconstructs each finger chain with analytic forward kinematics. For inverse mapping, it extracts palm offsets and bone lengths from the input frame, then solves a nonlinear least-squares problem so the reconstructed joints match the target keypoints.

The practical detail I like is the treatment of heterogeneous robot logs. Each available robot-hand DoF is inserted into the semantically corresponding UDHM coordinate, and missing coordinates are zero-padded. This “semantic insertion” is stronger than simply appending all available coordinates at the front of a vector, because the model can learn that a coordinate means “index MCP pitch” across embodiments without memorizing source-specific column positions.

All raw joint angles are converted to radians and normalized by a fixed scale of (\pi). That fixed normalization keeps tokens from depending on train-test split statistics and keeps MPJAE interpretable after denormalization.

UniDexTok Architecture

Given a standardized hand state (x \in \mathbb{R}^{22}) and a hand-type or source label (h), the tokenizer learns

[ \hat{x} = \pi D(Q(E(\tilde{x}, h_{\mathrm{embed}})), h_{\mathrm{embed}}), \quad \tilde{x}=x/\pi. ]

The encoder projects the 22-dimensional normalized state into (N=8) latent tokens with width (C=512). Transformer blocks then model correlations among joint coordinates. Hand-type conditioning enters through adaptive layer normalization:

[ \mathrm{AdaLN}(z,h_{\mathrm{embed}})=\gamma(c_h)\odot \mathrm{LN}(z)+\beta(c_h). ]

This conditioning matters because the same semantic pose can require different numeric joint patterns on different hands. The shared encoder and decoder learn the cross-hand structure, while the conditioning keeps hardware-specific conventions available.

The decoder mirrors the encoder. It takes quantized tokens, adds positional embeddings and hand conditioning, and reconstructs the normalized 22-dimensional state. The reconstruction objective is based on normalized joint-angle reconstruction, and the full objective adds the VQ commitment loss:

[ L = L_{\mathrm{rec}} + L_{\mathrm{vq}}, ]

with

[ L_{\mathrm{vq}}=\beta\lVert \mathrm{sg}[q]-u\rVert_2^2+\lVert q-\mathrm{sg}[u]\rVert_2^2, ]

where (\beta=0.25) in the reported experiments.

Factorized Codebook

A simple 256-entry codebook would give only 256 choices per quantized vector. UniDexTok instead factorizes the vocabulary across channel groups. The encoder output is projected from 512 to 256 dimensions, divided into (K=8) groups, and each group uses a 32-entry sub-codebook. That means one token can express (32^8) code combinations while storing only (32 \times 8 = 256) learned code vectors.

For each token position and group, the nearest normalized code vector is selected by cosine similarity:

[ i_{n,k}=\arg\max_j \left\langle \frac{u_{n,k}}{\lVert u_{n,k}\rVert_2}, \frac{e_{k,j}}{\lVert e_{k,j}\rVert_2} \right\rangle. ]

This design is important for discrete representation quality. In the paper’s gesture benchmark, UniHM’s single-codebook quantized features drop from 96.15% to 84.62% linear probing accuracy, while UniDexTok keeps 100% accuracy after quantization. The interpretation is straightforward: a single codebook can collapse different gestures into the same discrete code; factorized VQ gives the token more compositional capacity.

Data and Evaluation

The training data combines two main sources. The first is coarse human hand-object interaction data from DexYCB, OakInk-v2, and EgoDex. UniDexTok does not retarget those motions to a robot hand. It reduces the MANO-style 45-DoF hand representation to the 22 active UDHM coordinates and trains on the standardized human hand as another embodiment.

The second source is real robot-hand data from LET, Dexora, and LinkerHand. These datasets differ in joint ordering, dimensionality, and units, so UniDexTok converts all values to radians, inserts available DoFs into the corresponding UDHM coordinates, and pads missing coordinates.

The paper evaluates reconstruction in joint-angle space and Cartesian space. MPJAE measures mean per-joint angle error in degrees. MPJPE measures mean per-joint position error in millimeters. FK error reports fingertip-position errors.

Main Results

Against UniHM on the standardized real datasets, UniDexTok reduces average MPJAE from 15.63 degrees to 0.16 degrees and average MPJPE from 18.51 mm to 0.18 mm. That is the paper’s headline result: reconstruction improves from centimeter-scale error to sub-millimeter error.

The improvement is consistent across LinkerHand L6, LinkerHand L10, LinkerHand L20, and Robotera XHand1. On the retargeted DexYCB evaluation, the gap is smaller because that protocol is closer to the retargeted data distribution used by UniHM, but UniDexTok still improves average MPJAE from 4.40 degrees to 2.83 degrees.

The zero-shot and few-shot results are also useful. On an unseen Inspire RH56E2 hand, zero-shot per-joint angle errors range from 4.14 to 7.85 degrees, and fingertip FK errors range from 7.73 to 16.05 mm. With only 4,528 frames, about 6.2% of the full Inspire dataset, fine-tuning for 2 epochs reduces joint errors to roughly 1.42 to 2.08 degrees and reduces fingertip errors by 58.5% to 78.8%.

This is the evidence for treating the tokenizer as a cross-embodiment representation, beyond a compression model for one hand. A new hand can enter the same token space and then adapt with a small amount of target data.

Ablations

The ablation on human-hand data is especially relevant for embodied datasets. With human-hand data included, the average MPJAE is 0.16 degrees and MPJPE is 0.18 mm. Without human-hand data, they rise to 0.37 degrees and 0.43 mm. The human hand is therefore useful as a training embodiment, not merely as a source for retargeting.

The semantic-insertion ablation supports the UDHM design. With semantic insertion, MPJAE, MPJPE, and FK error are 0.24 degrees, 0.25 mm, and 0.53 mm. Without semantic insertion, they degrade to 0.53 degrees, 0.57 mm, and 1.20 mm. The benefit comes from aligning coordinates by joint meaning.

Strengths and Limitations

The strength of UniDexTok is that it attacks the representation bottleneck directly. It gives heterogeneous hand datasets a common joint-semantic interface, learns discrete tokens from real standardized states, and avoids making retargeted trajectories the primary source of robot-hand state data. This is a good fit for future dexterous VLA systems, where an action or state token needs a stable meaning across hardware.

The limitations are also clear. UDHM assumes a human-hand-like kinematic structure, so non-anthropomorphic grippers, soft hands, heavily underactuated hands, tendon coupling, compliance, and actuator-level dynamics are only partially captured. The tokenizer focuses on state reconstruction and does not model contacts, tactile signals, object geometry, force, or temporal action dynamics. Low reconstruction error is valuable, but it does not by itself prove better downstream manipulation.

Takeaway

UniDexTok is best understood as infrastructure for dexterous foundation models. It leaves manipulation policy learning to the next layer and solves a lower-level but very important problem: make heterogeneous real hand states share one semantic, discrete, embodiment-conditioned token space.

The larger lesson is that cross-embodiment learning needs more than a big model and more data. It needs careful state standardization. Once the joint semantics are aligned, human data, robot data, seen hands, and new hands can contribute to the same representation without living in isolated codebooks.

UniDexTok 研究的是灵巧操作策略学习之前的一个表示问题：不同 human hands 和 robot hands 的数据，怎样放进同一个状态空间，再变成未来 VLA 或 world-action models 可以使用的离散 tokens？

我的理解：这篇的核心贡献是一个 retargeting-free 的状态 tokenizer。它没有先把所有手部轨迹通过 MANO-to-robot retargeting 转成某一种目标机器人手，而是先用 UDHM 把人手和机器人手状态标准化到一个 22-DoF semantic interface，然后训练一个跨 embodiment 共享 encoder、codebook 和 decoder 的 conditional tokenizer。关键变化是从“让所有手模仿某一个机器人手”转向“让所有手使用同一种关节语义语言”。

Paper Info

论文是 “UniDexTok: A Unified Dexterous Hand Tokenizer from Real Data”，作者为 Dong Fang, Youjun Wu, Yuanxin Zhong, Rui Zhang, Yunlong Wang, Xiaosong Jia, and Yu-Gang Jiang，来自 Fudan University、Rimbot、Hefei University of Technology 和 Beijing University of Posts and Telecommunications。论文链接是 arXiv:2606.10683。

为什么 Dexterous Hand State 难共享

Parallel gripper 很容易表示：开合宽度、夹爪位姿，也许再加一个 open-close binary state。Dexterous hand 就复杂很多。不同灵巧手有不同手指数、关节名、关节顺序、耦合关节、控制范围和单位。甚至同一类手采集的数据，也可能使用不同记录约定，包括 radians、degrees 或离散 bins。

这会让 cross-embodiment learning 很别扭。如果每个机器人手都有单独 tokenizer，每个 codebook 就会学习自己的 private latent space。模型也许可以重建每只手，但离散 token 在不同硬件之间没有稳定含义。如果 pipeline 依赖把 human motion retarget 到每只 robot hand，表示就会继承 retargeting 带来的几何误差，也会丢掉真实机器人数据原本的 state statistics。

UniDexTok 的回答是：先统一 state space，再做 tokenization。

UDHM：22-DoF Semantic Interface

Unified Dexterous Hand Model，也就是 UDHM，是这篇的共享坐标系。它假设一种 human-hand-like kinematic structure，把 MANO-style human keypoints 和 robot hand joints 映射到 22 维 active joint interface。

对于 index、middle 和 ring fingers，UDHM 每根手指使用四个 active coordinates：MCP abduction/adduction、MCP flexion/extension、PIP flexion/extension 和 DIP flexion/extension。Pinky 额外增加一个 twist DoF，用来覆盖更大的侧向变化。Thumb 使用五个坐标：CMC flexion、CMC spread、MCP flexion、MCP abduction 和 IP flexion。合起来是 22 个 active coordinates。

模型会根据 wrist 和 MCP keypoints 拟合 palm plane，用 palm normal 和 local lateral axes 定义 motion planes，再用解析 forward kinematics 重建每根手指链。做 inverse mapping 时，它从输入帧中提取 palm offsets 和 bone lengths，然后求解 nonlinear least-squares，使重建关节尽量匹配目标 keypoints。

我觉得很实用的一点是它处理异构 robot logs 的方式。每个可用的 robot-hand DoF 会被插入到语义对应的 UDHM coordinate，缺失坐标用 0 padding。这个 semantic insertion 比简单地把所有可用坐标 append 到向量前面更强，因为模型能学到某一维表示的是“index MCP pitch”，而不只是记住某个数据源里的列号。

所有原始关节角都会转换成 radians，并用固定尺度 (\pi) 归一化。固定归一化避免 token 依赖某个 train-test split 的统计量，也让 denormalization 之后的 MPJAE 仍然可解释。

UniDexTok Architecture

给定一个标准化手部状态 (x \in \mathbb{R}^{22})，以及 hand-type 或 source label (h)，tokenizer 学习：

[ \hat{x} = \pi D(Q(E(\tilde{x}, h_{\mathrm{embed}})), h_{\mathrm{embed}}), \quad \tilde{x}=x/\pi. ]

Encoder 先把 22 维 normalized state 投影成 (N=8) 个 latent tokens，每个 token 宽度 (C=512)。Transformer blocks 再建模不同关节坐标之间的相关性。Hand-type conditioning 通过 adaptive layer normalization 注入：

[ \mathrm{AdaLN}(z,h_{\mathrm{embed}})=\gamma(c_h)\odot \mathrm{LN}(z)+\beta(c_h). ]

这个 conditioning 很关键，因为同一种语义姿态在不同手上可能对应不同数值关节模式。共享 encoder 和 decoder 学 cross-hand structure，conditioning 保留硬件特定约定。

Decoder 结构和 encoder 对称。它接收 quantized tokens，加上 positional embeddings 和 hand conditioning，然后重建 normalized 22-dimensional state。重建目标基于 normalized joint-angle reconstruction，完整目标再加 VQ commitment loss：

[ L = L_{\mathrm{rec}} + L_{\mathrm{vq}}, ]

其中

[ L_{\mathrm{vq}}=\beta\lVert \mathrm{sg}[q]-u\rVert_2^2+\lVert q-\mathrm{sg}[u]\rVert_2^2, ]

实验中 (\beta=0.25)。

Factorized Codebook

普通 256-entry codebook 对每个 quantized vector 只有 256 种选择。UniDexTok 把 vocabulary 沿 channel groups 分解。Encoder output 从 512 维投影到 256 维，然后切成 (K=8) 个 groups，每个 group 使用 32-entry sub-codebook。这样一个 token 可以表达 (32^8) 种 code combinations，但只需要存储 (32 \times 8 = 256) 个 learned code vectors。

对于每个 token position 和 group，quantizer 通过 cosine similarity 选择最近的 normalized code vector：

[ i_{n,k}=\arg\max_j \left\langle \frac{u_{n,k}}{\lVert u_{n,k}\rVert_2}, \frac{e_{k,j}}{\lVert e_{k,j}\rVert_2} \right\rangle. ]

这个设计直接影响离散表示质量。在论文的 gesture benchmark 中，UniHM 的 single-codebook quantized features 让 linear probing accuracy 从 96.15% 掉到 84.62%；UniDexTok 在 quantization 之后仍然保持 100%。直观解释是：single codebook 容易把不同 gesture collapse 到同一个 discrete code；factorized VQ 给 token 更强的组合容量。

数据与评估

训练数据主要有两类。第一类是 DexYCB、OakInk-v2 和 EgoDex 中的 coarse human hand-object interaction data。UniDexTok 没有把这些 motion retarget 到机器人手，而是把 MANO-style 45-DoF hand representation 降到 22 个 active UDHM coordinates，把标准化后的人手当成另一个 embodiment 来训练。

第二类是真实 robot-hand data，包括 LET、Dexora 和 LinkerHand。这些数据集在 joint ordering、dimensionality 和 units 上都不同，所以 UniDexTok 先把所有值转换为 radians，再把可用 DoF 插入对应 UDHM coordinates，并 padding 缺失坐标。

论文在 joint-angle space 和 Cartesian space 两个层面评估重建质量。MPJAE 是 mean per-joint angle error，单位是 degrees。MPJPE 是 mean per-joint position error，单位是 millimeters。FK error 报告 fingertip-position errors。

主要结果

在标准化真实数据集上，相比 UniHM，UniDexTok 把平均 MPJAE 从 15.63 degrees 降到 0.16 degrees，把平均 MPJPE 从 18.51 mm 降到 0.18 mm。这是这篇最核心的结果：重建误差从厘米级降到了亚毫米级。

这个提升在 LinkerHand L6、LinkerHand L10、LinkerHand L20 和 Robotera XHand1 上都成立。在 retargeted DexYCB evaluation 上，差距小一些，因为这个协议更接近 UniHM 使用的 retargeted data distribution，但 UniDexTok 仍然把平均 MPJAE 从 4.40 degrees 降到 2.83 degrees。

Zero-shot 和 few-shot 结果也很有价值。对于训练中没见过的 Inspire RH56E2 hand，zero-shot per-joint angle errors 在 4.14 到 7.85 degrees 之间，fingertip FK errors 在 7.73 到 16.05 mm 之间。只用 4,528 frames，也就是完整 Inspire dataset 的 6.2%，fine-tuning 2 个 epochs 后，joint errors 降到大约 1.42 到 2.08 degrees，fingertip errors 降低 58.5% 到 78.8%。

这说明 tokenizer 更像一种 cross-embodiment representation，而不只是某只手的压缩模型。新手型可以进入同一个 token space，再用少量目标数据适配。

Ablations

Human-hand data 的 ablation 对 embodied datasets 很有启发。加入 human-hand data 时，平均 MPJAE 是 0.16 degrees，MPJPE 是 0.18 mm。去掉 human-hand data 后，它们升到 0.37 degrees 和 0.43 mm。所以人手在这里是有效的 training embodiment，不只是 retargeting source。

Semantic-insertion ablation 支持了 UDHM 设计。使用 semantic insertion 时，MPJAE、MPJPE 和 FK error 分别是 0.24 degrees、0.25 mm 和 0.53 mm。不使用 semantic insertion 时，三者退化到 0.53 degrees、0.57 mm 和 1.20 mm。收益来自按关节语义对齐坐标。

优点与限制

UniDexTok 的优点是直接处理 representation bottleneck。它给异构手部数据集建立共同的 joint-semantic interface，从标准化真实状态中学习 discrete tokens，并避免把 retargeted trajectories 当作机器人手状态数据的主要来源。这很适合未来 dexterous VLA 系统，因为 action 或 state token 需要在不同硬件上有稳定含义。

限制也很清楚。UDHM 假设 human-hand-like kinematic structure，所以 non-anthropomorphic grippers、soft hands、强 underactuated hands、tendon coupling、compliance 和 actuator-level dynamics 都只能部分表达。当前 tokenizer 关注 state reconstruction，没有显式建模 contacts、tactile signals、object geometry、force 或 temporal action dynamics。低重建误差很有价值，但它本身还不能证明下游 manipulation 一定更好。

Takeaway

UniDexTok 最适合理解为 dexterous foundation models 的基础设施。它把 manipulation policy learning 留给下一层，自己解决一个更底层但很关键的问题：让异构真实手部状态共享一个有语义、离散化、带 embodiment conditioning 的 token space。

更大的启发是，cross-embodiment learning 需要的不只是更大的模型和更多数据，还需要仔细的 state standardization。一旦关节语义对齐，人手数据、机器人数据、已见手型和新手型就能共同服务于同一个 representation，而不是散落在彼此隔离的 codebooks 里。

[Paper Notes] CHORD: Contact Wrench Guidance From Human Demonstration

2026-07-02T00:00:00+00:00

CHORD studies a central problem in dexterous robot learning: human demonstrations contain rich manipulation knowledge, yet robot hands cannot directly replay human hand motion. Different morphology, kinematics, fingertip geometry, and contact timing can turn the same-looking motion into a different object effect.

My read: the paper’s useful abstraction is object-centric contact wrench guidance. CHORD asks the robot to reproduce the object-level mechanical effect of a human contact, measured by the force-torque directions that contacts can induce on the object. This gives reinforcement learning a contact-rich reward that is less tied to a specific hand shape than raw contact positions.

Paper Info

The paper is “Learning Dexterous Manipulation Using Contact Wrench Guidance From Human Demonstration” by Xinghao Zhu, Zixi Liu, Shalin Jain, Chenran Li, Milad Noori, Huihua Zhao, John Welsh, Michael Andres Lin, Wei Liu, Tingwu Wang, Xingye Da, Zhengyi Luo, Vishal Kulkarni, Naema Bhatti, Yuke Zhu, Linxi Fan, Bowen Wen, Danfei Xu, Soha Pouya, and Yan Chang from NVIDIA. The project page is nvidia-isaac.github.io/video_to_data/chord, and the technical report PDF is linked there.

CHORD stands for Contact Wrench Guidance from Human Demonstration in Robotic Dexterous Manipulation. The reported benchmark contains 4,739 bimanual dexterous manipulation tasks, and the main evaluation covers 1,831 tasks. The paper reports 82.12% average success on this large-scale simulation evaluation, 90.77% success on whole-body manipulation tasks, and successful transfer to a real Dexmate robot with two Sharpa dexterous hands.

Why Human Motion Is Not Enough

A human demonstration gives at least three signals: hand motion, object motion, and hand-object contact. Purely tracking object motion is often too sparse for reinforcement learning because the robot receives useful feedback only after making the right contact. Purely tracking hand motion is brittle because the robot hand may need a different pose to generate the same effect. Tracking contact locations helps, yet it can still be mechanically wrong: touching the same object patch with a different surface normal or force direction may push, lever, or slide the object in the wrong way.

CHORD’s answer is to move the comparison into wrench space. A contact is judged by the force and torque directions it can apply to an object part. This makes the target more functional: the robot can choose embodiment-specific contacts as long as those contacts support object motion aligned with the human reference.

The Core Formulation

Each reference trajectory is written as

[ \tau^{\mathrm{ref}}={x_t^{\mathrm{human}}, x_t^{\mathrm{object}}}_{t=1}^{H}, ]

where the human term contains 3D hand keypoints and the object term contains the poses of (K) rigid bodies or articulated parts. The human hand motion is first retargeted by inverse kinematics to a robot reference (x_t^{\mathrm{robot}}). The policy then receives robot observations, object observations, and reference motion, and learns actions that make the rollout object poses track the reference.

The reward has three parts:

[ r = r_{\mathrm{task}} + r_{\mathrm{imit}} + r_{\mathrm{contact}}. ]

The task term tracks object-part poses. For multi-object interactions such as insertion, pouring, scooping, and tool use, CHORD also adds a relative object-pose reward that activates during demonstrated object-object interaction phases. The imitation term keeps the robot near the retargeted motion. The contact term is the main contribution.

For object part (k), CHORD extracts human contact positions (p_{h,k}^{i}) and normals (n_{h,k}^{i}) in the object frame. It approximates each Coulomb friction cone with (d) edge forces. Each edge force creates a primitive wrench:

[ w_{h,k}^{i,j} = \begin{bmatrix} f_{h,k}^{i,j}
p_{h,k}^{i} \times f_{h,k}^{i,j} \end{bmatrix} \in \mathbb{R}^{6}. ]

Collecting all primitive wrenches gives the human wrench matrix:

[ \mathcal{W}{h,k}\in \mathbb{R}^{6\times(c{h,k}d)}. ]

Directly comparing human and robot wrench matrices is awkward because the two hands may have different contact counts and different ordering of contact primitives. CHORD therefore compares the geometry of the induced wrench set through a support function. With pre-sampled unit basis directions (\mathcal{B}\in\mathbb{R}^{6\times b}),

[ \sigma_{h,k} = \max_{\mathrm{col}}(\mathcal{B}^{\top}\mathcal{W}_{h,k})\in\mathbb{R}^{b}, ]

and (\sigma_{r,k}) is computed the same way for the robot. The contact wrench-space reward then keeps the robot support close to the human support under a relative tolerance (\beta):

[ r_{\mathrm{cws}}^{k} = \exp\left( -\frac{\lVert\max(0,(1-\beta)\sigma_{h,k}-\sigma_{r,k})\rVert_2^2}{v_{\mathrm{cws}}} -\frac{\lVert\max(0,\sigma_{r,k}-(1+\beta)\sigma_{h,k})\rVert_2^2}{v_{\mathrm{cws}}} \right). ]

The lower-bound term encourages the robot to supply enough support in the same wrench directions as the human contact. The upper-bound term prevents excessive or mechanically unrelated support. The paper also adds penalties for unintended contacts and missed contacts because the exponential kernel alone would still give a positive value for mismatched contact states.

Training Recipe

CHORD still uses practical scaffolding around the reward. It uses a virtual object controller (VOC) during training, resets the simulator to arbitrary frames along the reference trajectory, keeps VOC active for a short stabilization window, and anneals assistance through a curriculum. The policy uses residual actions around the retargeted robot motion, so RL learns corrections instead of rediscovering the whole movement from scratch.

The paper adds two robustness choices. First, object parts are perturbed by wrenches sampled from the human contact wrench matrix, which creates disturbances aligned with the demonstrated contact mechanics. Second, when contact estimates are noisy, CHORD can switch from exact wrench matching to a reduced force-closure-style objective:

[ r_{\mathrm{fc}}^{k}=\frac{1}{B}\sum_{b=1}^{B}\mathbf{1}[\sigma_{r,k,b}>\epsilon]. ]

That fallback is less behavior-specific, but it avoids trusting corrupted contact normals and positions from noisy video reconstruction.

Benchmark and Results

The benchmark is a large part of the contribution. CHORD processes human hand-object interaction data from ARCTIC, TACO, HOT3D, OakInk2, DexYCB, GRAB, and H2O, then imports the retargeted tasks into Isaac Lab. The resulting library includes rigid objects, articulated objects, multi-object interactions, long horizons, and dense contact events.

On 1,831 sampled tasks, CHORD reports an average success rate of 82.12% with one shared training recipe across tasks. Compared with prior methods under their own evaluation protocols, CHORD matches or exceeds DexMachina, ManipTrans, and SPIDER across the tested suites. The contact reward ablations are especially clear:

Setting	Main finding
CHORD contact wrench support	Best performance on box grabbing, mixer use, and whole-body ablations
Contact position reward	Weaker when the same object patch does not imply the same force/torque effect
No contact reward	Weakest on contact-rich manipulation because kinematic tracking alone gives poor exploration guidance

The paper also reports that normalized contact wrench reward correlates strongly with task success across 1,831 runs, with Pearson correlation around 0.80. That is an important empirical check: the proposed reward is not just elegant mechanics; it tracks downstream manipulation performance.

For whole-body manipulation, CHORD handles both hand-only references and third-person whole-body references. Hand-only references can be expanded with a motion inpainting module that predicts whole-body motion from end-effector trajectories. For noisy third-person reconstructions, the reduced force-closure objective is used. The whole-body evaluation reports 90.77% success, and a cross-embodiment ablation shows a large gap between wrench guidance and contact position guidance when transferring five-finger human contacts to the three-fingered Dex3 hand on a Unitree G1.

Real-World Deployment

The real-world experiments use a Dexmate robot with two Sharpa dexterous hands. The paper tests both open-loop action-chunk execution and closed-loop inference. Object and robot poses are tracked with a six-camera Vicon setup, and the deployed policy runs as an ONNX model at 20 Hz while lower-level bridges hold commands at higher frequency.

This is a state-based real-world transfer result, not a vision policy. That matters for interpreting the result: the paper shows that the learned contact behavior can leave simulation, while visual perception remains future work.

Strengths and Limitations

The strongest part of CHORD is the abstraction boundary. It does not ask morphology-specific robot hands to copy human contact positions. It asks them to create contacts with similar object-level wrench support. That is a better match for dexterous manipulation because functional contact matters more than visual similarity of the hand pose.

The second strength is scale. The paper moves beyond a small set of hand-picked manipulation tasks and evaluates RL-based dexterous manipulation over thousands of long-horizon bimanual tasks. This makes the reward design feel less like a single-task trick and more like a reusable training signal.

The limitations are also concrete. Real-world deployment currently uses state-based observations from motion capture. Accurate contact guidance still depends on reasonably clean demonstrations; when contact positions or normals become very noisy, the method falls back to a less specific force-closure objective. The evaluation metric is object-pose tracking, which may miss task functionality in both directions: a small pose error can break an insertion task, while a larger pose error may be acceptable for a loose placement task.

Takeaway

CHORD’s reusable idea is: transfer human dexterous demonstrations through the mechanics of contact, not through the appearance of contact. Represent each human and robot contact set by the wrench directions it can induce on the object, compare those supports with tolerance, and use the result as a dense RL reward. For dexterous grasping and manipulation, this is a powerful shift because it respects embodiment differences while preserving the part of the demonstration that actually moves the object.

CHORD 研究的是灵巧手学习里的一个核心问题：人类示范里有大量操作知识，但机器人手不能直接复放人手动作。手型、运动学、指尖几何、接触时机一变，看起来相似的动作就可能给物体带来完全不同的效果。

我的理解是：这篇论文最值得抓住的抽象是 object-centric contact wrench guidance。CHORD 让机器人复现人类接触对物体产生的力学效果，用接触能在物体上诱导出的 force/torque directions 来衡量。这样 RL 得到的是一个接触丰富、又不过度绑定具体手型的奖励。

论文信息

论文标题是 “Learning Dexterous Manipulation Using Contact Wrench Guidance From Human Demonstration”，作者包括 NVIDIA 的 Xinghao Zhu, Zixi Liu, Shalin Jain, Chenran Li, Milad Noori, Huihua Zhao, John Welsh, Michael Andres Lin, Wei Liu, Tingwu Wang, Xingye Da, Zhengyi Luo, Vishal Kulkarni, Naema Bhatti, Yuke Zhu, Linxi Fan, Bowen Wen, Danfei Xu, Soha Pouya, and Yan Chang。项目页是 nvidia-isaac.github.io/video_to_data/chord，技术报告 PDF 也在项目页中。

CHORD 的全称是 Contact Wrench Guidance from Human Demonstration in Robotic Dexterous Manipulation。论文报告的 benchmark 包含 4,739 个双手灵巧操作任务，主实验评估了 1,831 个任务。结果中，大规模仿真平均成功率是 82.12%，whole-body manipulation 成功率是 90.77%，并且策略可以迁移到配备两只 Sharpa 灵巧手的 Dexmate 真实机器人上。

为什么只模仿人手动作不够

人类示范至少给出三类信号：手的运动、物体的运动、手和物体的接触。只跟踪物体运动对 RL 来说经常太稀疏，因为机器人只有先碰对了，后续物体轨迹奖励才有意义。只跟踪手的运动也很脆，因为机器人手可能需要用不同姿态才能产生同样的物体效果。跟踪接触位置会更好一些，但仍然可能在力学上错掉：接触同一个物体区域，如果表面法向或施力方向不同，物体可能被推、撬、滑向完全不同的方向。

CHORD 的做法是把比较对象搬到 wrench space。接触根据它能对物体部件施加什么 force 和 torque 来评估。这个目标更偏功能性：机器人可以选择适合自身 embodiment 的接触，只要这些接触支持的物体运动和人类参考一致。

核心方法

每条参考轨迹写成：

[ \tau^{\mathrm{ref}}={x_t^{\mathrm{human}}, x_t^{\mathrm{object}}}_{t=1}^{H}, ]

其中 human term 是 3D hand keypoints，object term 是 (K) 个 rigid bodies 或 articulated parts 的 pose。论文先用 IK 把人手动作 retarget 到机器人参考 (x_t^{\mathrm{robot}})。策略随后接收 robot observations、object observations 和 reference motion，学习让 rollout 中的 object poses 跟随参考轨迹。

总奖励有三部分：

[ r = r_{\mathrm{task}} + r_{\mathrm{imit}} + r_{\mathrm{contact}}. ]

task term 跟踪 object-part poses。对于 insertion、pouring、scooping、tool use 这类 multi-object interaction，CHORD 还会在示范中发生物体-物体交互的阶段加入 relative object-pose reward。imitation term 让机器人不要偏离 retargeted motion。真正的贡献在 contact term。

对 object part (k)，CHORD 在物体坐标系下提取人类接触位置 (p_{h,k}^{i}) 和法向 (n_{h,k}^{i})。每个 Coulomb friction cone 被近似成 (d) 条 edge forces。每条 edge force 会产生一个 primitive wrench：

[ w_{h,k}^{i,j} = \begin{bmatrix} f_{h,k}^{i,j}
p_{h,k}^{i} \times f_{h,k}^{i,j} \end{bmatrix} \in \mathbb{R}^{6}. ]

把所有 primitive wrenches 收集起来，就得到 human wrench matrix：

[ \mathcal{W}{h,k}\in \mathbb{R}^{6\times(c{h,k}d)}. ]

直接比较 human 和 robot 的 wrench matrices 很麻烦，因为两只手的接触数量可能不同，primitive wrench 的顺序也没有可比性。CHORD 用 support function 比较诱导出的 wrench geometry。给定预采样单位方向 (\mathcal{B}\in\mathbb{R}^{6\times b})，有：

[ \sigma_{h,k} = \max_{\mathrm{col}}(\mathcal{B}^{\top}\mathcal{W}_{h,k})\in\mathbb{R}^{b}, ]

机器人侧的 (\sigma_{r,k}) 用同样方式计算。contact wrench-space reward 用相对容差 (\beta) 约束机器人 support 接近人类 support：

lower-bound term 鼓励机器人在和人类接触相同的 wrench directions 上提供足够支撑。upper-bound term 防止机器人产生过强或不相关的 wrench support。论文还额外惩罚 unintended contacts 和 missed contacts，因为指数核即使在接触状态不匹配时也会给正奖励。

训练配方

CHORD 的 reward 之外还有一些很实用的训练支架。训练中使用 virtual object controller (VOC)，从参考轨迹中的任意帧 reset simulator，并在短暂稳定窗口内保持 VOC fully active，随后通过 curriculum 逐步 anneal assistance。策略使用 residual action space，以 retargeted robot motion 作为 prior，让 RL 学 correction，避免从零重新发现整段动作。

论文还有两个鲁棒性设计。第一，用 human contact wrench matrix 中采样的 wrenches 去扰动物体部件，这些扰动和示范中的接触力学一致。第二，当接触估计很噪时，CHORD 可以从精确的 wrench matching 切换到 reduced force-closure-style objective：

[ r_{\mathrm{fc}}^{k}=\frac{1}{B}\sum_{b=1}^{B}\mathbf{1}[\sigma_{r,k,b}>\epsilon]. ]

这个 fallback 的行为约束更弱，但可以避免过度相信由 noisy video reconstruction 得到的错误接触法向和位置。

Benchmark 和结果

benchmark 本身是这篇论文的重要贡献。CHORD 处理了来自 ARCTIC, TACO, HOT3D, OakInk2, DexYCB, GRAB, H2O 的人手-物体交互数据，再把 retargeted tasks 导入 Isaac Lab。最终任务库包含 rigid objects、articulated objects、multi-object interactions、长时间 horizon 和密集 contact events。

在 1,831 个采样任务上，CHORD 用一套共享训练配方达到 82.12% 平均成功率。和 prior methods 比较时，论文在各自原始 evaluation protocols 下复现并比较 DexMachina、ManipTrans 和 SPIDER，CHORD 在测试套件上整体持平或超过这些方法。contact reward 的 ablation 特别清楚：

Setting	Main finding
CHORD contact wrench support	在 box grabbing、mixer use 和 whole-body ablation 上表现最好
Contact position reward	同一个 object patch 不一定对应同样的 force/torque effect，因此性能明显变弱
No contact reward	对接触丰富任务最弱，因为单纯 kinematic tracking 给不了足够好的 exploration guidance

论文还显示，在 1,831 次 runs 中，normalized contact wrench reward 和 task success 的 Pearson correlation 约为 0.80。这点很关键：这个 reward 的价值同时体现在力学表达和最终操作成功的相关性上。

在 whole-body manipulation 上，CHORD 同时处理 hand-only references 和第三人称 whole-body references。hand-only references 可以通过 motion inpainting module 从 end-effector trajectories 补出 whole-body motion。对于 noisy third-person reconstructions，则使用 reduced force-closure objective。whole-body 实验报告 90.77% 成功率；cross-embodiment ablation 也显示，把五指人手接触迁移到 Unitree G1 的三指 Dex3 手时，wrench guidance 明显优于 contact position guidance。

真实机器人部署

真实实验使用 Dexmate robot 和两只 Sharpa 灵巧手。论文测试了 open-loop action-chunk execution 和 closed-loop inference。物体和机器人 pose 由六相机 Vicon 系统跟踪，部署策略以 ONNX model 运行，policy loop 频率是 20 Hz，底层 bridge 以更高频率 hold commands。

这是一组 state-based real-world transfer 结果，vision policy 还没有覆盖。解读时需要注意：论文证明了学到的接触行为可以离开仿真，但视觉感知部署仍然是未来工作。

优点和局限

CHORD 最强的地方是抽象边界选得好。它避免让不同形态的机器人手复制人手接触位置，改为要求机器人产生相似的 object-level wrench support。对灵巧操作来说，这比手势外观相似更贴近真正的功能目标。

第二个优点是规模。论文把 RL-based dexterous manipulation 的评估推进到数千个 long-horizon bimanual tasks，规模不再停留在少量精心挑选的任务。这让 reward 设计看起来更像一个可复用训练信号，而非单任务技巧。

局限也很具体。真实部署目前依赖 motion capture 提供 state-based observations。准确的 contact guidance 仍然需要相对干净的人类示范；当接触位置或法向噪声很大时，方法会退回到更粗的 force-closure objective。评估指标主要是 object-pose tracking，而 object pose error 并非完美的任务成功代理：插入任务里很小的 pose error 可能导致功能失败，宽松放置任务里较大的 pose error 也可能仍然可接受。

Takeaway

CHORD 最值得复用的思想是：用接触力学迁移人类灵巧示范，让接触外观退到次要位置。把人类和机器人接触集合都表示成它们能在物体上诱导出的 wrench directions，用带容差的 support function 比较，再把这个比较结果作为 dense RL reward。对于灵巧抓取和操作，这个转变很有力量，因为它允许 embodiment 差异，同时保留了示范中真正推动物体运动的那部分信息。

[Paper Notes] DexWorldModel: Causal Latent World Modeling towards Automated Learning of Embodied Tasks

2026-07-02T00:00:00+00:00

DexWorldModel proposes a Causal Latent World Model (CLWM) for manipulation. Its main design choice is to move world-action generation away from pixel reconstruction and into a semantic latent space: predict future DINOv3 features, then decode action chunks conditioned on those predicted future semantics.

My read: the paper is about making world-action models deployable, not just imaginative. CLWM targets three bottlenecks at once: pixel-level prediction wastes capacity on visual texture, autoregressive history makes memory grow with horizon, and diffusion-style generation slows closed-loop control. The proposed system answers these with DINOv3 latent targets, Dual-State TTT Memory, and Speculative Asynchronous Inference.

Paper Info

The paper is “DexWorldModel: Causal Latent World Modeling towards Automated Learning of Embodied Tasks” by Yueci Deng, Guiliang Liu, and Kui Jia from DexForce AI. It appears on arXiv as arXiv:2604.16484, submitted on April 13, 2026.

The paper also introduces EmbodiChain as the training-data engine behind CLWM’s post-training stage. The referenced platform is github.com/DexForce/EmbodiChain.

Core Problem

World Action Models (WAMs) extend VLA policies by generating future world states before generating actions. A common formulation is:

[ \hat{o}{t+1}\sim p\theta(\cdot\mid o_{\le t},a_{<t},l), ]

[ a_t\sim g_\psi(\cdot\mid o_{\le t},a_{<t},\hat{o}_{t+1},l). ]

This gives the policy an explicit future-state interface. The cost is heavy: if (\hat{o}_{t+1}) lives in pixel space or low-level VAE latent space, the model spends capacity on lighting, texture, and background details. If the model keeps all history through a Transformer KV cache, memory scales as (O(T)). If each action waits for sensing, full denoising, and execution in sequence, the robot pays a large latency penalty at every control chunk.

DexWorldModel’s thesis is that a manipulation world model should predict the part of the future that matters for action: interaction semantics.

CLWM Architecture

CLWM has two coupled generative modules under a Mixture of Transformers (MoT) design:

Module	Target	Role
Latent Video Model	future DINOv3 feature (f_{t+1})	predicts semantic world evolution
Action Model	action chunk (a_t)	decodes control from predicted future semantics

The visual feature is extracted by a frozen DINOv3 model:

[ f_t=\Phi_{\mathrm{DINO}}(o_t)\in\mathbb{R}^{C\times H’\times W’}. ]

The paper uses patch size (P=16). The two generators share a core Transformer initialized from Wan2.2-5B, while keeping separate input/output projections and flow-time embeddings:

[ \phi_{\mathrm{vid}}=\phi^{out}{\mathrm{vid}}\circ\phi{\mathrm{share}}\circ\phi^{in}{\mathrm{vid}}, \quad \phi{\mathrm{act}}=\phi^{out}{\mathrm{act}}\circ\phi{\mathrm{share}}\circ\phi^{in}_{\mathrm{act}}. ]

Generation is autoregressive and causal. First, the Latent Video Model predicts the next semantic feature via conditional flow matching:

[ \mathcal{L}{\mathrm{video}} = \mathbb{E}\left[ \left\lVert v{\phi_{\mathrm{vid}}}(f^{(s)}{t+1},s\mid h{\le t},l) - \dot{f}^{(s)}_{t+1} \right\rVert^2 \right]. ]

Then the Action Model predicts an action chunk:

[ a_t={a_{t,1},a_{t,2},\ldots,a_{t,\tau}}, ]

with (\tau=16) in the experiments. The action vector is compact for dual-arm control: each arm contributes a 7-DoF end-effector pose, 7 joint positions, and 1 gripper state, giving ((7+7+1)\times 2=30) continuous dimensions.

The action loss is also a flow-matching objective, conditioned on history, language, and the predicted future feature:

[ \mathcal{L}{\mathrm{action}} = \mathbb{E}\left[ \left\lVert v{\phi_{\mathrm{act}}}(a^{(s)}t,s\mid \tilde{h}{\le t},l,\tilde{f}_{t+1}) - \dot{a}^{(s)}_t \right\rVert^2 \right]. ]

A small but important detail is history augmentation. During training, with probability (p=0.5), the model injects Gaussian noise into historical latent features:

[ \tilde{f}{\le t}=(1-s{\mathrm{aug}})\epsilon+s_{\mathrm{aug}}f_{\le t}, \quad s_{\mathrm{aug}}\in[0.5,1]. ]

This teaches the action model to tolerate imperfect predicted histories. That matters later because SAI deliberately starts inference from speculative future features before the real observation arrives.

Dual-State TTT Memory

Standard autoregressive world models preserve history through KV cache. For long-horizon robot interaction, that creates a growing memory and latency burden:

[ \text{KV cache memory}\sim O(T). ]

CLWM replaces the cache with a Test-Time Training (TTT) Memory layer. Instead of storing all historical tokens, the model compresses history into dynamically updated layer weights. The TTT-MLP is trained through a self-supervised reconstruction objective:

[ \ell_{\mathrm{self}}(W;z_t) = \lVert f(\theta_K z_t;W)-\theta_V z_t\rVert^2. ]

After the inner-loop weights update to (W_t), the query projection extracts the hidden state:

[ l_t=f_{\mathrm{TTTmlp}}(\theta_Q z_t;W_t). ]

The paper wraps this in a Dual-State design:

Memory	Updated by	Purpose
Long-Term TTT Memory	real observations and executed actions	anchors true physical history
Working TTT Memory	forked copy plus predicted future features	conditions short-term generation

The long-term memory update is:

[ W^{long}t = W^{long}{t-1} - \eta\nabla_W\ell_{\mathrm{self}}(W^{long}_{t-1};h_t). ]

During generation, CLWM forks:

[ W^{work}_t\leftarrow W^{long}_t. ]

The working memory stays frozen during continuous ODE integration, then updates once the predicted future feature (\hat{f}_{t+1}) is produced:

[ W^{work\prime}t \leftarrow W^{work}_t - \eta\nabla_W\ell{\mathrm{self}}(W^{work}t;\hat{f}{t+1}). ]

This separation is the paper’s cleanest systems idea. Real observations update the durable memory; imagined future features update a temporary fork. The model can use predicted context for action generation while keeping physical history anchored to measured data.

Speculative Asynchronous Inference

Even with constant memory, a normal robot loop is sequential:

execute the current action;
wait for the next observation;
run denoising to predict the next future state and action;
execute again.

CLWM uses Speculative Asynchronous Inference (SAI) to overlap neural computation with robot motion. While the robot executes (a_{t-1}), the true next observation (o_t) is unavailable. CLWM already predicted (\hat{f}_t), so it treats that feature as a surrogate observation and starts partial denoising in the background:

[ s=0\rightarrow s_{\mathrm{mid}}. ]

When the real observation arrives, DINOv3 extracts (f_t), the Long-Term TTT Memory is calibrated with ground truth, and the ODE solver only finishes the remaining denoising interval:

[ s_{\mathrm{mid}}\rightarrow 1. ]

The paper reports that this cuts blocking latency by about 50% compared with a sequential autoregressive pipeline. The history augmentation from training is what makes the speculative phase plausible: the model has already learned to generate stable vector fields under noisy or partially predicted histories.

EmbodiChain and Online Data Streaming

The other half of the paper is the data engine. CLWM pretraining uses aggregated open-source robot manipulation datasets, mainly RoboMind, Agibot World Beta, and InternData-A1. For post-training, the paper says it avoids manually collected real-world or downstream demonstrations and relies on EmbodiChain to generate physics-grounded simulation data.

EmbodiChain has three roles:

Component	What it adds
Generative simulation	creates assets, scenes, layouts, physical metadata, and simulation-ready USD assets
Domain expansion	expands trajectories through reachability-aware sampling, recovery data, visual augmentation, and physics-grounded variation
Online Data Streaming	streams fresh synthetic trajectories into training through a lock-free shared-memory pipeline

The training principle is called the Efficiency Law of Embodied Intelligence. The paper frames useful scaling through experience throughput (E), the volume of unique state-action pairs consumed per training iteration. For fixed compute (C) and parameters (P), performance improves when fresh experience throughput exceeds a critical threshold:

[ E>\tau(C,P). ]

In practice, this means the optimizer should see continuously refreshed, physically valid interactions. A static dataset of the same nominal size can still overfit because each trajectory is replayed too many times.

Training Setup

The reported implementation is substantial:

Detail	Value
Visual encoder	DINOv3 base, frozen
Generative backbone	MoT initialized from Wan2.2-5B
Patch size	(P=16)
Action chunk size	(\tau=16)
Dual-arm action size	30 continuous dimensions
Pretraining optimizer	AdamW
Pretraining learning rate	(1\times10^{-4})
Global batch size	128
Pretraining duration	about 20 epochs
Compute	64 NVIDIA H100 GPUs for about 20 days
RoboTwin fine-tuning	25,000 synthetic trajectories, 40k iterations, (1\times10^{-5}) learning rate

Results

On RoboTwin, CLWM is compared against (\pi0.5), X-VLA, Motus, and LingBot-VA. The average success rates are:

Method	Average success
(\pi0.5)	76.76%
X-VLA	72.84%
Motus	87.02%
LingBot-VA	91.55%
CLWM	94.00%

The gain is most meaningful on tasks that require multi-step manipulation, dual-arm coordination, and robustness to object or layout variation. CLWM loses a few individual rows, while reaching the strongest table average.

The efficiency experiments support the architectural claims:

Claim	Evidence reported
Dual-State TTT Memory gives constant memory	flat (O(1)) peak GPU memory over a 2,000-step episode
SAI reduces control blocking time	about 50% lower blocking latency

The EmbodiChain ablation is also clear. On three representative tasks, adding domain expansion modules improves both in-distribution and out-of-distribution success:

Configuration	ID success	OOD success
Spatial randomization only	64%	25%
+ Visual augmentation	75%	42%
+ Physics-grounded generation	81%	56%
+ Reachability-aware sampling	95%	82%

For Online Data Streaming, lower replay bounds mean fresher data. The reported success rises as each trajectory is reused fewer times:

Training configuration	Hanging Mug	Turn Switch	Stack Bowls
Static baseline, 1,500 demos	62%	85%	88%
ODS sample 213	60%	84%	85%
ODS sample 50	92%	92%	96%
ODS sample 10	96%	98%	98%

This supports the paper’s scaling argument: data freshness and physical diversity matter as much as raw demonstration count.

Real-Robot Evaluation

The real-world platform is Agilex CobotMagic. The paper evaluates four bimanual everyday manipulation tasks:

Method	Water Pouring	Table Rearrangement	Hand-Over and Place	Pan Open and Place
(\pi0)	25%	20%	20%	5%
GR00T N1.5	35%	20%	15%	5%
Sim2Real-VLA	80%	80%	40%	35%
CLWM	95%	90%	80%	65%

The comparison is especially pointed because CLWM and Sim2Real-VLA are trained with simulation data from the EmbodiChain pipeline, while (\pi0) and GR00T N1.5 are finetuned with 50 real-world expert demonstrations per task. The paper’s claim is zero-shot sim-to-real transfer from simulation-only training.

Strengths and Caveats

The main strength is that CLWM attacks representation, memory, latency, and data generation as one system. DINOv3 latent targets reduce low-level visual burden; TTT memory gives long-horizon context without a growing cache; SAI improves the robot loop; EmbodiChain supplies diverse trajectories for sim-to-real training.

The caveats are worth keeping visible. Many claims depend on a large integrated stack: Wan2.2-5B initialization, 64 H100 training, EmbodiChain data generation, RoboTwin, and CobotMagic deployment. The paper reports strong success rates, but reproducibility will depend on how much of the full training and data pipeline becomes accessible. The TTT memory and SAI claims are compelling, yet the paper gives more system-level performance evidence than fine-grained failure analysis. For dexterous hands specifically, this paper is more about general embodied manipulation and bimanual robot control than a narrow multi-finger hand benchmark.

Takeaway

DexWorldModel is useful to remember as a latent semantic WAM. The reusable recipe is:

encode observations with a robust frozen visual foundation model;
predict future semantic features through conditional flow matching;
decode action chunks from the predicted future;
store long history in TTT weights instead of a KV cache;
overlap speculative denoising with real execution;
train with continuously generated, physics-grounded trajectories.

The paper’s bigger message is that world models for robotics need deployment mechanics. Predicting future images is insufficient if memory grows without bound, the robot waits for every denoising pass, or post-training data stays static. CLWM is a concrete attempt to make a world-action model fast enough, memory-stable enough, and data-fed enough for real manipulation.

DexWorldModel 提出的是一个面向操作任务的 Causal Latent World Model (CLWM)。它最核心的设计，是把 world-action generation 从 pixel reconstruction 转到 semantic latent space：先预测未来的 DINOv3 features，再基于这些未来语义特征解码 action chunks。

我的理解是：这篇论文关注的是让 world-action model 真正可部署。CLWM 同时处理三个瓶颈：pixel-level prediction 会把容量浪费在 texture 和背景细节上，autoregressive history 会让 memory 随 horizon 增长，diffusion-style generation 会拖慢 closed-loop control。论文分别用 DINOv3 latent targets、Dual-State TTT Memory 和 Speculative Asynchronous Inference 来解决。

论文信息

论文标题是 “DexWorldModel: Causal Latent World Modeling towards Automated Learning of Embodied Tasks”，作者是 Yueci Deng, Guiliang Liu, Kui Jia，机构为 DexForce AI。arXiv 页面是 arXiv:2604.16484，提交日期是 2026 年 4 月 13 日。

论文还把 EmbodiChain 作为 CLWM post-training 阶段的数据引擎。文中引用的平台是 github.com/DexForce/EmbodiChain。

核心问题

World Action Models (WAMs) 在 VLA policy 的基础上加入未来世界状态生成。一个常见形式是：

[ \hat{o}{t+1}\sim p\theta(\cdot\mid o_{\le t},a_{<t},l), ]

[ a_t\sim g_\psi(\cdot\mid o_{\le t},a_{<t},\hat{o}_{t+1},l). ]

这个形式给 policy 增加了显式 future-state interface。代价也很重：如果 (\hat{o}_{t+1}) 位于 pixel space 或低层 VAE latent space，模型会花大量容量建模 lighting、texture 和 background；如果所有历史都通过 Transformer KV cache 保存，memory 会按 (O(T)) 增长；如果每个 action 都串行等待 sensing、完整 denoising 和 execution，机器人每个 control chunk 都会付出较高 latency。

DexWorldModel 的判断是：manipulation world model 应该预测对 action 真正有用的未来部分，也就是 interaction semantics。

CLWM 架构

CLWM 在 Mixture of Transformers (MoT) 设计下包含两个耦合的生成模块：

模块	目标	作用
Latent Video Model	future DINOv3 feature (f_{t+1})	预测 semantic world evolution
Action Model	action chunk (a_t)	从预测未来语义中解码控制

视觉特征由 frozen DINOv3 提取：

[ f_t=\Phi_{\mathrm{DINO}}(o_t)\in\mathbb{R}^{C\times H’\times W’}. ]

论文使用 patch size (P=16)。两个 generator 共享一个从 Wan2.2-5B 初始化的 core Transformer，同时保留各自的 input/output projections 和 flow-time embeddings：

生成过程是 autoregressive 和 causal 的。第一步，Latent Video Model 通过 conditional flow matching 预测下一时刻语义特征：

[ \mathcal{L}{\mathrm{video}} = \mathbb{E}\left[ \left\lVert v{\phi_{\mathrm{vid}}}(f^{(s)}{t+1},s\mid h{\le t},l) - \dot{f}^{(s)}_{t+1} \right\rVert^2 \right]. ]

第二步，Action Model 预测 action chunk：

[ a_t={a_{t,1},a_{t,2},\ldots,a_{t,\tau}}, ]

实验中 (\tau=16)。dual-arm control 的 action vector 很紧凑：每只手臂包含 7-DoF end-effector pose、7 个 joint positions 和 1 个 gripper state，所以总维度是 ((7+7+1)\times 2=30)。

action loss 同样是 flow-matching objective，条件包括历史、语言和预测出的未来 feature：

[ \mathcal{L}{\mathrm{action}} = \mathbb{E}\left[ \left\lVert v{\phi_{\mathrm{act}}}(a^{(s)}t,s\mid \tilde{h}{\le t},l,\tilde{f}_{t+1}) - \dot{a}^{(s)}_t \right\rVert^2 \right]. ]

一个小但关键的细节是 history augmentation。训练时，模型以 (p=0.5) 的概率向 historical latent features 注入 Gaussian noise：

[ \tilde{f}{\le t}=(1-s{\mathrm{aug}})\epsilon+s_{\mathrm{aug}}f_{\le t}, \quad s_{\mathrm{aug}}\in[0.5,1]. ]

这会训练 action model 容忍有误差的 predicted histories。后面的 SAI 会在真实 observation 到达前，用 speculative future features 提前启动推理，因此这个训练技巧很重要。

Dual-State TTT Memory

标准 autoregressive world models 通过 KV cache 保存历史。对长程机器人交互来说，这会带来持续增长的 memory 和 latency：

[ \text{KV cache memory}\sim O(T). ]

CLWM 用 Test-Time Training (TTT) Memory layer 替换 cache。模型把历史压缩进动态更新的 layer weights，避免显式保存所有历史 token。TTT-MLP 通过一个 self-supervised reconstruction objective 训练：

[ \ell_{\mathrm{self}}(W;z_t) = \lVert f(\theta_K z_t;W)-\theta_V z_t\rVert^2. ]

inner-loop weights 更新到 (W_t) 后，通过 query projection 取出 hidden state：

[ l_t=f_{\mathrm{TTTmlp}}(\theta_Q z_t;W_t). ]

论文把它封装成 Dual-State 设计：

Memory	更新来源	作用
Long-Term TTT Memory	real observations 和 executed actions	锚定真实物理历史
Working TTT Memory	forked copy 加 predicted future features	为短程生成提供条件

Long-term memory 的更新是：

[ W^{long}t = W^{long}{t-1} - \eta\nabla_W\ell_{\mathrm{self}}(W^{long}_{t-1};h_t). ]

生成阶段，CLWM 先 fork：

[ W^{work}_t\leftarrow W^{long}_t. ]

working memory 在连续 ODE integration 期间保持冻结；当预测未来特征 (\hat{f}_{t+1}) 产生后，它再更新一次：

[ W^{work\prime}t \leftarrow W^{work}_t - \eta\nabla_W\ell{\mathrm{self}}(W^{work}t;\hat{f}{t+1}). ]

这个分离是论文里很干净的系统设计。真实 observation 更新 durable memory；想象出来的 future feature 只更新 temporary fork。模型可以使用 predicted context 来生成 action，同时避免把真实物理历史污染掉。

Speculative Asynchronous Inference

即使 memory 已经是 constant，一个普通机器人 loop 仍然是串行的：

执行当前 action；
等待下一个 observation；
运行 denoising，预测下一步 future state 和 action；
再执行。

CLWM 使用 Speculative Asynchronous Inference (SAI) 把 neural computation 和 robot motion 重叠起来。当机器人正在执行 (a_{t-1}) 时，真实的下一帧 observation (o_t) 还不可见。CLWM 已经预测过 (\hat{f}_t)，于是把这个 feature 当作 surrogate observation，在后台提前做一段 partial denoising：

[ s=0\rightarrow s_{\mathrm{mid}}. ]

真实 observation 到达后，DINOv3 提取 (f_t)，Long-Term TTT Memory 用 ground truth 校准，ODE solver 只需要完成剩余 denoising：

[ s_{\mathrm{mid}}\rightarrow 1. ]

论文报告，这使 blocking latency 相比串行 autoregressive pipeline 降低约 50%。前面的 history augmentation 是 SAI 能工作的训练基础：模型已经被训练过在 noisy 或 partially predicted histories 下生成稳定 vector fields。

EmbodiChain 和 Online Data Streaming

CLWM pretraining 使用聚合的开源 robot manipulation datasets，主要包括 RoboMind、Agibot World Beta 和 InternData-A1。post-training 阶段，论文表示绕开人工采集的 real-world 或 downstream demonstrations，使用 EmbodiChain 生成 physics-grounded simulation data。

EmbodiChain 有三层作用：

Component	提供什么
Generative simulation	生成 assets、scenes、layouts、physical metadata 和 simulation-ready USD assets
Domain expansion	通过 reachability-aware sampling、recovery data、visual augmentation 和 physics-grounded variation 扩展轨迹
Online Data Streaming	通过 lock-free shared-memory pipeline 把新合成轨迹流式送入训练

论文把训练原则称为 Efficiency Law of Embodied Intelligence。它用 experience throughput (E) 来描述每次训练迭代消费的 unique state-action pairs 数量。在固定 compute (C) 和参数量 (P) 下，当 fresh experience throughput 超过某个阈值时，性能会更有效地提升：

[ E>\tau(C,P). ]

落到实践上，就是 optimizer 应该持续看到刷新过的、物理有效的 interactions。一个名义上同样大小的 static dataset，如果每条 trajectory 被反复 replay，也容易过拟合。

训练设置

论文报告的实现规模比较大：

细节	数值
Visual encoder	DINOv3 base, frozen
Generative backbone	MoT initialized from Wan2.2-5B
Patch size	(P=16)
Action chunk size	(\tau=16)
Dual-arm action size	30 continuous dimensions
Pretraining optimizer	AdamW
Pretraining learning rate	(1\times10^{-4})
Global batch size	128
Pretraining duration	about 20 epochs
Compute	64 NVIDIA H100 GPUs, about 20 days
RoboTwin fine-tuning	25,000 synthetic trajectories, 40k iterations, (1\times10^{-5}) learning rate

实验结果

在 RoboTwin 上，CLWM 和 (\pi0.5)、X-VLA、Motus、LingBot-VA 对比。平均成功率如下：

Method	Average success
(\pi0.5)	76.76%
X-VLA	72.84%
Motus	87.02%
LingBot-VA	91.55%
CLWM	94.00%

这个提升在 multi-step manipulation、dual-arm coordination 和 object/layout variation 相关任务上更有意义。CLWM 并没有每一行都赢，但整体平均最好。

efficiency 实验支持了架构主张：

Claim	论文报告的证据
Dual-State TTT Memory gives constant memory	2,000-step episode 中 peak GPU memory 保持 flat (O(1))
SAI reduces control blocking time	blocking latency 约降低 50%

EmbodiChain ablation 也比较清晰。在三个代表性任务上，逐步加入 domain expansion modules 后，ID 和 OOD success 都提升：

Configuration	ID success	OOD success
Spatial randomization only	64%	25%
+ Visual augmentation	75%	42%
+ Physics-grounded generation	81%	56%
+ Reachability-aware sampling	95%	82%

对 Online Data Streaming 来说，replay bound 越低，说明数据越新鲜。论文报告，每条 trajectory 被复用越少，成功率越高：

Training configuration	Hanging Mug	Turn Switch	Stack Bowls
Static baseline, 1,500 demos	62%	85%	88%
ODS sample 213	60%	84%	85%
ODS sample 50	92%	92%	96%
ODS sample 10	96%	98%	98%

这支持了论文的 scaling argument：data freshness 和 physical diversity 与 raw demonstration count 一样关键。

真实机器人实验

真实平台是 Agilex CobotMagic。论文评估了四个双臂日常操作任务：

Method	Water Pouring	Table Rearrangement	Hand-Over and Place	Pan Open and Place
(\pi0)	25%	20%	20%	5%
GR00T N1.5	35%	20%	15%	5%
Sim2Real-VLA	80%	80%	40%	35%
CLWM	95%	90%	80%	65%

这个对比点比较强：CLWM 和 Sim2Real-VLA 使用 EmbodiChain pipeline 产生的 simulation data；(\pi0) 和 GR00T N1.5 则使用每个任务 50 条 real-world expert demonstrations 做 finetuning。论文主张的是 simulation-only training 后的 zero-shot sim-to-real transfer。

优点和注意点

CLWM 的主要优点，是把 representation、memory、latency 和 data generation 当成一个系统一起处理。DINOv3 latent targets 降低低层视觉负担；TTT memory 避免长程上下文导致 cache 增长；SAI 改善机器人执行循环；EmbodiChain 为 sim-to-real training 提供多样轨迹。

需要注意的是，很多结果依赖一个大型集成栈：Wan2.2-5B initialization、64 H100 训练、EmbodiChain 数据生成、RoboTwin benchmark 和 CobotMagic deployment。论文报告了很强的 success rates，但可复现性取决于完整训练和数据 pipeline 的开放程度。TTT memory 和 SAI 的方向很有吸引力，不过论文给出的主要是 system-level performance evidence，细粒度 failure analysis 还不多。对于 dexterous hands 来说，这篇更偏 general embodied manipulation 和 bimanual robot control，并非专门的多指灵巧手 benchmark。

Takeaway

DexWorldModel 可以记成一个 latent semantic WAM。可复用的 recipe 是：

用 robust frozen visual foundation model 编码 observation；
通过 conditional flow matching 预测 future semantic features；
从 predicted future 中解码 action chunks；
用 TTT weights 保存 long history，替代 KV cache；
把 speculative denoising 和真实执行时间重叠；
用连续生成的 physics-grounded trajectories 训练。

这篇论文更大的信息是：robotics world model 需要 deployment mechanics。只会预测 future images 还不够；如果 memory 无界增长、机器人每次都等待完整 denoising、post-training data 保持静态，world model 很难变成可靠控制器。CLWM 是一次把 world-action model 做到 faster、memory-stable、data-fed 的系统尝试。

Lixin Xu

[Paper Notes] TeleDexter: Towards Human-level Dexterous Teleoperation

Paper Info

The Missing Layer in Dexterous Teleoperation

1. Geometry-Aware Reference Motions

2. Consecutive Subgoals Give RL Room to Discover Contact

3. The Learned Controller and Its Information Boundary

4. Random Action Masking Bridges Asynchronous Hardware

5. Real-Time Teleoperation Has Two Control Modes

Experiments: Where the Gain Appears

From Teleoperation to Autonomous Policies

How to Interpret the 75.2% Result

Failure Modes and Research Opportunities

Takeaway

论文信息

灵巧遥操作缺失的中间层

1. 带物体几何约束的参考动作

2. 连续子目标给 RL 留出发现接触策略的空间

3. 控制器观测与信息边界

4. Random Action Masking 处理硬件异步响应

5. 真实遥操作包含两种控制模式

实验：优势出现在哪些阶段

从遥操作数据到自主策略

75.2% 这个结果能说明什么

失败模式与后续方向

总结

[Paper Notes] FIGAROH: a Python toolbox for dynamic identification and geometric calibration of robots and humans

Paper Info

Two Problems in One Toolbox

Geometric calibration

Dynamic identification

Rank Deficiency and Base Parameters

Parameter Estimation

Optimal Experiment Design

Selecting calibration postures

Generating exciting motions

What the Paper Demonstrates

Reading the Current figaroh-plus Code

Workflow layer

Tools layer

Backend layer

Paper Versus Current Repository

Strengths

Limitations and Cautions

Takeaway

论文信息

一个工具箱里的两个问题

Geometric calibration

Dynamic identification

Rank Deficiency 与 Base Parameters

Parameter Estimation

Optimal Experiment Design

选择 calibration postures

生成 exciting motions

论文做了哪些验证？

阅读当前 figaroh-plus 代码

Workflow layer

Tools layer

Backend layer

论文与当前仓库的关系

优点

局限与注意事项

总结

[Paper Notes] OmniContact: Chaining Meta-Skills via Contact Flow for Generalizable Humanoid Loco-Manipulation

Paper Info

The Problem: What Should a High-Level Planner Command?

Contact Flow

CF-Track: The Learned Low-Level Executor

CF-Gen: Rule-Based Contact-Flow Synthesis

1. Select object-centric contact anchors

2. Generate keyframes with selective IK

3. Interpolate a dense kinematic reference

4. Convert the reference into sparse contact flow

Closed-Loop Chaining and Recovery

OmniContact Dataset

Main Results

Real-World Deployment

VLM Integration

What Is Actually New?

Limitations

Reading the Current `figaroh-plus` Code

阅读当前 `figaroh-plus` 代码