[Paper Notes] UniDexTok: A Unified Dexterous Hand Tokenizer from Real Data
Published:
UniDexTok studies a representation problem that sits before dexterous manipulation policy learning: how can data from different human and robot hands be put into one shared state space, then converted into discrete tokens that future VLA or world-action models can use?
My read: the paper’s core contribution is a retargeting-free state tokenizer. UniDexTok avoids converting every hand trajectory into a target robot hand through MANO-to-robot retargeting. It first standardizes human and robot hand states into a 22-DoF semantic interface called UDHM, then trains one conditional tokenizer with a shared encoder, shared codebook, and shared decoder across embodiments. The important shift is from “make every hand imitate one robot” to “make every hand speak the same joint-semantic language.”
Paper Info
The paper is “UniDexTok: A Unified Dexterous Hand Tokenizer from Real Data” by Dong Fang, Youjun Wu, Yuanxin Zhong, Rui Zhang, Yunlong Wang, Xiaosong Jia, and Yu-Gang Jiang from Fudan University, Rimbot, Hefei University of Technology, and Beijing University of Posts and Telecommunications. It is available as arXiv:2606.10683.
Why Dexterous Hand States Are Hard to Share
Parallel grippers are easy to represent: opening width, gripper pose, maybe a binary open-close state. Dexterous hands are messier. Different hands have different numbers of fingers, joint names, joint orders, coupled joints, control ranges, and units. Even datasets collected with the same family of hands can log joint states in different conventions, including radians, degrees, or discretized bins.
This makes cross-embodiment learning awkward. If every robot hand has a separate tokenizer, then each codebook learns a private latent space. The model may reconstruct each hand, but the discrete tokens do not have a stable meaning across hardware. If the pipeline relies on retargeting human motion into every robot hand, the representation inherits geometric mismatch from retargeting and loses the native state statistics of real robot data.
UniDexTok’s answer is to standardize the state space first, then tokenize.
UDHM: A 22-DoF Semantic Interface
The Unified Dexterous Hand Model, or UDHM, is the paper’s shared coordinate system. It assumes a human-hand-like kinematic structure and maps MANO-style human keypoints and robot hand joints into a 22-dimensional active joint interface.
For the index, middle, and ring fingers, UDHM uses four active coordinates per finger: MCP abduction/adduction, MCP flexion/extension, PIP flexion/extension, and DIP flexion/extension. The pinky adds one twist degree of freedom to cover larger lateral variation. The thumb uses five coordinates: CMC flexion, CMC spread, MCP flexion, MCP abduction, and IP flexion. Together these choices give 22 active coordinates.
The model fits a palm plane from the wrist and MCP keypoints, uses the palm normal and local lateral axes to define motion planes, and reconstructs each finger chain with analytic forward kinematics. For inverse mapping, it extracts palm offsets and bone lengths from the input frame, then solves a nonlinear least-squares problem so the reconstructed joints match the target keypoints.
The practical detail I like is the treatment of heterogeneous robot logs. Each available robot-hand DoF is inserted into the semantically corresponding UDHM coordinate, and missing coordinates are zero-padded. This “semantic insertion” is stronger than simply appending all available coordinates at the front of a vector, because the model can learn that a coordinate means “index MCP pitch” across embodiments without memorizing source-specific column positions.
All raw joint angles are converted to radians and normalized by a fixed scale of (\pi). That fixed normalization keeps tokens from depending on train-test split statistics and keeps MPJAE interpretable after denormalization.
UniDexTok Architecture
Given a standardized hand state (x \in \mathbb{R}^{22}) and a hand-type or source label (h), the tokenizer learns
[ \hat{x} = \pi D(Q(E(\tilde{x}, h_{\mathrm{embed}})), h_{\mathrm{embed}}), \quad \tilde{x}=x/\pi. ]
The encoder projects the 22-dimensional normalized state into (N=8) latent tokens with width (C=512). Transformer blocks then model correlations among joint coordinates. Hand-type conditioning enters through adaptive layer normalization:
[ \mathrm{AdaLN}(z,h_{\mathrm{embed}})=\gamma(c_h)\odot \mathrm{LN}(z)+\beta(c_h). ]
This conditioning matters because the same semantic pose can require different numeric joint patterns on different hands. The shared encoder and decoder learn the cross-hand structure, while the conditioning keeps hardware-specific conventions available.
The decoder mirrors the encoder. It takes quantized tokens, adds positional embeddings and hand conditioning, and reconstructs the normalized 22-dimensional state. The reconstruction objective is based on normalized joint-angle reconstruction, and the full objective adds the VQ commitment loss:
[ L = L_{\mathrm{rec}} + L_{\mathrm{vq}}, ]
with
[ L_{\mathrm{vq}}=\beta\lVert \mathrm{sg}[q]-u\rVert_2^2+\lVert q-\mathrm{sg}[u]\rVert_2^2, ]
where (\beta=0.25) in the reported experiments.
Factorized Codebook
A simple 256-entry codebook would give only 256 choices per quantized vector. UniDexTok instead factorizes the vocabulary across channel groups. The encoder output is projected from 512 to 256 dimensions, divided into (K=8) groups, and each group uses a 32-entry sub-codebook. That means one token can express (32^8) code combinations while storing only (32 \times 8 = 256) learned code vectors.
For each token position and group, the nearest normalized code vector is selected by cosine similarity:
[ i_{n,k}=\arg\max_j \left\langle \frac{u_{n,k}}{\lVert u_{n,k}\rVert_2}, \frac{e_{k,j}}{\lVert e_{k,j}\rVert_2} \right\rangle. ]
This design is important for discrete representation quality. In the paper’s gesture benchmark, UniHM’s single-codebook quantized features drop from 96.15% to 84.62% linear probing accuracy, while UniDexTok keeps 100% accuracy after quantization. The interpretation is straightforward: a single codebook can collapse different gestures into the same discrete code; factorized VQ gives the token more compositional capacity.
Data and Evaluation
The training data combines two main sources. The first is coarse human hand-object interaction data from DexYCB, OakInk-v2, and EgoDex. UniDexTok does not retarget those motions to a robot hand. It reduces the MANO-style 45-DoF hand representation to the 22 active UDHM coordinates and trains on the standardized human hand as another embodiment.
The second source is real robot-hand data from LET, Dexora, and LinkerHand. These datasets differ in joint ordering, dimensionality, and units, so UniDexTok converts all values to radians, inserts available DoFs into the corresponding UDHM coordinates, and pads missing coordinates.
The paper evaluates reconstruction in joint-angle space and Cartesian space. MPJAE measures mean per-joint angle error in degrees. MPJPE measures mean per-joint position error in millimeters. FK error reports fingertip-position errors.
Main Results
Against UniHM on the standardized real datasets, UniDexTok reduces average MPJAE from 15.63 degrees to 0.16 degrees and average MPJPE from 18.51 mm to 0.18 mm. That is the paper’s headline result: reconstruction improves from centimeter-scale error to sub-millimeter error.
The improvement is consistent across LinkerHand L6, LinkerHand L10, LinkerHand L20, and Robotera XHand1. On the retargeted DexYCB evaluation, the gap is smaller because that protocol is closer to the retargeted data distribution used by UniHM, but UniDexTok still improves average MPJAE from 4.40 degrees to 2.83 degrees.
The zero-shot and few-shot results are also useful. On an unseen Inspire RH56E2 hand, zero-shot per-joint angle errors range from 4.14 to 7.85 degrees, and fingertip FK errors range from 7.73 to 16.05 mm. With only 4,528 frames, about 6.2% of the full Inspire dataset, fine-tuning for 2 epochs reduces joint errors to roughly 1.42 to 2.08 degrees and reduces fingertip errors by 58.5% to 78.8%.
This is the evidence for treating the tokenizer as a cross-embodiment representation, beyond a compression model for one hand. A new hand can enter the same token space and then adapt with a small amount of target data.
Ablations
The ablation on human-hand data is especially relevant for embodied datasets. With human-hand data included, the average MPJAE is 0.16 degrees and MPJPE is 0.18 mm. Without human-hand data, they rise to 0.37 degrees and 0.43 mm. The human hand is therefore useful as a training embodiment, not merely as a source for retargeting.
The semantic-insertion ablation supports the UDHM design. With semantic insertion, MPJAE, MPJPE, and FK error are 0.24 degrees, 0.25 mm, and 0.53 mm. Without semantic insertion, they degrade to 0.53 degrees, 0.57 mm, and 1.20 mm. The benefit comes from aligning coordinates by joint meaning.
Strengths and Limitations
The strength of UniDexTok is that it attacks the representation bottleneck directly. It gives heterogeneous hand datasets a common joint-semantic interface, learns discrete tokens from real standardized states, and avoids making retargeted trajectories the primary source of robot-hand state data. This is a good fit for future dexterous VLA systems, where an action or state token needs a stable meaning across hardware.
The limitations are also clear. UDHM assumes a human-hand-like kinematic structure, so non-anthropomorphic grippers, soft hands, heavily underactuated hands, tendon coupling, compliance, and actuator-level dynamics are only partially captured. The tokenizer focuses on state reconstruction and does not model contacts, tactile signals, object geometry, force, or temporal action dynamics. Low reconstruction error is valuable, but it does not by itself prove better downstream manipulation.
Takeaway
UniDexTok is best understood as infrastructure for dexterous foundation models. It leaves manipulation policy learning to the next layer and solves a lower-level but very important problem: make heterogeneous real hand states share one semantic, discrete, embodiment-conditioned token space.
The larger lesson is that cross-embodiment learning needs more than a big model and more data. It needs careful state standardization. Once the joint semantics are aligned, human data, robot data, seen hands, and new hands can contribute to the same representation without living in isolated codebooks.