[Paper Notes] Grasp Multiple Objects with One Hand
Published:
MultiGrasp enables a Shadow Hand to grasp and lift multiple independent objects concurrently. Its decisive choice is to make every stage object-aware: the pre-grasp objective asks for force closure on each object, the diffusion model keeps object identities separate, and the lifting reward is controlled by the lowest object. A grasp that secures only one of two targets cannot score well.
My read: the paper succeeds by dividing the problem at the boundary between static contact geometry and dynamic execution. An augmented differentiable force-closure optimizer and a diffusion model construct a multi-contact hand pose; motion planning and goal-conditioned reinforcement learning then reach that pose and preserve all contacts during lift. This split is especially important because moving one finger to help one object can destabilize the other.
Paper Info
“Grasp Multiple Objects with One Hand” is by Yuyang Li, Bo Liu, Yiran Geng, Puhao Li, Yaodong Yang, Yixin Zhu, Tengyu Liu, and Siyuan Huang. It appeared in IEEE Robotics and Automation Letters (RA-L), Vol. 9, No. 5, 2024.
Why Two Objects Change the Problem
A single-object dexterous grasp can often envelop the target and close all fingers toward it. With two independent objects, improving contact on one target may release the other. The hand must use its full workspace—including finger lengths and side surfaces—and maintain a separate force-closure condition for every object.
MultiGrasp studies tabletop arrangements
[ O={O_j}_{j=1}^{N_o}, ]
where each object is represented by a surface point cloud. The output is a sequence of hand actions that simultaneously grasps and lifts all targets. The objects must be close enough and small enough to fit in one hand; this is a geometric assumption of the task, not a capability learned around arbitrary placements.
The complete pipeline has two high-level stages:
- propose and refine a pre-grasp pose that encloses all targets;
- plan a collision-aware reach, then use a learned policy to lift every object.
1. Independent Force Closure for Every Object
The synthetic grasp generator extends Differentiable Force Closure (DFC). For hand configuration (H) and objects (O), it minimizes
[ E(H,O) = \sum_{j=1}^{N_o} \min_{x_j\subset S(H)} E_{\mathrm{FC}}(x_j,O_j) +\lambda_p E_p(H,O) +\lambda_{sp}E_{sp}(H) +\lambda_qE_q(H). ]
The first term is the key multi-object design. For each object (O_j), the optimizer selects hand contacts (x_j) and independently evaluates force-closure error. Stability cannot come solely from treating the pair as one combined shape. The remaining terms penalize hand-object penetration, hand self-collision, and joint-limit violations. Gradient optimization with Metropolis-Adjusted Langevin Algorithm sampling explores multiple local solutions, followed by filtering.
This objective encourages different parts of the dexterous hand to support different objects. The allocation is produced by contact optimization instead of a fixed assignment such as “thumb and index finger for object A.”
2. A Diffusion Model That Preserves Object Identity
DFC synthesis is effective and slow—roughly 1000 seconds in the reported batched setting. MultiGrasp therefore learns a SceneDiffuser-style DDPM that maps the object point clouds to a pre-grasp pose:
[ p(H^{(0)}\mid O) = \prod_{t=1}^{T}p(H^{(t-1)}\mid H^{(t)},O). ]
PointNet++ extracts features independently from each object. The denoising network uses the noisy hand representation as queries and object features as keys and values in cross-attention. A learned object embedding is attached to every point feature: points from the same object share an embedding, while different objects receive different embeddings. The network can therefore reason about two independent targets instead of seeing one merged point set.
The hand pose is represented by 31 Cartesian keypoints distributed over the palm and finger links. An optimization-based inverse-kinematics solver converts these points to joint angles. Cartesian keypoints expose local finger-link/object relations directly. In the ablation, this representation reaches 40.20% execution success, compared with 19.31% when the model directly generates joint angles.
The training set, Grasp’Em, contains about 90,000 synthetic pre-grasps:
- 16.4k single-object grasps;
- 73.7k dual-object grasps;
- 8 objects and 36 single/dual combinations from YCB and ContactDB;
- randomized stable tabletop positions and orientations.
The objects are rescaled so several can fit in the hand. Training also normalizes the palm’s projected direction around the tabletop’s vertical axis. At inference, rotating the scene into this canonical orientation and rotating the predicted hand back allows the system to control palm direction and avoid particularly difficult approaches.
3. Contact Refinement Before Execution
Diffusion samples can contain shallow penetration or leave useful fingers floating near a surface. A coarse-to-fine refinement step minimizes penetration while attracting nearby potential hand contacts toward object surfaces. Its distance threshold decreases from 2 mm to 1 mm.
This small-looking stage has a large effect. Removing refinement increases maximum penetration from 1.67 mm to 7.68 mm and reduces success from 40.20% to 16.24%. The comparison shows that a visually plausible hand pose is insufficient; multi-object lifting needs contacts that survive dynamics.
4. Reach First, Then Learn to Lift
Execution is split into reaching and lifting. The reaching stage linearly interpolates from a flat hand to the pre-grasp, then optimizes the trajectory for smoothness and hand-object collision reduction. Small remaining collisions may displace an object, so the lifting policy is trained to recover from the resulting state.
The PPO lifting policy controls 18 actuated joint-angle targets and a 6D hand-base wrench. Its observations include proprioception, fingertip wrenches, current and goal hand states, object states, and point-cloud features. The policy is goal-conditioned on the pre-grasp configuration instead of discarding it after reaching.
The central reward is
[ r = \omega_{\mathrm{lift}}r_{\mathrm{lift}} +\omega_{\mathrm{succ}}\mathbf{1}{\mathrm{succ}} +\omega_r r_r +\omega_q r_q +\omega{\mathrm{obj}}r_{\mathrm{obj}}, \qquad r_{\mathrm{lift}}=\min_j h_j. ]
Here (h_j) is the height of object (j). Taking the minimum makes the lowest object the bottleneck: lifting one target high cannot compensate for leaving or dropping another. A success bonus is activated only when all objects exceed the training threshold. Additional terms maintain the goal hand orientation, joint configuration, and each object’s position relative to the hand.
This reward encodes simultaneous grasping in the dynamic stage. The ablation is direct: the complete training setting obtains 45.25%, while removing RL leaves only 1.37% success. Removing goal observations and rewards gives 16.79%.
5. Specialists, Vision Distillation, and Error Adaptation
Object shape, pair identity, and placement relative to the palm produce different lifting dynamics. The authors cluster training grasps by object combination and by the direction connecting object centers in the palm frame. A specialist policy is trained for each cluster. Their demonstrations are distilled with DAgger into a vision-based generalist that consumes a fused scene point cloud from three RGB-D cameras.
The final policy also needs to tolerate imperfect pre-grasps. MultiGrasp uses a three-phase curriculum:
- train on high-quality synthetic pre-grasps;
- mix synthetic samples with generated poses and states disturbed during reaching;
- emphasize generated and displaced states.
Without adaptation to imperfect poses, success drops to 25.05%; removing the curriculum yields 24.88%. This training recipe closes the gap between a static target pose and the actual state encountered after physical reaching.
What the Experiments Establish
Success is defined as lifting all objects above 10 cm. Evaluation uses 512 unique poses with five trials per pose. For unseen object placements using synthetic pre-grasps, state-based specialists achieve 68.34%, while the distilled vision generalist achieves the paper’s headline 44.13%. With diffusion-generated pre-grasps on unseen placements, the corresponding results are 40.20% / 30.24%.
The number 44.13% therefore has a precise scope: it is the vision-based generalist’s simulated dual-object success with synthetic pre-grasps on unseen placements. It is not an unrestricted real-world success rate.
The pre-grasp ablations isolate the representation choices:
| Pre-grasp model | Q1 ↑ | Penetration (mm) ↓ | Success ↑ |
|---|---|---|---|
| Full model | 0.29 | 1.67 | 40.20% |
| Joint-angle representation | 0.18 | 5.50 | 19.31% |
| Without object embedding | 0.27 | 1.38 | 37.21% |
| Without refinement | 0.29 | 7.68 | 16.24% |
The method also demonstrates three to five small cylinders. As object count grows, inter-object contact becomes increasingly important; the five-object case uses an inverted, scooping hand pose. These tests use synthesized grasps and case-specific state-based execution policies, so they demonstrate scalability of the formulation more than a single general policy across arbitrary object counts.
On the physical system, a Shadow Hand mounted on a UR10e lifts two objects. The authors precompute execution trajectories in simulation and replay them on the robot. This experiment establishes physical feasibility while leaving online perception and closed-loop sim-to-real robustness open.
Strengths and Limitations
The strongest aspect is alignment across stages. Static synthesis, learned generation, execution reward, and evaluation all require success on every object. The method also preserves the distinction between two independent objects through explicit embeddings and per-object state goals.
Its main boundaries are equally clear. Objects must be close and fit in one hand. The data contains only eight mostly convex training objects, and performance falls on unseen combinations and geometries. Vision suffers from severe occlusion around a closed hand, and the physical demonstration replays trajectories prepared in simulation. Typical failures come from missing force closure, penetration, dropping an object during lift, or failing to raise every target. The paper identifies tactile sensing and improved sim-to-real modeling as natural next steps.
Takeaways
MultiGrasp’s recipe for concurrent dual-object grasping can be summarized as
[ \boxed{ \text{per-object force closure} +\text{object-aware diffusion} +\text{Cartesian hand keypoints} +\text{contact refinement} +\min_j h_j\text{-driven lifting} }. ]
The first and last terms are the most fundamental. Per-object force closure defines a static grasp that supports both targets; the minimum-height reward prevents the controller from sacrificing one target during execution. The keypoint generator, refinement, specialist distillation, and curriculum make that objective computationally practical and robust to imperfect reaching.