[Paper Notes] InterMimic: Towards Universal Whole-Body Control for Physics-Based Human-Object Interactions
Published:
InterMimic learns one physics-based controller for diverse whole-body human-object interactions (HOIs) from imperfect motion-capture data. Its central recipe is “perfect first, then scale up”: small teacher policies first retarget and physically repair the captured interactions; a Transformer student then absorbs the teachers’ corrected references and actions, followed by reinforcement-learning fine-tuning.
My read: the most important idea is that the teachers are more than action-label generators. Their simulated rollouts become a cleaner training distribution for the student. This converts teacher-student learning into two coupled operations—reference distillation cleans what the student should track, while policy distillation teaches how to track it. PPO then resolves conflicts left by supervised aggregation.
Paper Info
The paper is “InterMimic: Towards Universal Whole-Body Control for Physics-Based Human-Object Interactions” by Sirui Xu, Hung Yu Ling, Yu-Xiong Wang, and Liang-Yan Gui, from the University of Illinois Urbana-Champaign and Electronic Arts. It was presented at CVPR 2025 as a Highlight paper.
The Problem: Large HOI Data Is Useful and Physically Messy
Physics-based motion imitation usually asks a simulated character to follow a reference trajectory. Human-object interaction raises the difficulty because errors couple across the body and the object. A floating hand contact can make the object fall; an inaccurate symmetric-object rotation can imply sliding; shape differences between captured subjects and the simulated body can destroy a valid interaction after retargeting. Many HOI datasets also omit detailed finger motion.
The target trajectory contains human and object states:
[ q_t = {q_t^h,q_t^o}, \qquad q_t^h={\theta_t^h,p_t^h}, \qquad q_t^o={\theta_t^o,p_t^o}. ]
The human model has 52 joints: 22 body joints and 30 hand joints. Fifty-one joints are actuated, and the action
[ a_t\in\mathbb{R}^{51\times 3} ]
specifies exponential-map joint targets that a PD controller converts into torques. The objective is therefore richer than pose matching: the controller must preserve the interaction outcome while repairing reference errors and accommodating a new embodiment.
The Two-Stage Curriculum
| Stage | Model | Training signal | Main job |
|---|---|---|---|
| 1. Perfect | Subject-specific MLP teachers | PPO with tracking, contact, and energy rewards | Retarget and physically refine small data subsets |
| 2. Scale | One Transformer student | Refined references, teacher actions, DAgger, then PPO | Integrate many subjects, objects, and interaction skills |
The decomposition is a space-time trade-off. Seventeen OMOMO teacher policies can be trained on subject-sized subsets in parallel. Their experience then bootstraps one large student, avoiding the sample cost of discovering every skill through PPO from scratch.
Policy Inputs: Geometry, Contact, and Future Goals
The policy state has a physical-state component (s_t^s) and a goal component (s_t^g):
[ s_t={s_t^s,s_t^g}. ]
The physical state contains human and object poses and velocities, plus two interaction descriptors:
- (d_t): vectors from human joints to the nearest points on each object surface;
- (c_t): body-part contact markers derived from applied forces.
The goal provides relative and absolute reference information at several future keyframes. Teachers use offsets (K={1,16}); the student uses (K={1,2,4,16}). The longer temporal context helps the student distinguish skills that look locally similar but require different future actions.
Most MoCap datasets have no reliable contact labels. InterMimic infers reference contact using object acceleration as evidence of human-applied force, then combines it with distance thresholds. Contacts fall into three zones: promote, neutral, or penalize. The neutral buffer is important because a noisy reference distance should not become a rigid physical constraint.
Stage 1: Imitation as Retargeting and Perfecting
Each teacher uses the same canonical human model while learning data from one captured subject. This embeds retargeting inside RL. The paper separates the reward into two kinds of evidence.
Embodiment-aware terms loosely preserve the captured kinematics. Joint-position tracking receives more weight near an object, where spatial precision controls contact; joint-rotation tracking dominates farther away. Joint-to-object distance tracking preserves the interaction layout.
Embodiment-agnostic terms preserve the interaction dynamics through object position and rotation tracking, promoted or penalized contacts, and energy penalties. The authors’ assumption is that body shape may change the exact kinematic trajectory while the human-object dynamics should remain consistent.
For a tracking cost (E), each corresponding reward takes the exponential form
[ R_E=\exp(-\lambda E). ]
The complete scalar reward combines human pose, interaction distance, object pose, contact, and energy terms. The contact-energy penalty discourages abrupt forces and jitter.
Recovering Missing Hand Interaction
OMOMO and BEHAVE often contain averaged or flattened hand poses. When a fingertip or palm lies close to an object, InterMimic activates a hand contact target and lets PPO discover a workable grasp under hand range-of-motion constraints. This recovers functional contact for the paper’s relatively low-dexterity tasks; it does not reconstruct precise captured finger articulation.
Physical State Initialization (PSI)
Reference State Initialization (RSI) starts an episode from a random MoCap frame. With faulty contacts, that frame may already be dynamically unrecoverable: the object drops immediately, or a floating hand cannot reach it in time.
PSI maintains an initialization buffer containing raw reference states and successful simulation states from earlier rollouts. After a rollout, high-return trajectory states enter the FIFO buffer. Future episodes can therefore begin from states that are close to the intended phase and physically achievable. PSI preserves access to later motion phases without repeatedly initializing the simulator inside broken reference configurations.
Interaction Early Termination (IET)
Standard motion-imitation termination checks character deviation and unwanted ground contacts. IET adds HOI-specific failure tests. An episode ends when:
- mean object-point deviation exceeds 0.5 m;
- the weighted joint-to-object distance differs from the reference by more than 0.5 m; or
- a required body-object contact is absent for over 10 consecutive frames.
This keeps PPO from spending rollout budget after the interaction has already failed.
Stage 2: Two Forms of Distillation
After teacher training, the teachers are frozen and rolled out online. Their trajectories provide both states (s^{(T)}) and actions (a^{(T)}).
Reference Distillation
Teacher simulation states replace noisy MoCap states when constructing the student’s goal and reward. This transfers corrected contacts, feasible hand placement, a canonical embodiment, and physically consistent object motion. The student learns from references that its simulator can actually realize.
Policy Distillation and RL Fine-Tuning
The teacher actions supervise the student through behavior cloning and DAgger. Training gradually shifts toward PPO:
[ \mathcal{L}t = w_t\mathcal{L}{\mathrm{PPO}} + (1-w_t)\lVert a^{(S)}-a^{(T)}\rVert, \qquad w_t\uparrow 1. ]
Early supervision provides efficient skill acquisition. Later PPO updates optimize the actual interaction reward and resolve ambiguous supervision, such as different teachers choosing different controls for similar observed states. The critic is trained throughout this transition.
The teachers are three-hidden-layer MLPs with widths 1024, 1024, and 512. The student uses a three-layer Transformer encoder with four heads, hidden size 256, and feed-forward size 512. All controllers run at 30 Hz in Isaac Gym.
Experiments
InterMimic draws on OMOMO, BEHAVE, HODome, IMHD, and HIMO. OMOMO is the main large-scale benchmark, with 15 objects and about 10 hours of motion. The authors train 17 subject-specific teachers, reserve subject 14 as the test subject, and discard a small amount of severely corrupted data that the teachers cannot repair.
The metrics measure success rate, imitation duration before IET, human joint-position error (E_h), and object point-position error (E_o).
Teacher-Level Correction
On one BEHAVE subject interacting with a yoga mat, the full teacher substantially improves over SkillMimic:
| Method | Duration ↑ | (E_h) (cm) ↓ | (E_o) (cm) ↓ |
|---|---|---|---|
| SkillMimic | 12.2 | 7.2 | 13.4 |
| InterMimic without IET | 40.3 | 6.7 | 9.9 |
| InterMimic without PSI | 36.1 | 6.6 | 10.2 |
| InterMimic | 42.6 | 6.4 | 9.2 |
Both PSI and IET contribute. The qualitative examples show the controller correcting floating contacts, misplaced hands, and implausible rotations of symmetric objects.
Student Scaling and Generalization
Direct PPO on raw MoCap reaches only 9.6% success on the held-out OMOMO subject. The complete Transformer student reaches 98.1%, with 176.5 seconds of tracked duration, 5.9 cm human error, and 11.3 cm object error. The full MLP student scores 95.5% on the same split, while the Transformer has the best test success and improves results on the InterDiff-generated references.
The broader table adds useful nuance. A ten-times-heavier-object test remains hard: the Transformer student’s success falls to 56.8%. Text-conditioned HOI-Diff references are harder still, with 12.5% success. On future interactions generated by InterDiff, success reaches 76.7%. These experiments support zero-shot compatibility with new references and geometries, while also showing that “universal” does not mean uniformly solved.
What the Ablations Say
The ablations give each training component a distinct role:
- PSI makes later and contact-sensitive motion phases reachable during training.
- IET removes rollouts whose human-object relationship has already broken.
- Reference distillation produces a large gain on the held-out body shape because the teachers have already retargeted all references to one embodiment.
- Policy distillation supplies a scalable supervised starting point.
- PPO fine-tuning resolves teacher conflicts and improves beyond demonstration averaging.
- The Transformer benefits from denser future keyframes and temporal modeling, especially on held-out and generated references.
This separation is a major strength of the paper: data repair, optimization efficiency, and policy capacity are handled by different mechanisms with corresponding measurements.
Limitations
The supplementary material identifies several concrete boundaries:
- Teachers fail when a reference contains too many severe errors, such as a flipped hand. Those clips are filtered before student training.
- The simulator can produce unnatural object support through penetration. Contact-energy penalties reduce the problem but do not eliminate it.
- Hand recovery works for the evaluated tasks and may be insufficient for dexterous manipulation requiring detailed finger motion.
- Soft-body interactions, such as carrying a bag by its strap, are excluded because the simulator lacks suitable support.
- Generalization still depends on coverage. More diverse objects and motions should improve performance, and the heavy-object and HOI-Diff results leave substantial room for progress.
The real-humanoid demonstrations are best read as evidence that the formulation can transfer to a Unitree G1 embodiment. The paper’s main quantitative claims concern physics simulation, so broad real-world robustness remains open.
Takeaways
InterMimic offers a reusable recipe for learning control from imperfect motion data:
- use small experts to turn noisy kinematic demonstrations into feasible physical trajectories;
- distill both the corrected targets and the expert actions;
- transition from imitation to reward optimization as the student becomes competent;
- encode geometry and contact explicitly so one controller can span object shapes and interaction modes.
The conceptual shift is subtle and valuable. The MoCap sequence becomes a task specification with uncertain measurements. Physics-based RL is allowed to modify the path while preserving the interaction, and the corrected simulations become the dataset used for scale.