Highlights:
- Researchers introduce a two-stage system enabling robots to mimic human actions from AI-generated videos.
- New pipeline converts noisy synthetic video data into structured 4D human motion representations.
- GenMimic, a physics-aware reinforcement learning policy, provides stable, realistic motion tracking on humanoid robots.
- A new benchmark dataset, GenMimicBench, is released to assess zero-shot generalization and robustness.
TLDR:
A team of researchers led by James Ni and colleagues introduced GenMimic, a novel physics-aware AI system that enables robots to learn and perform human actions directly from AI-generated videos, paving the way for more autonomous and adaptable humanoid control systems.
A groundbreaking new study titled ‘From Generated Human Videos to Physically Plausible Robot Trajectories’ is redefining the frontier between synthetic video generation and embodied robotics. Authored by James Ni, Zekai Wang, Wei Lin, Amir Bar, Yann LeCun, Trevor Darrell, Jitendra Malik, and Roei Herzig, the paper addresses one of the most challenging questions in artificial intelligence: how can a humanoid robot execute complex, realistic human motions derived from AI-generated videos? With the steady rise of diffusion-based video generation models capable of producing lifelike human motion across diverse contexts, this research takes a critical step toward translating visual imagination into physical action.
The team’s innovation lies in a two-stage framework. First, the method transforms raw video pixels into a 4D human motion representation—enabling spatial and temporal coherence. This intermediate model captures the structural essence of movement and aligns it with the morphology of a target humanoid robot. In the second stage, the authors present GenMimic, a physics-aware reinforcement learning (RL) policy. This system operates by conditioning actions on 3D keypoints while leveraging symmetry regularization and keypoint-weighted tracking rewards to ensure consistent, stable movement. The result is a policy that can replicate the intended human-like motions even when the source video contains noise or visual distortion.
To validate their approach, the authors developed a synthetic motion dataset called GenMimicBench, composed of numerous human-motion sequences generated from leading video generation models. This dataset serves as a benchmark for evaluating zero-shot generalization and policy robustness. In simulation experiments and real-world tests, GenMimic consistently outperformed strong baselines and demonstrated smooth, stable trajectories on the Unitree G1 humanoid robot—all without additional fine-tuning. This achievement signals a significant leap toward using generative AI as a high-level planner for robot control, potentially leading to robots that can autonomously synthesize, interpret, and perform new behaviors directly from visual imagination.
The research highlights a transformative path where deep learning, physics-aware modeling, and embodied AI converge. By making generated visual data actionable for humanoid systems, the study opens new possibilities for robotics applications in automation, entertainment, virtual training, and human-robot collaboration. As video generation models continue to evolve, frameworks like GenMimic may help translate even the most imaginative visual scenes into physically sound robotic movements.
Source:
Source:
Original research paper: Ni, J., Wang, Z., Lin, W., Bar, A., LeCun, Y., Darrell, T., Malik, J., & Herzig, R. (2025). From Generated Human Videos to Physically Plausible Robot Trajectories. arXiv:2512.05094 [cs.RO]. https://arxiv.org/abs/2512.05094