Cosmos 3 splits physics simulation from rendering

NVIDIA is splitting interactive world simulation into two distinct approaches with four new 16B parameter Cosmos 3 open checkpoints. Instead of forcing all physical dynamics into a single model, the release separates visual rendering from mechanical simulation. One checkpoint, Sim-Transfer, lets a conventional physics simulator compute contacts, depth, and segmentation boundaries while the neural network generates photorealistic camera feeds. Another checkpoint, Sim-Bimanual, attempts to predict future visual frames directly from raw robot control vectors across 59 joint values.

That split reflects where generative robotics has struggled. Closed-loop action-to-video prediction frequently drifts because small physical errors compound over successive frames, turning an intended grip into visual noise. NVIDIA addresses rollout stability in these models with a four-stage training recipe, using consistency distillation down to four denoising steps and training the student on its own generated history. Even so, handing spatial ground truth to an existing physics engine before generating camera frames remains far more reliable for validating robot control policies than hoping a network memorized inertia.

Releasing these checkpoints with open weights under the OpenMDW license changes the practical equation for robotics teams. Rather than renting closed simulation platforms or relying solely on manual CAD environments, developers can post-train a 16B base model on their own sensor layouts and camera geometries. The real test is measuring whether driving and manipulation policies trained on these distilled rollouts actually transfer cleanly to physical hardware.