Real robot evaluations need open failure logs

Testing physical robot policies on actual hardware usually hits an economic wall. Running physical trials across varied environments demands fleets of machines and constant manual resetting. As a result, the field relies either on simulation that glides over contact friction or on selective lab runs where only the successful grasps make it into the write-up.

RoboColosseum, a collaborative project partnered with Hugging Face, aims to turn that testing into shared infrastructure. The initiative provides access to more than 100 physical robots, ranging from Unitree G1 humanoids to LeRobot SO-101 and Franka arms across 140 environments. The partners plan to offer 10,000 evaluation hours and publish every rollout on the Hub, including the failed attempts.

Making raw failure traces public is the decision that could change how these models improve. Foundation models for manipulation fail in production precisely because they rarely encounter boundary conditions during training. When an arm slips, catches on an obstacle, or drops an object, curated datasets offer no path back to stability. If an open registry documents exactly how and where real hardware falters across hundreds of setups, the ecosystem gets something much more valuable than another leaderboard score: the negative examples needed to make autonomous systems recover.