Colosseum
The COLOSSEUM:
A Benchmark for Evaluating Generalization for Robotic Manipulation
1Universidad Católica San Pablo 2University of Southern California 3University of Washington 4Allen Institute for Artifical Intelligence 5NVIDIA
* Equal contribution
Abstract
To realize effective large-scale, real-world robotic applications, we must evaluate how well our robot policies adapt to changes in environmental conditions. Unfortunately, a majority of studies evaluate robot performance in environments closely resembling or even identical to the training setup.
We present Colosseum, a novel simulation benchmark,with 20 diverse manipulation tasks, that enables systematical evaluation of models across 14 axes of environmental perturbations. These perturbations include changes in color, texture, and size of objects, table-tops, background and object physical properties; we also vary lighting, distractors, and camera pose. Using Colosseum, we compare 5 state-of-the-art manipulation models to reveal that their success rate degrades between 30-50% across these perturbation factors.
When multiple perturbations are applied in unison, the success rate degrades > 75%. We identify that changing the number of distractor objects, target object color, or lighting conditions are the perturbations that reduce model performance the most. To verify the ecological validity of our results, we show that our results in simulation are correlated (R2 = 0.614) to similar perturbations in real-world experiments. We open source code for others to use Colosseum, and also release code to 3D print the objects used to replicate the real-world perturbations. Ultimately, we hope that Colosseum will serve as a benchmark to identify modeling decisions that systematically improve generalization for manipulation.
Leaderboard on THE COLOSSEUM
Perturbations
Perturbations Factors
Perturbation
applied to task
Failure cases
Failure cases for PerAct
Failure cases for RVT
Failure cases for R3M
Failure cases for MVP
Failure cases for VOXPOSER
Reproducibility in real-world experiments
Examples of real-world perturbation results with PerAct
BibTeX
@article{pumacay2024colosseum,
title = {THE COLOSSEUM: A Benchmark for Evaluating Generalization for Robotic Manipulation},
author = {Pumacay, Wilbert and Singh, Ishika and Duan, Jiafei and Krishna, Ranjay and Thomason, Jesse and Fox, Dieter},
booktitle = {arXiv preprint arXiv:2402.08191},
year = {2024},
}