Robotic cloth manipulation suffers from a lack of standardized benchmarks and shared datasets for evaluating and comparing different approaches. To address this, we created a benchmark and organized the ICRA 2024 Cloth Competition, a unique head-to-head evaluation focused on grasp pose selection for in-air robotic cloth unfolding. Eleven teams participated in the competition, utilizing the publicly released dataset of 500 real-world robotic grasp attempts for cloth unfolding and employing diverse approaches to generate in-air unfolding grasps. Analysis of the competition results revealed insights about the trade-off between grasp success and coverage, the surprisingly strong achievements of hand-engineered methods and a significant discrepancy between competition performance and prior work, underscoring the importance of independent, out-of-the-lab evaluation in robotic cloth manipulation. We also expanded the dataset with 176 competition evaluation trials, resulting in a dataset of 679 unfolding demonstrations across 34 garments. This dataset is a valuable resource for developing and evaluating grasp selection methods, particularly for learning-based approaches. We hope that the benchmark, dataset, and competition results can serve as a foundation for future benchmarks and drive further progress in data-driven robotic cloth manipulation.
Gusseme et al. (Wed,) studied this question.