Evaluating maps produced by urban search and rescue robots: lessons learned from RoboCup

This paper presents the map evaluation methodology developed for the Virtual Robots Rescue competition held as part of RoboCup. The procedure aims to evaluate the quality of maps produced by multi-robot systems with respect to a number of factors, including usability, exploration, annotation and other aspects relevant to robots and first responders. In addition to the design choices, we illustrate practical examples of maps and scores coming from the latest RoboCup contest, outlining strengths and weaknesses of our modus operandi. We also show how a benchmarking methodology developed for a simulation testbed effortlessly and faithfully transfers to maps built by a real robot. A number of conclusions may be derived from the experience reported in this paper and a thorough discussion is offered.