Human-Integrated Multi-Agent Exploration Using Semantic Communication and Extended Reality Simulation

In hazardous or inhospitable environments such as Mars, exploration and maintenance tasks pose high risks to human operators. Hence, the assistance of mobile robotic systems is required. One of the main applications of outdoor exploration is resource localization. This scenario imposes several challenges: Resources to be found are scarce, and their distribution is unknown. Further, we have limited processing power. Dangerous situations require fast decisions, and thus minimal processing and communication latency. Lastly, the huge amount of information received from the rovers can become quite difficult to interpret by the human operators. To cope with these challenges, we introduce a conceptual framework for the integration of humans with a multi-agent system in the planetary exploration context. The scenario in consideration is that of human-integrated multi-robotic exploration by a team on Mars consisting of rovers and a few astronauts for outdoor sampling and remote operation from within an extraterrestrial habitat. The rover team adapts its behavior to the status of the communication, and with semantic communication, we convey the meaning of the desired exploration data into the data transmission. Then, based on the received semantic information, we visualize the information to provide humans relevant information for decision-making. In order to implement the common framework in a virtual scenario, we combine a human-in-the-loop simulation in extended reality with a rover simulation environment.