VLM-based multi-agent architecture for multifunctional robotic health attendant (RHA) using phase-prompt task design

Introduction Healthcare robots are increasingly needed to support clinicians whose work involves diverse tasks and frequent task switching. However, most healthcare robots developed to assist clinicians either perform a single function or implement multiple functions as fixed task pipelines that are difficult to adapt to different patients, rooms, or care teams. This study presents a vision language model (VLM) based multiagent architecture for a multifunctional Robotic Health Attendant (RHA). Methods In this architecture, each task can be designed by writing natural language phase specifications that describe how available tools should be used during task execution. A main agent interprets natural language requests from clinicians or patients and routes each request to the appropriate task agent. During execution, the VLM uses language, visual input, and contextual information to select tools and interpret their results. A shared Model Context Protocol (MCP) tool layer provides elemental robot functions such as navigation, manipulation, and perception across tasks. The architecture was implemented on a TIAGo Pro robot and evaluated in an unoccupied patient room across five tasks including object delivery, documentation, visual monitoring, patient education, and medication reminders. Results The case study demonstrated the feasibility of the proposed architecture as a reasoning core for a multifunctional healthcare robot by executing multiple tasks and adapting task behavior by editing natural language phase specifications. The comparison across three VLMs showed that longer tool-call sequences led to skipped steps and, in some cases, false reports of task success, which is a key reliability concern for healthcare robots performing long, multi-step tasks. Discussion The study also identified technical limitations and future research needs that must be addressed to make the architecture more robust in realistic care settings.