Prototyping Virtual Reality Interactions in Medical Simulation Employing Speech Recognition

Medical simulation has seen the adoption of virtual reality as a tool to expose learners to scenarios otherwise impossible in real life. In medical simulation, patient interactions are practiced employing standardized patients (i.e., actors) and most recently artificial intelligence driven avatars. Typical avatar interactions found in virtual simulators rely on buttons, text boxes, and drop-down menus to enable the interactions. While a similar approach is also employed in virtual reality, virtual keyboard inputs and point and click tasks on graphical user interface items using a mouse can be challenging and lead to increased cognitive loads. In this paper, we outline an application for virtual avatar communication by employing speech recognition and gestures to convey non-verbal cues during an eye fundus examination. Our goal is to understand the face validity of speech and gesture driven interactions. A demo was conducted at an open house were attendees tried the scenario and provided their opinions on the experience. From this preliminary experience, we were able to identify challenges and opportunities when deploying speech-based interactions outside a controlled laboratory space.