M2EEG-VR: Validation of EEG Visualization and Sonification for the Detection of Neonatal Seizures on a Virtual Reality Platform

Electroencephalography (EEG) is a noninvasive tool used by healthcare professionals to measure brain electrical activity. EEG analysis can indicate various anomalies linked to different brain pathologies, including seizures. Traditionally, the analysis is confined to two-dimensional displays and relies exclusively on the visual modality, limiting a comprehensive overview. EEG analysis through visualisation is challenging and time-consuming, and artificial intelligence (AI) is increasingly used to aid the process of seizure detection. However, the educational value of AI-assisted seizure detection models depends on the explainability of the underlying models. Explainable AI can help learners understand the features and patterns associated with seizure detection and also support informed use of AI-based decision support systems. M2EEG-VR leverages the focus and immersive capabilities of virtual reality (VR) with the aim of developing a multi-modal platform for EEG seizure detection analysis with a human-in-the-loop. The ability to understand EEG and seizure patterns is key to addressing and effectively treating many neurological conditions. Neonatal seizure detection is particularly challenging where seizure patterns are subtle and context dependent. This study advances toward multi-modal analysis by encoding EEG signals into auditory representations using AI that aids in the acoustic detection of the presence of neonatal seizures in EEG. The platform also introduces a 3D brain model with a spatial mapping of seizure regions. In a user study (N = 20, 4 prior EEG experience, 16 no prior EEG experience), participants achieved higher seizure detection accuracy in the combined visual and auditory condition (mean = 7.6 ± 1.2) than in visual-only or audio-only modes. These preliminary findings suggest that a multi-modal environment may improve the accuracy of detection. However, further controlled studies are needed to ascertain the performance benefits. Usability was rated excellent (SUS = 83 ± 11), and task load remained moderate (NASA-TLX = 36.6). The findings suggest that VR multi-modal interaction can reduce cognitive load and enhance the explainability of complex EEG data in a focused virtual environment. The analysis of the diagnostic accuracy showed that participants without prior EEG knowledge performed similarly across all modalities to those with prior EEG knowledge. This implies that the accessibility barrier is reduced for novice users using the tool for the EEG review/detection task. This, together with high usability and moderate task load scores, indicates that the tool may be suitable for medical training applications. A multi-modal EEG in VR may prove useful in education and also be used as a test bench to further explore AI with human-in-the-loop paradigms for seizure detection.