Automated assessment of healthcare de-escalation communication skills in a conversational AI virtual reality simulation

Introduction Digital health training environments increasingly require scalable methods for assessing communication skills in realistic, interactive scenarios. This paper presents a comparative study of automated assessment approaches within a conversational AI virtual reality simulation designed for verbal de-escalation training. Methods A total of 20 nursing students completed the simulation, where their spoken responses were captured and converted to text through a speech-to-text system, and were evaluated across 11 de-escalation competencies. The evaluation compared three fine-tuned deep learning models (BERT, RoBERTa, and XLNet) with a modified RATAS, a rubric-driven framework that leverages large language models. Two experiments were conducted: one using controlled 100-item validation datasets for each section, and another using the real-world responses collected during the user study. Results On the validation datasets, the deep learning models achieved the highest performance, with RoBERTa obtaining an average accuracy of 0.9927, followed by BERT at 0.9900 and XLNet at 0.9790, while RATAS achieved 0.8554. On the student-response dataset, performance decreased for all approaches, with RoBERTa obtaining 0.8590 accuracy, RATAS 0.8181, BERT 0.8000, and XLNet 0.7681. Discussion These results suggest that transformer-based models perform best under controlled conditions, whereas RATAS demonstrated stability in grading across different dataset types. These preliminary findings support the feasibility of integrating deep learning and rubric-driven LLM-based automated assessment approaches for communication and de-escalation training in nursing education.