Applying Large Language Models to Enhance Dialogue and Communication Analysis for Adaptive Team Training

Abstract Adaptive training environments that can analyze team communication content and provide remediation to facilitate team coordination offer great potential for enhancing adaptive training systems for teams. Developing computational models that can perform robust team communication analytics based on small datasets is challenging. Large language models (LLMs) offer significant potential to address these challenges and enhance dialogue act classification performance using zero-shot and few-shot learning. This paper evaluates LLMs against previous state-of-the-art methods, with an emphasis on dialogue act recognition performance and error analysis for identifying frequently misclassified instances. Results from a small team communication dataset indicate that zero-shot LLMs, particularly those utilizing GPT-4 and refined through robust prompt engineering, achieve significant classification performance improvements in dialogue act recognition compared to previous state-of-the-art transformer-based models fine-tuned with team communication data. Error analysis shows that the prompt refinements, especially those aimed at clarifying confusion between dialogue acts, result in superior recall rates for challenging dialogue act labels by effectively handling complex dialogue scenarios and ambiguities within communication data. Our transformer-based framework demonstrates its effectiveness in achieving high accuracy rates in dialogue act recognition with minimal training data, underscoring its potential to enhance team training programs by providing adaptive feedback. This approach paves the way for developing AI-enabled training systems that can adapt to the dynamic communication styles of different teams.