Multimodal Dataset of Social Skills Training in Natural Conversational Setting

Social Skills Training (SST) is commonly used in psychiatric rehabilitation programs to improve social skills. It is especially effective for people who have social difficulties related to mental illnesses or developmental difficulties. Previous studies revealed several communication characteristics in Schizophrenia and Autism Spectrum Disorder. However, a few pieces of research have been conducted in natural conversational environments with computational features since automatic capture and analysis are difficult in natural settings. Even if the natural data collection is difficult, the data clearly have much better potential to identify the real communication characteristics of people with mental difficulties and the interaction differences between participants and trainers. Therefore, we collected a one-on-one SST multimodal dataset to investigate and automatically capture natural characteristics expressed by people who suffer from such mental difficulties as Schizophrenia or Autism Spectrum Disorder. To validate the potential of the dataset, using partially annotated data, we trained a classifier for Schizophrenia and healthy control with audio-visual features. We achieved over 85% accuracy, precision, recall, and f1-score in the classification task using only natural interaction data, instead of data captured in the specific tasks designed for clinical assessments.