Investigating the quality of daily-life symptom prediction at the individual level – An in-depth analysis on the example of psychotic symptom prediction [Author Accepted Manuscript]

Background: Predicting mental health states in daily life could improve the prevention and treatment of mental disorders. Although these models are intended to operate at the individual level, they are typically evaluated using sample-level metrics, potentially obscuring differences in predictive accuracy between individuals. Further, whereas prediction models are often calculated using data from all participants (global models), personalized models trained only on data from the respective individual may better capture person-specific patterns and improve predictive performance. To explore this, we examined the individual-level predictive accuracy of global and personalized models in predicting psychotic experiences in daily life. Method: Fourteen people with schizophrenia-spectrum disorders completed a 12-day Ecological Momentary Assessment (EMA) and answered questionnaires on psychotic experiences and associated distress while wearing ambulatory sensors measuring autonomic arousal. Using random forests, we predicted the presence (yes/no) of auditory verbal hallucinations, paranoia, or associated distress at each EMA assessment based on sensor data from 20 minutes prior to each assessment (heart rate variability, electrodermal activity, movement). Individual-/sample-level predictive accuracy from global models was compared with personalized models. Results: Across all predicted outcomes, sensitivity and accuracy varied greatly among participants for both the personalized (sensitivity=6-90%; accuracy=40-80%) and global models (sensitivity=0-97%; accuracy=44-100%). On average, personalized models yielded lower accuracy (61%) than global models (75%). Conclusion: Predictive accuracy varied considerably across individuals, and sample-level-metrics obscured difficulties in predicting psychotic experiences, highlighting the need for individual-level accuracy assessments. Additionally, heterogeneous predictive performance may indicate that autonomic arousal is not equally relevant for predicting psychotic experiences across individuals.