Estimating Psychometric Reliability of Machine-Learning-Based Fluid Reasoning Scores Derived from Smartphone Data

Psychological assessment increasingly extends beyond standardized testing situations to behavioral data collected in everyday life. Machine-learning models can be trained on behavioral features derived from smartphone records to predict individual differences in psychological traits. However, only a few studies in this field evaluate psychometric quality of ML-based scores beyond convergent validity (predictive accuracy). Smartphone sensing data consist of large sets of behavioral features that are not constructed according to a theory-driven measurement model, and there is currently no established standard for quantifying reliability in this context. This project aims to compare several partition-based reliability estimates for ML-based fluid reasoning scores derived from smartphone data, investiges the effect of temporal distance between data partitions, and examines how the different reliability estimates relate to predictive accuracy.