Machine learning algorithms predicting mental wellbeing from smartphone data provided no significant improvement over simply guessing a user's average state, demonstrating that comparisons to population baselines create false optimism.
Meta-Analysis (n=46)
Does comparing machine learning algorithms to personal baselines rather than population baselines change the perceived accuracy of predicting mental wellbeing?
Evaluating medical machine learning algorithms against population baselines rather than personal baselines leads to falsely optimistic performance estimates.
A new trend in medicine is the use of algorithms to analyze big datasets, e.g. using everything your phone measures about you for diagnostics or monitoring. However, these algorithms are commonly compared against weak baselines, which may contribute to excessive optimism. To assess how well an algorithm works, scientists typically ask how well its output correlates with medically assigned scores. Here we perform a meta-analysis to quantify how the literature evaluates their algorithms for monitoring mental wellbeing. We find that the bulk of the literature (∼77%) uses meaningless comparisons that ignore patient baseline state. For example, having an algorithm that uses phone data to diagnose mood disorders would be useful. However, it is possible to explain over 80% of the variance of some mood measures in the population by simply guessing that each patient has their own average mood-the patient-specific baseline. Thus, an algorithm that just predicts that our mood is like it usually is can explain the majority of variance, but is, obviously, entirely useless. Comparing to the wrong (population) baseline has a massive effect on the perceived quality of algorithms and produces baseless optimism in the field. To solve this problem we propose "user lift" that reduces these systematic errors in the evaluation of personalized medical monitoring.
DeMasi et al. (Tue,) conducted a meta-analysis in Mental wellbeing (stress and happiness) (n=46). Personalized machine learning models vs. Personal baselines (guessing the individual's average state) and population baselines was evaluated on User lift (improvement of algorithm predictions over the personal baseline). Machine learning algorithms predicting mental wellbeing from smartphone data provided no significant improvement over simply guessing a user's average state, demonstrating that comparisons to population baselines create false optimism.