Computerized predictive models using logistic regression achieved a higher AUC of 0.775 for detecting undiagnosed diabetes compared to 0.699 for the best classical paper-and-pencil based test.
Cross-Sectional (n=14,207)
Do advanced predictive models improve risk estimation performance for detecting diabetes and pre-diabetes compared to major online diabetes risk calculators?
Advanced statistical predictive models significantly outperform classical paper-and-pencil based online diabetes risk calculators in detecting undiagnosed diabetes and pre-diabetes.
Absolute Event Rate: 0.775% vs 0.699%
Classical paper-and-pencil based risk assessment questionnaires are often accompanied by the online versions of the questionnaire to reach a wider population. This study focuses on the loss, especially in risk estimation performance, that can be inflicted by direct transformation from the paper to online versions of risk estimation calculators by ignoring the possibilities of more complex and accurate calculations that can be performed using the online calculators. We empirically compare the risk estimation performance between four major diabetes risk calculators and two, more advanced, predictive models. National Health and Nutrition Examination Survey (NHANES) data from 1999-2012 was used to evaluate the performance of detecting diabetes and pre-diabetes. American Diabetes Association risk test achieved the best predictive performance in category of classical paper-and-pencil based tests with an Area Under the ROC Curve (AUC) of 0.699 for undiagnosed diabetes (0.662 for pre-diabetes) and 47% (47% for pre-diabetes) persons selected for screening. Our results demonstrate a significant difference in performance with additional benefits for a lower number of persons selected for screening when statistical methods are used. The best AUC overall was obtained in diabetes risk prediction using logistic regression with AUC of 0.775 (0.734) and an average 34% (48%) persons selected for screening. However, generalized boosted regression models might be a better option from the economical point of view as the number of selected persons for screening of 30% (47%) lies significantly lower for diabetes risk assessment in comparison to logistic regression (p < 0.001), with a significantly higher AUC (p < 0.001) of 0.774 (0.740) for the pre-diabetes group. Our results demonstrate a serious lack of predictive performance in four major online diabetes risk calculators. Therefore, one should take great care and consider optimizing the online versions of questionnaires that were primarily developed as classical paper questionnaires.
Štiglic et al. (Wed,) conducted a cross-sectional in Undiagnosed type 2 diabetes and pre-diabetes (n=14,207). Computerized predictive models (Logistic Regression and Generalized Boosted Models) vs. Four major online diabetes risk calculators (ADA, CANRISK, LRA, AUSDRISK) was evaluated on Area Under the ROC Curve (AUC) for detecting undiagnosed diabetes. Computerized predictive models using logistic regression achieved a higher AUC of 0.775 for detecting undiagnosed diabetes compared to 0.699 for the best classical paper-and-pencil based test.