The quality of diagnosis and triage advice provided by free online symptom checkers and apps in Australia

OBJECTIVES: To investigate the quality of diagnostic and triage advice provided by free website and mobile application symptom checkers (SCs) accessible in Australia. DESIGN: 36 SCs providing medical diagnosis or triage advice were tested with 48 medical condition vignettes (1170 diagnosis vignette tests, 688 triage vignette tests). MAIN OUTCOME MEASURES: Correct diagnosis advice (provided in first, the top three or top ten diagnosis results); correct triage advice (appropriate triage category recommended). RESULTS: The 27 diagnostic SCs listed the correct diagnosis first in 421 of 1170 SC vignette tests (36%; 95% CI, 31-42%), among the top three results in 606 tests (52%; 95% CI, 47-59%), and among the top ten results in 681 tests (58%; 95% CI, 53-65%). SCs using artificial intelligence algorithms listed the correct diagnosis first in 46% of tests (95% CI, 40-57%), compared with 32% (95% CI, 26-38%) for other SCs. The mean rate of first correct results for individual SCs ranged between 12% and 61%. The 19 triage SCs provided correct advice for 338 of 688 vignette tests (49%; 95% CI, 44-54%). Appropriate triage advice was more frequent for emergency care (63%; 95% CI, 52-71%) and urgent care vignette tests (56%; 95% CI, 52-75%) than for non-urgent care (30%; 95% CI, 11-39%) and self-care tests (40%; 95% CI, 26-49%). CONCLUSION: The quality of diagnostic advice varied between SCs, and triage advice was generally risk-averse, often recommending more urgent care than appropriate.