A systematic review assessing the state of analytical validation for connected, mobile, sensor-based digital health technologies

Background Despite the recent proliferation of digital health technologies (DHTs), there is a lack of formal, industry-wide standards to evaluate the performance of the product’s algorithm in terms of its ability to measure, detect, or predict a clinical state. The advancement and successful use of DHTs in medicine requires that all stakeholders – clinicians, patients, payers, regulators, pharmaceutical companies, and the medical products industry – have a common understanding of what it means when a DHT has been analytically validated. Objective We conducted a systematic review to assess the state of the science on analytical validation for DHTs, using the criteria established by the V3 Framework and EVIDENCE checklist. Methods The systematic review was conducted according to the PRISMA guidelines. A multi-tier PubMed search identified studies published between April 15, 2020, and April 15, 2023, on analytical validation of DHTs; thereafter, each paper was assessed against the EVIDENCE checklist items specific to analytical validation. All studies were required to report quantitative data collected from a connected, mobile sensor product for the measurement, diagnosis, and/or treatment of a behavioral or physiological function, and compare the outcome measures to an established reference standard. Results Of the 1201 papers identified in the literature search, we identified 303 reporting the results of a DHT analytical validation study. The most prevalent therapeutic areas of focus were neurological (26%), cardiovascular (18%), and sleep conditions (17%). Health outcome categories most frequently captured by DHTs were gait (15%), heart rate/rhythm (15%), blood pressure and/or arterial stiffness (11%), sleep staging (10%), and mobility (9%). Only 208 papers (69%) reported all components of the EVIDENCE checklist focused on analytical validation, with the exception of software version and race/ethnicity, thereby meeting our definition of high-quality evidence reporting. Conclusion We are encouraged by the emerging literature evaluating whether outcome measures assessed by DHTs adequately reflect the physiological or behavioral parameter of interest; however, the quality of reporting is not yet sufficient to ensure the advancement of digital clinical measures that are fit-for-purpose for all members of a defined population and eliminate the need for redundant studies. We recommend that journals publishing analytical validation studies require the use of the EVIDENCE checklist as a reporting standard for these manuscripts.