Assessing Usefulness and Reliability of the DIRE Checklist to Evaluate Equity in Public Health Dashboards: Reliability Study (Preprint)

BACKGROUND The COVID-19 pandemic revealed considerable public health equity challenges. Although dashboards remained a critical source of health information for decision makers, key gaps in equity-based decision support were revealed. In prior research, the Dashboard Instrument to Review Equity (DIRE) checklist was developed and refined to be a practical tool for public health departments to use in evaluating equity-based decision support mechanisms in their dashboards. OBJECTIVE The objective of this study was to measure the reliability of the DIRE checklist as a practical instrument for data practitioners to evaluate dashboards. METHODS Multiple steps were completed to measure the reliability of the DIRE checklist. At first, specific parameters were used to select a sample of U.S. state-based dashboards (n=26). Two reviewers then applied the DIRE checklist to each dashboard. Three different methods were used to assess DIRE checklist’s reliability: inter-rater agreement, Intraclass Correlation Coefficients (ICC), and the cosine similarity. These methods were applied to different scores captured by the DIRE checklist (i.e., an overall checklist score, category-specific scores, and a subjective reviewer score). Finally, qualitative notes captured by the reviewers were assessed using a thematic analysis approach. RESULTS A total of 26 dashboards were evaluated using the DIRE checklist by two reviewers. The overall percentage comparison for the overall checklist scores were 31.7% and 41.8%, resulting in a percentage agreement of 72.7% among reviewers. Most of the categories scored less than 50% by the reviewers; however, these scores showed substantial to high agreement across most categories within each DIRE checklist category, with absolute percentage agreement being more than 55% for most categories. The ICC of the category-specific scores resulted in varying levels of agreement across different categories (i.e., ICC3k ranging from 0 to 0.74). The average subjective reviewer score was 9.24 out of a maximum score of 13. CONCLUSIONS The study results confirm an acceptable level of reliability for the DIRE checklist to measure health equity representation in public health dashboards. The DIRE checklist can assist public health data practitioners in integrating equity-focused functionalities and decision support in dashboard design and development.