Setting and maintaining standards for patient-reported outcome measures: can we rely on the COSMIN checklists?

As test-developers we have often been troubled by published reviews of patient-reported outcome measures (PROMs). Too often minor issues are judged important while other reviews exclude the best measures available. Perhaps this led several groups to make recommendations for evaluating the quality of PROMs. The COnsensus-based Standards for the selection of health Measurement INstruments (COSMIN) checklist is the latest set of recommendations. While reviewing the COSMIN literature and reviews conducted using their recommendations several concerns became apparent. The checklist is not evidence-based, relying on the opinion of researchers experienced in health-related quality of life. PROMs measuring other types of outcomes are inadequately covered by the checklist. COSMIN choose to focus on Classical Test Theory and the checklists are not appropriate for use with PROMs developed using modern measurement. Such an approach only obstructs progress in the field of outcome measurement. The retrospective nature of the evaluations also penalizes new PROMs. While the checklists imply that composite, ordinal level measurement is acceptable, crucial aspects of instrument development and quality are excluded. Reviews based on the COSMIN checklist produce contradictory conclusions and fail to provide evidence to support the recommendations. These problems suggest that the checklists themselves lack reliability and validity. It is also clear that several reviewers lack the expertise to apply the checklists. Researchers require a good grounding in instrument development and psychometrics to produce quality reviews. The science of modern PROM development is still in an early phase. Few available PROMs have sufficient quality, limiting the need for complex reviews. Standards need to be agreed for high quality outcome measurement. The issue is who should set these standards? Most published reviews merely scratch the surface and lack essential detail. All reviews of PROMs should be treated with caution, irrespective of whether the COSMIN checklist was employed.