Circular validation in diagnostic research: re-examination of a widely cited study on primary hemostasis screening

Introduction A 2004 study reported high sensitivities for laboratory assays used in the preoperative detection of primary hemostasis disorders and has since been widely cited. A re-evaluation of its methodological validity is performed. Methods The study design and calculations were reviewed focusing on the definition of “impaired hemostasis” and the derivation of diagnostic performance. A synthetic dataset of random values was generated, and assay-specific cut-offs were defined to reproduce the proportions of positive results reported in the original study. The detection of “impaired hemostasis” was then calculated using a self-referential approach. Results Three sets of performance figures are provided in the 2004 study. In one set, the results of the evaluated tests were used to define the reference standard against which their own sensitivities and specificities were then calculated (incorporation bias). For one assay a second set of performance figures was reported. The index test had been performed first in all patients; the further diagnostic work-up and the classification of clinical findings depended on its result (verification bias). The cut-off used for the index test was determined post hoc by ROC analysis, which often inflates performance metrics. The sensitivity and specificity values reported for the individual questionnaire items are likewise affected by incorporation bias. The synthetic dataset reproduced the reported detection rates despite consisting of random numbers. Conclusion The widely cited results of the 2004 study cannot be used as evidence of diagnostic accuracy, because the reference standard was established using the very tests whose performance was then evaluated against it.