Validation Studies in Epidemiologic Research: Caution When Estimating Sensitivity and Specificity in Internally Validated Claims Data

Administrative claims databases are increasingly used in psychiatric epidemiology because they enable large-scale, longitudinal analyses. In Taiwan, the National Health Insurance Research Database (NHIRD) covers over 99% of the population and is therefore a cornerstone data source for population-based research [1]. But despite a substantial body of published psychiatric epidemiology research using the NHIRD, the diagnostic accuracy of claims-based codes for mental disorders remains unclear. The Validation of Psychiatric Diagnoses in the National Health Insurance Research Database Given this situation, Wang et al. [2] make an important contribution by doing the first validation of NHIRD psychiatric diagnoses for mental disorders. Their work established a standardized chart review procedure with a stratified sample of 727 psychiatric inpatients. Although the results report ranges of sensitivity and positive predictive value (PPV) for psychotic and affective disorders, PPV estimates are robust, whereas sensitivity estimates were likely biased due to the sampling design. Sampling Strategies in Validation Study Three common sampling strategies for internal validation studies have been described, along with the distinct parameters estimable under each design [3]. In the first design, the sample is selected based on the error-prone exposure status; in the second, it is selected based on the gold-standard exposure-based measure; and in the third, a simple random sample is drawn. In the first design, investigators can validly estimate PPV and negative predictive value (NPV) but not sensitivity or specificity (at least not directly); in the second, sensitivity and specificity can be estimated but predictive values cannot; and in the third, sensitivity, specificity, and predictive values are estimable. Estimate–design Mismatch: Sensitivity under Claims-based Sampling Viewed through this framework, the Wang et al.’s study [2] is most consistent with the first design – selection conditional on the error-prone (claims-based) diagnosis from the NHIRD. In their sampling frame, eligible inpatients were first identified from the hospital’s integrated database based on discharge diagnostic categories recorded in claims, and then, a feasibility-driven, stratified sample was drawn. The validity indices evaluated in the study were the PPV, the conditional probability of a diagnosis in the review-based diagnosis given that it appeared in the claims data, and the sensitivity, the conditional probability of a diagnosis recorded in the NHIRD given that it appeared in the review-based diagnosis. But Wang and colleagues [2] asserted to have overlooked that, under their sampling design, the sensitivity estimate is not directly valid. When the validation sample is selected within strata defined by the error-prone, claims-based classification (e.g., “exposed” vs. “unexposed”), the sampling procedure alters the marginal exposure prevalence in the validation sample relative to the source population. As a result, predictive values estimated conditional on the classified status (PPV/NPV) remain unbiased because they do not rely on the marginal prevalence, whereas sensitivity and specificity are biased. This is illustrated in Panels A and B of Table 1, adapted from Fox et al. in 2020 [3], which specifies the calculation procedures. However, because predictive values depend on the true prevalence of the exposure, and Wang et al.’s study [2] was done using participants from the National Taiwan University Hospital Integrated Medical Database, the generalizability of the reported PPVs is limited if the prevalence of mental disorders differs significantly from that of other hospitals or community clinics.Table 1: Impact of error-prone exposure-based design on validity estimates and the implementation of bias-corrected calculationsCorrecting Sensitivity and Specificity under Error-prone Exposure-based Sampling Although Fox and colleagues noted that, under error-prone exposure-based design, sensitivity and specificity can be estimated indirectly, they did not provide a detailed procedure [3]. Here, I provide a simple method to indirectly estimate sensitivity and specificity under this sampling design. The method requires the observed population prevalence of exposure (Panel C of Table 1). Because PPV and NPV are unbiased under this design, combining the sample PPV and NPV with the observed population prevalence of exposed and unexposed individuals yields the bias-corrected 2 × 2 cell counts (Panel D of Table 1), from which unbiased estimates of sensitivity and specificity can be derived. Because Wang et al. [2] did not report the observed population prevalence of mental disorders and NPV, I cannot directly adjust sensitivity for their study. In addition to this method to correct sensitivity and specificity, methods that use external validation studies can be applied [4]. One can adjust for bias in the results reported by Wang et al. [2] by incorporating assumptions regarding the NPV and the prevalence of mental disorders in the NHIRD, or by applying external data pertaining to these parameters. Biased Direction of Sensitivity in Wang et al.’s study [2] Although I cannot directly adjust sensitivity for Wang et al.’s study [2], it is possible to infer the biased direction of sensitivity. Given the prevalence of mental disorders and their underdiagnosis in Taiwan [5], sensitivity is expected to be relatively low and specificity to be high – approaching 1 – implying a low false-positive rate and (similar to Panel A of Table 1). Meanwhile, because sampling is based on the error-prone exposure-based measure (observed mental disorders in the national registered database), the numbers of true positives and false positives are inflated. Because sensitivity is calculated as true positives divided by the sum of true positives and false negatives, this sampling scheme yields an overestimation of sensitivity. For disorders characterized by relatively high specificity and low sensitivity, sensitivity tends to be overestimated and specificity underestimated under an error-prone, exposure-based validation design – a pattern evident in Panels A and B of Table 1 for sensitivity (0.76 to 0.83) and specificity (0.96 to 0.94). Therefore, in Wang et al.’s study, the reported sensitivity for psychotic and affective disorders is likely overestimated [2]. Conclusion Although administrative claim databases, such as Taiwan’s NHIRD, enable large-scale psychiatric epidemiology, assessing the validity of claims-based diagnostic codes remains essential. Wang et al. [2], in 2025, did a validation study reporting generally robust PPV for psychotic and affective disorders, but their sensitivity estimates are not directly interpretable. This misunderstanding about reporting sensitivity and specificity when the cohort is defined by the error-prone claims-based classification has been noted across many studies and settings. I present an illustration and a corrective method to address this issue and help researchers avoid misleading estimates in subsequent validation studies. Author’s Contributions Conceptualization, Methodology, Software, Validation, Formal analysis, Investigation, Resources, Data curation, Writing – original draft preparation, Writing – review & editing, Y.L.C. The author has read and agreed to the final version of the manuscript. Data Sharing Statement Data sharing is not available to this letter-to-the-editor because no datasets were generated and/or tested in writing this letter. Financial Support and Sponsorship The author denies receiving any financial support in writing this letter. Conflicts of Interest The author reports no conflicts of interest in writing this letter.