Why the study?
Correct identification of dialysis-dependent ESRD is crucial for risk-adjusted quality reporting and revascularization outcome analyses, but STS and NCDR instruments had not been validated.
Do the STS and NCDR data instruments accurately identify dialysis-dependent ESRD in patients undergoing CABG or PCI compared to the USRDS?
Observational (n=126,165)
Yes
Do the STS and NCDR data instruments accurately identify dialysis-dependent ESRD in patients undergoing CABG or PCI compared to the USRDS?
Answers are grounded in this paper and cite their sources.
STS and NCDR databases have high sensitivity and specificity but low positive predictive value for identifying dialysis-dependent ESRD, though this misclassification has minimal impact on overall procedural risk prediction models.
Cardiovascular revascularization procedural data are routinely collected to assess procedural quality, create risk adjustment tools, and assess outcomes in populations not well-studied in clinical trials. The National Cardiovascular Data Registry (NCDR) and the Society for Thoracic Surgeons (STS) data sets are widely used for these purposes,1Weintraub W.S. Grau-Sepulveda M.V. Weiss J.M. et al.Comparative effectiveness of revascularization strategies.N Engl J Med. 2012; 366: 1467-1476Crossref PubMed Scopus (461) Google Scholar, 2Welke K.F. Peterson E.D. Vaughan-Sarrazin M.S. et al.Comparison of cardiac surgery volumes and mortality rates between the Society of Thoracic Surgeons and Medicare databases from 1993 through 2001.Ann Thorac Surg. 2007; 84: 1538-1546Abstract Full Text Full Text PDF PubMed Scopus (48) Google Scholar and their data collection instruments are utilized as “off-the shelf” tools to facilitate standardized data capture. Correct identification of dialysis-dependent end-stage renal disease (ESRD) is particularly important because ESRD is a potent risk factor for cardiovascular mortality and procedural complications.3US Renal Data SystemUSRDS 2013 Annual Data Report: Atlas of End-Stage Renal Disease in the United States. Bethesda, MD, National Institutes of Health, National Institute of Diabetes and Digestive and Kidney Diseases2013Google Scholar, 4Shahian D.M. O'Brien S.M. Sheng S. et al.Predictors of long-term survival after coronary artery bypass grafting surgery: results from the Society of Thoracic Surgeons Adult Cardiac Surgery Database (The ASCERT Study).Circulation. 2012; 125: 1491-1500Crossref PubMed Scopus (193) Google Scholar, 5Cooper W.A. O'Brien S.M. Thourani V.H. et al.Impact of renal dysfunction on outcomes of coronary artery bypass surgery: results from the Society of Thoracic Surgeons National Adult Cardiac Database.Circulation. 2006; 113: 1063-1070Crossref PubMed Scopus (410) Google Scholar, 6Roques F. Nashef S.A. Michel P. et al.Risk factors and outcome in European cardiac surgery: analysis of the EuroSCORE multinational database of 19030 patients.Eur J Cardiothorac Surg. 1999; 15 (discussion 822-823): 816-822Crossref PubMed Scopus (1464) Google Scholar, 7Society of Thoracic Surgeons. Adult Cardiac Surgery Database. Available at: http://www.sts.org/sts-national-database/database-managers/adult-cardiac-surgery-database. Accessed July 22, 2019.Google Scholar Incorrect identification could impact risk-adjusted quality reporting for cardiac procedures as well as the retrospective analyses widely used to assess revascularization outcomes in dialysis patients. However, to our knowledge, the STS and NCDR instruments for identification of dialysis patients have not been validated. We assessed accuracy of dialysis identification by linking United States Renal Data System (USRDS) data to Massachusetts Data Analysis Center statewide data collected using the STS and NCDR instruments under a legal mandate requiring universal data capture on all patients undergoing coronary artery bypass graft (CABG) and percutaneous coronary intervention (PCI).8Normand SL. Adult Percutaneous Coronary Artery Intervention FY14 Annual Report. In (vol 2018). Massachusetts Data Analysis Center, Boston, MA, 2016.Google Scholar, 9Normand SL. Adult Coronary Artery Bypass Graft Surgery FY14 Annual Report. In (vol 2018). Massachusetts Data Analysis Center, Boston, MA, 2016.Google Scholar We identified 26,317 individuals undergoing CABG, and 99,848 undergoing PCI. The mean age was 66.8 ± 10.5 years in the CABG group, and 64.7 ± 12.6 years in the PCI group. Subjects were primarily White (CABG 89.9%; PCI 88.9%), with 55.1% of CABG and 71.7% of PCI patients admitted with acute coronary syndrome. Many procedures were performed urgently (CABG 62.7%; PCI 45.3%). Emergent or salvage procedures were rare for CABG (2.9%) but not PCI patients (23.9%). Diabetes, heart failure, and hypertension were common (Table 1).Table 1Overall populationVariableCABGaFor CABG, hospital procedural volume is defined as low ≤1418, medium >1418 to ≤2175, high >2175. (N = 26,317)PCIbHospital procedural volume for PCI is defined as low ≤914 cases in total during the study period, medium >914 to ≤4625, high ≥4625. (N = 99,848)N%N%Demographics and insurance Age (mean ± SD), yr66.8 ± 10.564.7 ± 12.6 Male20,01876.0669,09069.20 RaceWhite23,66689.9388,73388.87Black6232.3727702.77Other20287.7183458.36 Insurance payorcEighty-seven (0.33%) CABG patients were missing payor.Private11,44043.4750,14550.22Government14,14653.7546,48246.55Other6442.4532213.23Dialysis status Dialysis identified in the Mass-DAC instruments4311.6414321.43 Dialysis identified through USRDS data2951.129500.95Baseline medical conditions Acute coronary syndrome14,49355.0971,59671.70 Diabetes10,41239.5629,08429.13 Heart failure463117.6011,44611.46 Hypertension22,14584.1575,35475.47 Hypercholesterolemia22,95087.2178,10178.22 Peripheral vascular disease429116.3111,42811.45 Prior myocardial infarction13,18350.0921,61521.65 Prior PCI317912.0815,34115.36 Prior CABG4081.5510,61310.63 CABG or PCI performed at a teaching hospital22,07383.8777,47277.59 Urgent status16,50762.7245,21045.28 Emergent or salvage status7532.8623,88523.92 Cardiogenic shock2170.8220822.09Hospital characteristics Hospital procedural volumeLow500019.050435.05Medium941435.826,54626.6High11,90345.268,25956.18CABG, coronary artery bypass grafting; Mass-DAC, Massachusetts Data Analysis Center; PCI, percutaneous coronary intervention; USRDS, US Renal Data System.a For CABG, hospital procedural volume is defined as low ≤1418, medium >1418 to ≤2175, high >2175.b Hospital procedural volume for PCI is defined as low ≤914 cases in total during the study period, medium >914 to ≤4625, high ≥4625.c Eighty-seven (0.33%) CABG patients were missing payor. Open table in a new tab CABG, coronary artery bypass grafting; Mass-DAC, Massachusetts Data Analysis Center; PCI, percutaneous coronary intervention; USRDS, US Renal Data System. After excluding patients with kidney transplants (N = 49 for CABG; N = 193 for PCI), 295 of 26,268 (1.1%) CABG patients were identified by USRDS as having dialysis-dependent ESRD (Table 2). Of these, 278 (94.2%) were correctly identified by the STS instrument, and 17 (5.8%) were not. Conversely, 147 of 25,973 (0.6%) non–dialysis patients were incorrectly flagged as dialysis patients at the time of their procedure. Sensitivity for identification of dialysis-dependent ESRD was 94.2%, specificity 99.4%, positive predictive value 65.4%, and negative predictive value 99.9%.Table 2Sensitivity and specificity for identification of chronic dialysis patientsIdentification of dialysis in the USRDS and Massachusetts state revascularization dataDialysis according to Mass-DAC coronary artery bypass–STS dataDialysis identified in USRDSNoYesTotalNo25,8261725,843Yes147278425Total25,97329526,268Sensitivity (%)Specificity (%)Positive predictive value (%)Negative predictive value (%)94.299.465.499.9Dialysis according to Mass-DAC percutaneous coronary intervention–NCDR dataDialysis identified in USRDSNoYesTotalNo98,1617498,235Yes5448761420Total98,70595099,655Sensitivity (%)Specificity (%)Positive predictive value (%)Negative predictive value (%)92.299.561.799.9Mass-DAC, Massachusetts Data Analysis Center; NCDR, National Cardiovascular Data Registry; STS, Society of Thoracic Surgeons; USRDS, US Renal Data System. Open table in a new tab Mass-DAC, Massachusetts Data Analysis Center; NCDR, National Cardiovascular Data Registry; STS, Society of Thoracic Surgeons; USRDS, US Renal Data System. There were 950 of 99,655 (0.95%) PCI patients on dialysis identified by the USRDS. Of these, 876 (92.2%) were correctly identified, and 74 (7.8%) were not identified by the NCDR instruments. Of 98,705 individuals not on dialysis, 544 (0.6%) were incorrectly flagged as receiving chronic dialysis at the time of their procedure. Sensitivity was 92.2%, specificity 99.5%, positive predictive value 61.7%, and negative predictive value 99.9%. A supplementary analysis (Supplementary Figure S1 and Supplementary Results) identified dialysis type, hospital procedural volume, and procedural urgency as characteristics common to both data sets that differed in the number of false-negative or false-positive patients compared with true-positive patients. Results for both PCI and CABG were similar following adjustment for hospital (Supplementary Table S1). Analyses stratified by STS version (CABG data) did not demonstrate significant variability in sensitivity according to instrument version (Figure 1), but specificity differed (sensitivity Ptrend = 0.84; specificity Ptrend = 0.01). However, differences were marginal, with an overall change in specificity of <0.4%. For PCI, sensitivity and specificity did not vary significantly by NCDR version (Figure 2; sensitivity Ptrend = 0.86; specificity Ptrend = 0.64). Trends across calendar years were qualitatively similar for the 2 data sets (not shown).Figure 2Sensitivity and specificity according to the version of the National Cardiovascular Data Repository (NCDR) instrument used. False-negative compared with true-positive identification (a) and true-negative compared with false-positive identification (b) of dialysis patients undergoing percutaneous coronary intervention (PCI) according to the version of the instrument. P values are provided for tests to assess differences between versions and trends across versions. The underlying data are provided at the top of the table. Mass-DAC, Massachussetts Data Analysis Center; USRDS, United States Renal Data System.View Large Image Figure ViewerDownload Hi-res image Download (PPT) For PCI, the area under the curve (AUC) was significantly lower (P = 0.02) for models incorporating the USRDS variable (0.899, 95% confidence interval [CI]: 0.891, 907) compared with the Massachusetts Data Analysis Center variable (0.900, 95% CI: 0.892, 0.908), but differences were marginal (Figure 3). Similarly, the continuous net reclassification index (–0.031, 95% CI: –0.058, –0.003) was consistent with weak effects on risk discrimination. For CABG, AUCs using the USRDS variable (0.765, 95% CI: 0.741, 0.790) and the Massachusetts Data Analysis Center variable (0.770, 95% CI: 0.745, 0.795) were not different (P = 0.06). The net reclassification index –0.045 (95% CI: –0.112, 0.022) was also consistent with only weak effects on reclassification. Lastly, differences in predicted risk were minimal, regardless of hospital procedural volume, for the vast majority of procedures (Supplementary Tables S2–S5 and Supplementary Figure S2). We assessed dialysis identification by the NCDR and STS data instruments for 2003–2012 Massachusetts patients receiving CABG or PCI, by linking our data to the USRDS. Specificity and negative predictive values for identification of chronic dialysis patients were high, and false-positive rates were low. However, the proportion of individuals receiving maintenance dialysis was small, and positive predictive values were low (62% for PCI; 65% for CABG). The impact on overall prediction of procedural risk was small, suggesting that use of these tools to compare facility outcomes is reasonable despite the misidentification of an important risk factor like chronic dialysis status. Prognostic risk scores derived from STS (the STS score) and NCDR data sets and instruments have been widely used to assess risk-adjusted outcomes, compare procedural results across providers, analyze outcomes of cardiac surgery, assess the impact of kidney disease and dialysis status on practice patterns, and assess postsurgical, postmyocardial infarction, and post-PCI outcomes.8Normand SL. Adult Percutaneous Coronary Artery Intervention FY14 Annual Report. In (vol 2018). Massachusetts Data Analysis Center, Boston, MA, 2016.Google Scholar, 9Normand SL. Adult Coronary Artery Bypass Graft Surgery FY14 Annual Report. In (vol 2018). Massachusetts Data Analysis Center, Boston, MA, 2016.Google Scholar,S1–S11 However, to our knowledge, the current investigation is the first to assess the precision of the dialysis variables, and our results suggest that their accuracy is suboptimal. Although sensitivity and specificity are high, the overall prevalence of maintenance dialysis patients was <1.5% in each cohort. Consequently, positive predictive values were low, with more than one third of patients identified by the STS instrument, and nearly 40% of those identified by the NCDR, not actually receiving maintenance dialysis. This raises questions about use of data based on the NCDR and STS instruments to assess cardiac procedures. Although the exclusion of patients with chronic kidney disease and ESRD from cardiovascular trialsS12 makes use of these data sets to investigate cardiac treatment strategies attractive, our results suggest that identification of chronic dialysis patients within NCDR- and STS-based data sets is not sufficiently accurate to provide reliable guidance for the care of dialysis patients. Misspecification of the dialysis variable could negatively influence adjustment for confounding and reduce the accuracy of public reporting of PCI and cardiac surgery outcomes. To avoid over-reliance on any single metric for assessing prognostic value, we examined changes in AUC and net reclassification index, and compared predicted and actual risk. We detected marginal effects on AUC, and predicted risk differed significantly in a minority of individual cases. Thus, in aggregate, our data suggest that the overall impact of dialysis status misspecification is small and unlikely to significantly compromise analyses of procedural risk and benefits or comparative hospital scorecards, although the net impact could be important in hospitals with a combination of low procedural volume and unusually high rates of misspecification. Determining the underlying reasons for and best response to the inaccuracies we identified is necessary. Our results suggest that including explicit variables for peritoneal dialysis, targeting training efforts at low volume centers, and considering enhanced validation of data gathered during emergent or urgent procedures are steps with potential utility. Although we lacked the data needed to investigate misidentification of individuals with dialysis-dependent acute kidney injury as patients with dialysis-dependent ESRD, we also believe that clarification of the instrument fields to better discriminate between acute and chronic kidney disease should be considered. Our analysis had several limitations. We analyzed data from a single state, and our results may not be fully generalizable. However, data collection using the NCDR and STS instruments is mandated in Massachusetts and is performed by trained staff; selected fields were audited to ensure fidelity. Nevertheless, better performance in the national data sets is theoretically possible. Additionally, state privacy regulations precluded use of social security numbers during matching to the USRDS, although we were able to utilize name, date of birth, and last known alive dates. Significant numbers of patients sharing these identifiers, within Massachusetts during the study period, is unlikely. In conclusion, we matched Massachusetts PCI and CABG patients from the USRDS to identification of chronic dialysis patients by the NCDR and STS. Neither accurately identified individuals with dialysis-dependent ESRD, suggesting that data collected using these instruments may not be useful for informing therapeutic choices in individuals requiring chronic dialysis and that efforts to improve these instruments are warranted. DMC received consulting fees from Amgen, Medtronic, and Lilly-Boehringer, and fees related to service on DSMBs (AstraZeneca, Allena Pharmaceuticals) and trial steering committees (Jannsen Pharmaceuticals-CREDECE Trial, Zoll Medical-WEDHEAD trial). All the other authors declared no competing interests. Funding for this work was provided by National Institutes of Health Grant R01HL118314-01A1. Data reported here have been supplied by the United States Renal Data System (USRDS). The interpretation and reporting of these data are the responsibility of the authors and in no way should be seen as official policy or interpretation of the US government. We thank Treacy Silbaugh for support on this project. Download .pdf (.29 MB) Help with pdf files Supplementary File (PDF)
No takes yet. Share an insight, caveat, or question.
Charytan et al. (2019) conducted an observational in Patients undergoing CABG or PCI (n=126,165). STS and NCDR data collection instruments vs. United States Renal Data System (USRDS) data was evaluated on Accuracy of dialysis-dependent ESRD identification (sensitivity, specificity, PPV, NPV).
Synapse has enriched 10 related papers. Consider them for comparative context: