The American College of Surgeons National Surgical Quality Improvement Program Flap Failure Data Are Inaccurate after 2010

Sir: When using the American College of Surgeons National Surgical Quality Improvement Program data set to identify trends in free flap failure rates, we observed that there was a 5-fold decrease (from 5.76 percent to 1.05 percent) in the rate of free flap failure after 2010 (Fig. 1). This trend was striking to us, and we tried to understand the reasons for this monumental improvement in our ability to successfully perform free flaps. Others have also noted similar findings associated with the National Surgical Quality Improvement Program flap failure variable. Massenburg et al. highlighted that there was not a single flap failure reported in any of the 2895 patients undergoing autologous breast reconstruction procedures in 2013.1Fig. 1.: Rate of free flap failure from 2005 to 2010 and from 2011 to 2013.After multiple inquiries were made to the American College of Surgeons National Surgical Quality Improvement Program central office, we found that their variable for flap failure had been inconsistently entered into the database. In 2011, sites transitioned from a “Classic” to an “Essentials” program. In the Essentials program, several outcome variables were no longer required to be collected by participating sites. Cases with missing data for the flap failure outcome variable were incorrectly recorded as “No Complication” instead of “Missing.” Thus, the American College of Surgeons National Surgical Quality Improvement Program published a Data Update with their 2014 Participant User File (PUF) stating that “Graft failure … should not be considered accurate for any PUF after 2010” (Fig. 2).2Fig. 2.: Data update published in the User Guide for the 2014 American College of Surgeons National Surgical Quality Improvement Program Participant Use Data File.Large databases are a powerful research tool. Modern computing not only allows us to collect surgical data on multiple procedures, from multiple surgeons, and from a wide variety of different patients, but it also allows us to analyze this vast quantity of data. Although large databases have great potential, they also need to be used with caution. The adage “garbage in, garbage out” is particularly important when using these large databases. To study and improve the quality of surgical care, the American College of Surgeons (arguably the largest and most prestigious surgical society in the world) created the National Surgical Quality Improvement Program database. Since its creation in 2005, the National Surgical Quality Improvement Program has become the gold standard of clinically relevant surgical outcomes research. In the plastic surgery literature alone, there have been over 100 articles published using the National Surgical Quality Improvement Program. Sites that collect data for the National Surgical Quality Improvement Program employ a full-time trained and certified nurse surgical clinical reviewer. These data are then centrally aggregated, audited, and finally distributed. Despite the National Surgical Quality Improvement Program’s rigorous data collection methodology, the errors in the collection of flap failure data were not discovered for 4 years. We encourage researchers to continue to use large databases such as the National Surgical Quality Improvement Program; however, at the same time, we caution them to scrutinize the data. Although results from the analysis of large databases are powerful, they can also be misleading if the data are incorrect. Furthermore, it is incumbent on professional societies to take ownership of these databases and to help maintain the integrity of their data collection. DISCLOSURE The authors have no financial interest to declare in relation to the content of this article. ACKNOWLEDGMENT The American College of Surgeons National Surgical Quality Improvement Program encourages users to evaluate data when performing analyses. Registries are complex and constantly evolving enterprises and, because of that, participant observations can be invaluable. The National Surgical Quality Improvement Program has been, and will continue to be, transparent with respect to data integrity and analytic issues. Alvin C. Kwok, M.D., M.P.H.Christopher J. Pannucci, M.D., M.S.Jayant P. Agarwal, M.D.Division of Plastic SurgeryUniversity of UtahSchool of MedicineSalt Lake City, Utah