Variations in Definition and Method of Retrieval of Complications Influence Outcomes Statistics after Pancreatoduodenectomy: Comparison of NSQIP with Non-NSQIP Methods

BACKGROUND: NSQIP and the Accordion Severity Grading System have recently been used to develop quantitative methods for measuring the burden of postoperative complications. However, other audit methods such as chart reviews and prospective institutional databases are commonly used to gather postoperative complications. The purpose of this study was to evaluate discordance between different audit methods in pancreatoduodenectomy--a common major surgical procedure. The chief aim was to determine how these different methods could affect quantitative evaluations of postoperative complications. STUDY DESIGN: Three common audit methods were compared with NSQIP in 84 patients who underwent pancreatoduodenectomy. The methods were use of a prospective database, a chart review based on discharge summaries only, and a detailed retrospective chart review. The methods were evaluated for discordance with NSQIP and among themselves. Severity grading was performed using the Modified Accordion System. RESULTS: Fifty-three complications were listed by NSQIP and 31 complications were identified that were not listed by NSQIP. There was poor agreement for NSQIP-type complications between NSQIP and the other audit methods for mild and moderate complications (kappa 0.381 to 0.744), but excellent agreement for severe complications (kappa 0.953 to 1.00). Discordance was usually due to variations in definition of the complications in non-NSQIP methods. There was good agreement among non-NSQIP methods for non-NSQIP complications for moderate and severe complications, but not for mild complications. CONCLUSIONS: There are important differences in perceived surgical outcomes based on the method of complication retrieval. The non-NSQIP methods used in this study could not be substituted for NSQIP in a quantitative analysis unless that analysis was limited to severe complications.