Setting Standards for Effectiveness: a Comparison of Expert Panels and Decision Analysis

OBJECTIVE: To compare criteria for coronary revascularization developed by the expert panel process and by decision analysis. METHOD: We reviewed the medical records of 3080 chronic stable angina patients who either underwent coronary artery bypass graft surgery (CABG) or percutaneous transluminal coronary angioplasty (PTCA) and determined the agreement between appropriateness ratings made by two expert physician panels, one from the United States and the second from The Netherlands. We also evaluated the agreement between these panels' appropriateness ratings and a decision analytic model's effectiveness categories. RESULTS: There was poor agreement between U.S. and Dutch panel appropriateness ratings for PTCA (kappa = 0.03) and slight agreement for bypass surgery (kappa = 0.18). Dutch ratings had substantial agreement with the decision analytic models effectiveness categories for both PTCA and CABG (kappa = 0.83 and 0.79, respectively) whereas there was no systematic agreement between U.S. ratings and the decision analytic model for PTCA and poor agreement for CABG (kappa = 0.00 and 0.18, respectively). CONCLUSIONS: Although the level of agreement between expert panels and decision analysis on when a procedure is appropriate or effective may vary by procedure and the strength of the scientific evidence, we found that Dutch physicians agree much more strongly with decision analysis than U.S. physicians.