Comparison of Cardiopulmonary-Based Risk Models with a Clinical Heart Failure Risk Model

This article refers to ‘Multiparametric prognostic scores in chronic heart failure with reduced ejection fraction: a long-term comparison’ by P. Agostoni et al., published in this issue on pages xxx. In this issue of the Journal, Agostoni and colleagues1 compared the newer cardiopulmonary exercise testing (CPET) risk model [Metabolic Exercise test data combined with Cardiac and Kidney Indexes (MECKI)],2 an older CPET model [Heart Failure Survival Score (HFSS)]3 and a clinical risk model [Seattle Heart Failure Model (SHFM)]4 in predicting the 2- and 4-year risk of cardiovascular death, left ventricular assist device (LVAD) implantation, and urgent cardiac transplant in 6112 patients. They found the newer CPET model was superior for this endpoint as measured by the area under the receiver operating curve (AUC) (all three models ≥0.72) and the calibration was appropriate for all models. This is a large cohort of patients undergoing CPET and the authors are congratulated on deriving and very importantly validating the MECKI risk model. Identifying ambulatory patients for listing for cardiac transplant is a difficult decision and often relies on CPET. CPET [specifically peak oxygen consumption (pVO2)] remains a 2016 International Society for Heart and Lung Transplantation (ISHLT) class 1 recommendation for listing for cardiac transplant with a pVO2 of ≤14 mL/kg/min if intolerant of a beta-blocker and ≤12 mL/kg/min if on a beta-blocker,5 which corresponds to an annual mortality of ∼15–20% in a large cohort of 2105 patients from the Cleveland Clinic Foundation in the 2005 era (Figure 1),6 similar to results we published.7 In the HF-ACTION cohort, the annual mortality was much lower for men and women with a pVO2 12 mL/kg/min, corresponding to an annual mortality of ∼10% in men and ∼5% in women (Figure 1).8 This lower mortality has recently been confirmed in 630 patients with heart failure and reduced ejection fraction (HFrEF) with a 1-year ∼9% risk of death/LVAD/urgent transplant for a pVO2 of 12 mL/kg/min.9 This illustrates the difficulty and complexity of using pVO2 as a single variable in making decisions for transplant listing along with the interaction of gender and beta-blockers on the absolute risk of death for any pVO2. Ventilatory efficiency [minute ventilation/carbon dioxide production (VE/VCO2) slope] has been shown to be predictive of mortality in heart failure patients with and without beta-blockers.10 It is as good as or superior to pVO2 in some2, 6, 10 but not all analyses.2, 8, 11 The proposed cut-point for high risk is a VE/VCO2 slope ≥ 34–35.10 Despite the large volume of evidence supporting VE/VCO2 slope, this was not included in the 2016 ISHLT guidelines for transplant listing, except for patients with a sub-maximal CPET. Heart failure prognosis scores like the HFSS3 and SHFM4 (and presumably MECKI) are an ISHLT class IIb recommendation, even though these scores have a higher discrimination than pVO25 and are better calibrated than pVO2 (Figure 1). In HF-ACTION, the SHFM had a predicted 1-year risk of death/LVAD/urgent transplant of 6.2% and the observed event rate was 5.9%.11 Similar results were obtained in a large community implantable cardioverter-defibrillator (ICD) cohort of 10 319 patients (1-year predicted mortality 5%/observed mortality 6% with AUC = 0.76)12 and in the current study. The 2016 ISHLT guidelines suggested using a SHFM estimated annual mortality of ≥20% for transplant listing, even though the pVO2 values for transplant listing (≤12–14 mL/kg/min) correspond to ∼5–10% annual mortality in the current era (Figure 1). This has led many to assume that multivariate risk models are not helpful to list patients for transplant (i.e. the team believes the patient meets transplant listing and the SHFM is 10% annual mortality), as the guideline has required a much higher annual mortality for the SHFM (≥20%) than the associated values for pVO2 (∼5–10%) (Figure 1). The MECKI score combines percent predicted pVO2 (based on height, age, and gender), VE/VCO2 slope, haemoglobin, serum sodium, and renal function [by the Modification of Diet in Renal Disease (MDRD) equation based on age, gender and creatinine] in a logistic regression model to provide a 2-year estimate of cardiovascular death and urgent transplant. The model is not provided in the original publication,2 but the authors did provide the estimates required for calculation in a subsequent publication.13 The MECKI score for events at 2 years = 10.3464 + (−0.0262*per cent predicted pVO2) + (0.0472*VE/VCO2 slope) + (−0.1086*haemoglobin) + (−0.0615*sodium) + (−0.0699*left ventricular ejection fraction) + (−0.0136*MDRD) and the 2-year death/urgent transplant can be calculated by 1/[1 + Exp (−MECKI score)]. The authors compare the MECKI model developed to predict cardiovascular death and LVAD implantation or urgent transplant (84% of the total events were cardiovascular death) with the HFSS (pVO2 and clinical data) and the SHFM (no CPET data). The cohort includes ∼44% of the same patients that were used to derive the MECKI model and 13% of the cohort were lost to follow-up. They found the MECKI score (AUC = 0.781) provided superior discrimination to the HFSS (0.723) and the SHFM (0.739) for the combined endpoint of cardiovascular death and urgent transplant. As pVO2 and VE/VCO2 are used by clinicians and many insurers for transplant listing, it is not too surprising that a risk model that includes these variables is superior in cardiovascular death and urgent transplant to a clinical risk model. Freitas et al.14 compared the MECKI, SHFM, HFSS, and MAGGIC risk models. They found all four models had a similar and not statistically significant different AUC for all-cause mortality at 2 years (AUC 0.78, 0.76, 0.70, and 0.76, respectively). In HF-ACTION, the addition of pVO2 (mL/kg/min) and VE/VCO2 to the SHFM increased the AUC by 0.02.11 Calibration of models is extremely important to enable appropriate clinical decisions as shown for pVO2 in Figure 1. Calibration is obtained by measuring the agreement between the predicted and observed event rate in the overall cohort. A potential difficulty with calibration is illustrated using quartiles of N-terminal pro B-type natriuretic peptide (NT-proBNP) in the I-Preserve [heart failure with preserved ejection fraction (HFpEF)] and Val-HeFT trials (HFrEF).15 In both of the trials, a natural log increase in NT-proBNP was associated with an ∼70% increase in mortality. To illustrate this, we built a risk model in I-Preserve using NT-proBNP and applied the model to patients in the Val-HeFT trial. NT-proBNP provides similar discrimination in both trials (mortality increases markedly with each quartile), but the calibration in Val-HeFT is inadequate with the observed mortality ∼60% higher than predicted (2-year predicted event rate in Val-HeFT 12.9% and observed event rate 18.5%). A recent publication demonstrated that mortality is >50% lower for any pVO2 for both heart failure with mid-range ejection fraction and HFpEF.9 Further assessment of calibration (predicted and observed event rates) among subgroups of patients (quartiles, gender, age categories, ejection fraction categories, medication use, device use, etc.) should be evaluated. Beta-blockers have been shown to alter the calibration of the HFSS16 and other models that include pVO2 with improved survival for the same pVO2 as also illustrated in Figure 1. The risk associated with a 1 mL/kg/min decrease in pVO2 is 50% lower with beta-blockers.6 The MECKI investigators reported a 43% adjusted lower risk of cardiovascular death/urgent transplant with beta-blocker use,17 although this important variable is not included in the MECKI model. Women have a lower risk of death for the same pVO2 in many cohorts. This is illustrated in HF-ACTION where a 10% 1-year mortality occurred at a pVO2 of 10.9 mL/kg/min in men and 5.3 mL/kg/min in women. In the MECKI cohort, women had a 42% lower risk of death adjusted for pVO2 and VE/VCO2.18 Female gender is not a variable in MECKI but is included in the MDRD and estimates of predicted pVO2. When the MECKI authors validated the model in additional 992 patients in a different publication, they observed a 60% lower risk of cardiovascular death or urgent transplant at 2 years (∼3.4%) than predicted by MECKI (∼8.5%).13 In an external validation of the model, the MECKI overestimated the cardiovascular death/transplant endpoint by 24% (2 year predicted 12.1% vs. observed 9.6%), the MAGGIC model overestimated all-cause mortality by 145% (predicted 2-year death rate 17% with observed rate 7%), whereas the HFSS and the SHFM showed appropriate calibration for all-cause mortality and urgent transplant.14 In our opinion, inclusion of heart failure medications and heart failure devices that alter mortality without altering pVO2 or VE/VCO2 is very important for models to accurately calibrate in new cohorts. In this report, fully 44% of the patients were used to derive the MECKI model. In the new cohort of patients (∼56% of the patients in the study), the observed event rate at 2 years was very low (∼4%) and was only ∼14% in the highest risk decile, suggesting very few of the patients undergoing a CPET would require LVAD or transplant within the next 2 years. The authors are encouraged to demonstrate that calibration of the MECKI score is appropriate for age categories, male vs. female, beta-blocker vs. no beta-blocker, ICD vs. no ICD, etc. MECKI needs to be validated in other large CPET cohorts, preferably with a larger number of higher risk patients, to demonstrate adequate calibration, and not just discrimination. The MECKI is an excellent model based on CPET parameters for identification of patients for LVAD and transplant and an improvement on the HFSS. MECKI includes important CPET and laboratory variables. There are iPad and web versions available. However, until we are certain the MECKI is appropriately calibrated for both men and women and patients both on and off beta-blockers, we need to exercise caution with using the MECKI model for clinical decisions, as with other clinical heart failure risk models. Conflict of interest: W.C.L. developed the Seattle Heart Failure Model and the University of Washington holds the copyright for the model. T.F.D. declares no conflict of interest.