Key result
In a discussed cohort of 22,570 cardiac surgery patients, 1-year mortality was 5.6% for those with an acute risk change >0 compared to 3.5% for those with <0, which the author attributes to score calibration and unmeasured heterogeneity.
Answers are grounded in this paper and cite their sources.
This editorial highlights the methodological pitfalls of calculating acute risk change by subtracting two differently calibrated risk scores (AusScore and APACHE 3) in cardiac surgery patients.
Nate Silver has written an excellent book entitled ‘The Signal and the Noise’ 1. He discusses the art and science of prediction, illustrated by economic crashes, gambling, the weather and earthquakes. His book is threaded with a warning: in the age of Big Data there are hundreds, thousands or millions of non-causative associations for each single causative association. Imagine one real hospital fire in a century of hourly alarms. Imagine 876,000 p < 0.05, only one due to cause and effect. Nate Silver warns that Big Data might yoke us to the production of meaningless data that will generate less knowledge, less understanding and less wisdom. Nate uses the ‘crash’ of 2008–2009 to illustrate how economists confidently misinterpreted their models because they believed that their complexities ensured validity. Nate concludes that we – the researcher, the clinician, the scientist – are the grist that can turn the Big Data millstone into a mill: choose carefully the associations you analyse and interpret them in context. The p value is no measure of sense. Tim Coulson et al. report the association of a variable with mortality within 1 year of cardiac surgery for 22,570 patients 2. The variable Coulson et al. studied was one estimate of survival (the AusScore, which is like the EuroScore) subtracted from another estimate of survival (APACHE 3) 3, 4. I will explain the italics later. The AusScore and APACHE 3 were developed in different populations using the same outcome, mortality, the former at 30 postoperative days and the latter in-hospital, with both incorporating risks associated with operations. The AusScore was developed with variables available before cardiac surgery and APACHE 3 was developed with variables available after admission to intensive care. Coulson et al. reasoned that the difference in scores, the ‘acute risk change (ARC)’, is the consequence of a patient's probability of dying being increased or decreased by whatever happens between the two calculations i.e. immediate peri-operative events. They found that the value of ARC was on average more positive in patients who died within one postoperative year i.e. the postoperative APACHE 3 estimate was relatively more for patients who died than who survived, or alternatively the AusScore was relatively less. They conclude ‘Acute risk change may represent peri-operative clinical events in combination with unmeasured patient risk and noise’. This conclusion seems sensible. Should we be content that the edge of Occam's razor, wielded by Coulson et al., is sufficiently subtle to sever Silver's thread? How much of ARC is signal and how much is noise? And what might ARC's signal mean? The ARC signal is quiet and the noise is loud, with differences between fitted APACHE and AusScore mortalities of 0.03%, 0.04%, 0.3%, 0.5%, 0.7% and 1.5% for the six subgroups presented in the authors' Table 1 2. The quietness of the signal would be more apparent if the results were tabulated for patients with APACHE values more than AusScore values (ARC > 0%) and APACHE values less than AusScore values (ARC < 0%), rather than columns for those alive or dead at one postoperative year. A threshold ARC of zero was exceeded by 358 patients per 1000, of whom 338 survived to 1 year and 20 (5.6%) died; conversely, of the 642 patients per 1000 with ARC < 0%, 620 survived to 1 year and 22 (3.5%) died. We could use a threshold value for AusScore of 2% for the same purpose: 323 patients per 1000 had an AusScore > 2%, of whom 295 survived to 1 year and 28 (8.6%) died; conversely, of the 677 patients per 1000 with an AusScore < 2%, 663 survived to 1 year and 14 (2.2%) died. In 1964, Arno Penzias and Robert Wilson were worried that pigeons, roosting in Bell Labs’ horn antenna, were responsible for an omnipresent 7.35 cm wavelength electromagnetic signal. What turned out to be the cosmic microwave background radiation persisted after the scientists removed the pigeons (and their droppings). What ‘pigeons’ might be carried by the ARC? Any candidate bird must mediate ARC's characteristics: negative values with increasing mortality risk (at least to the mean value of 4.7% in the ‘high’ risk group); yet, relatively positive values for patients who died. The first bird roosted because Coulson et al. compared two different scores instead of a single score calculated both before and after surgery. The scores reported as ‘AusScore’ and ‘APACHE 3’ were forced to (almost) agree by calibrating the original scores to the mortality observed by Coulson et al. in the first postoperative month, 451/22,570. These (italicised) scores were therefore different to the originals and they therefore weren't predictions. The cohort mortality estimates for the fitted ‘AusScore’ and ‘APACHE 3’ were 465 (2.06%) and 454 (2.01%), respectively, p = 0.74. The ARC for these mean values is −0.05%. Three of the six subgroup ARC values reported by Coulson et al. are more negative than this mean value and three are more positive. The shapes of the lines that describe how estimated mortality increases with increasing scores are different for different scores 5, 6. Recalibration, which ‘shifts, squashes and stretches’ lines, cannot match the shape of one score to another unless the individual components of each score is calibrated against the same outcome. The individual components weren't recalibrated, so the scores don't match, and if they did, the ARC would be zero and I wouldn't be writing this editorial. Acute risk change values that become more negative as estimated mortality increases might be a consequence of comparing two scores that have different shapes. This does not explain why patients who died within one postoperative year had relatively more positive values (you might expect them to have more negative values). The second bird to pull out of the antenna is that one can calculate estimated mortality in two ways with AusScore (and AusScore). The referenced AusScore gives three ‘case studies’, illustrating how to calculate mortality estimates with AusScore's 12 beta coefficients 3. The alternative, used by Coulson et al., is to use the integer score (range −3 to 39). The integer score has 43 unique values, whereas the beta coefficients generate 21,265 unique values. Every patient with an integer AusScore of 5 (the median value in survivors) is assigned a mortality estimate of 1.22%, but the beta coefficients attributed integers of 5 generate mortality estimates of 1.09% to 1.68%. Similarly, every patient with an integer score of 10 (the median value in patients who died), who is assigned a 3.11% risk, may have ‘real’ pre-operative risks anywhere from 2.51% to 4.36%. Patients assigned the same pre-operative risk, based on an integer score, would usually have different pre-operative risks based upon beta coefficients. Now, here comes the rub. Patients with higher AusScore estimated mortalities will generally have higher APACHE 3 estimated mortalities: patients who die will have higher APACHE 3. Because ARC is calculated by subtracting the (pre-operative) integer AusScore from the (postoperative) APACHE, ARC values will be more positive for patients who died than patients with the same integer AusScore who survived, who would have lower APACHE. The problem of different AusScores being treated as the same could be confounded by combining AusScore (integer) risks into risk categories. For instance, four AusScore integers (4–7) constitute the ‘moderate’ risk category, 1–2%, equivalent to AusScore (beta coefficient) risks 0.9–2.4%. The 174/4666 deaths in this risk category at one postoperative year would be associated with the more positive ARC values at each integer score, which I've estimated at −0.11, −0.02, 0.02 and 0.15, while the 4492 survivors would be associated with the more negative ARC values at these scores (−0.53, −0.62, −0.72 and −0.79). Figure 1 illustrates these concepts. The only factors common to AusScore and APACHE 3 are age and type of cardiac surgery, each of which is treated differently by the two scores. Time changes the relationship between risk factors and mortality, for shared factors and for the majority that are not shared. Recalibration cannot extinguish these differences. Up to this point I have treated estimated risks as if they were exact. The risks estimated by original and calibrated scores are all accompanied by uncertainty; 95% confidence intervals that are substantial on occasion for individual components and that, even when small, accumulate substantial uncertainties when combined into estimated mortalities. There appear to be a flock of reasons that would justify a default interpretation of ARC as a measure of how the AusScore was systematically underestimated in patients with higher risk, relative to APACHE 3. It is also possible that patients at lower mortality risk had AusScore systematically overestimated relative to APACHE 3. Given an apparently homogeneous cohort, for instance patients with the same pre-operative coefficient AusScore risk, how should we interpret some dying sooner than others? Were the patients the same? Are the dead just unlucky? Or might the dead have been different to the living? Might higher postoperative APACHE 3 scores in patients who died result from surgery exposing their relative fragility, which the pre-operative measure failed to detect? Or were they killed by worse care? Coulson et al.'s study has made me think that it is difficult to disentangle how much variations in care, luck and undetected risk contribute to the variation in postoperative mortality. One might assume that ‘the patient is to blame’ for 96% of the variation in mortality 7. The surgical exposure of hidden heterogeneity through survival selection leaves traces in postoperative survival curves. I used Australian actuarial general population data to estimate a pre-operative monthly mortality rate of 41/22,570 (0.184%) had the cohort been acutely well (patient characteristics in the paper's table 1) 2, 8. The 11-fold mortality increase in the first postoperative month, to 451/22,570, is a consequence of acute pre-operative medical illness and ‘postoperative disease’. I used the survival calculator I'm familiar with (but using Australian rather than UK population data) to generate survival curves 9. The survival curve generated for a population with a 0.184% monthly mortality fits the observed survival for 14,486 patients with ARC < 0 if mortality in the first postoperative month increases to 1% i.e. a temporary postoperative 5.4-fold increase in mortality. The postoperative survival trajectory for 8084 patients with ARC > 0 was consistent with a peri-operative 20-fold mortality increase (to 3.7%). The two subgroups either side of ARC = 0 had, on average, identical pre-operative mortalities of 0.184%, when well. I state this because the shape of survival curves is very sensitive to starting conditions: if you know the initial mortality rate you can accurately predict the observed survival (with a few exceptions); and conversely the observed survival is consistent with a very narrow range of starting mortalities i.e. find the modelled curve that fits the observed survival and you can discover the starting mortality rate. The sum of 145 deaths (14,486 × 1%) and 299 deaths (8084 × 3.7%), 444/22,570, is insignificantly different to the observed 451 deaths, p = 0.84. The leverage provided by the long arms of survival curves allows one to disentangle background mortalities from acute changes much more precisely than the short stub of one postoperative month. Coulson et al. further unravel cohort heterogeneity with survival curves for the ARC subgroups within AusScore categories < 1%, 1–2% and > 2% (the authors’ Fig. 2a–c) 2. The curves were consistent with the pre-operative and postoperative monthly mortality rates and temporary peri-operative risks listed in Table 1. There is a pattern to these values that I don't understand. The ARC categories (< 0 and > 0) in the lowest and highest AusScore categories, < 1% and > 2%, have similar ratios of monthly mortalities and temporary peri-operative risks, derived from their survival curves. In contrast, for the middle AusScore category (1–2%) pre-operative mortalities derived from survival curves for ARC < 0 and > 0 are less similar than for the other two AusScore categories, while acute peri-operative risks are more similar (Table 1 and Fig. 2). I think that one should not exclude worse care as a cause of greater mortality when postoperative mortality estimates exceed pre-operative mortality estimates: at least one can try and do something about it. However, I think that the observed survival curves signal that acute and chronic pre-operative differences in patients, undetected by AusScore, account for much of the association of ARC with death, combined with differences in patients that AusScore could detect but that were lost when they were pooled by the integer model. I think that our understanding of peri-operative mortality would be improved by developing a single model that can accommodate additional information as the patient progresses through their peri-operative care; a universal model common to all patients having any operation. The initial calculation would generate a personalised background survival curve, as if the patient had stable health or stable ill-health, but not acute disease. This curve would be specific to several variables including the year of surgery, thereby accounting for changes in population survival. The first pre-operative modification would adjust the survival curve by incorporating increased risk from acute physiological disease. The second pre-operative modification would incorporate the average risk associated with the intended operation. Intra-operative modifications would incorporate specific deviations from the average value of factors independently associated with survival, for instance excessive blood loss. Subsequent sequential postoperative modifications would incorporate deviations from the expected recovery trajectory. I hope that my idea reminds you of various existing scores, including AusScore, APACHE and POSSUM. The only difference is that a single score is used that provides long-term survival curves, sequentially modifying the initial fit to an appropriate background population. I anticipate that in this context we will judge whether care has been good or bad by what we do to patients, rather than what happens to patients. My interpretation of their work differs from Coulson et al.'s: neither of us knows which – if either – is correct. The scientific method would be for each of us to make one or more mutually-exclusive predictions (in the true sense of the word) and test their truth, or rather the falsehood of the null hypothesis. If you can think of such a testable hypothesis, let me know. Experimentation distils knowledge from information, the signal from the noise. JC is an editor for Anaesthesia. No external funding or competing interests declared.
No takes yet. Share an insight, caveat, or question.
J. B. Carlisle (2017) conducted an editorial in Cardiac surgery (n=22,570). Acute risk change (ARC) > 0 vs. ARC < 0 was evaluated on Mortality at 1 year. In a discussed cohort of 22,570 cardiac surgery patients, 1-year mortality was 5.6% for those with an acute risk change >0 compared to 3.5% for those with <0, which the author attributes to score calibration and unmeasured heterogeneity.
Synapse has enriched 10 related papers. Consider them for comparative context: