Big data, big problems: causation, confounders and cohorts

In this issue of Anaesthesia, Chang et al. 1 have presented findings that suggest general, as opposed to neuraxial, anaesthesia is associated with a higher incidence of newly-diagnosed epilepsy in the first postoperative year. The authors analysed the database of re-imbursement claims spanning seven years from Taiwan's National Health Insurance Program to identify a suitable population. The readers are referred to a recent editorial commenting on the merits of retrospective designs 2, but in this narrative, we want to take the opportunity to comment on a few statistical terms and techniques in the context of the study by Chang et al. Without the luxury of a prospective randomised cohort, one needs to ‘match’ patients in order to reduce imbalance in the empirical distribution of the pre-anaesthesia confounders between groups. If conducted adequately, the resulting data set after matching is reminiscent of that which might have resulted from a randomised experiment. Investigators of the current study have chosen to match age and sex using a one-to-one patient matching method, resulting in a group of 235,066 patients who underwent surgery under general anaesthesia and a group of 235,066 patients who received neuraxial anaesthesia. As can be seen from Table 1 1, because the patients were only matched on two covariates (sex and age), there are statistically significant differences in baseline characteristics between the groups in terms of income, and comorbidities such as hypertension, chronic obstructive pulmonary disease (COPD) and anaemia. These differences were noted by the investigators and are addressed using stratification; that is, sub-grouping patients according to age, sex, income category and type of surgery, and then calculating rate ratios for each of these sub-groups comparing neuraxial and general anaesthesia (Table 3) 1. In retrospective analyses, propensity matching is commonly used to avoid imbalance between patient groups 2-4. Ideally, one would match each patient in the general anaesthesia group with a neuraxial anaesthesia patient that was an exact match in terms of each of the observed covariates (e.g. sex, age, comorbidities); however, as the number of covariates increases, it becomes less likely that an exact match would be found. Because which group a patient belongs to is usually a dichotomous variable (in this case, general or neuraxial anaesthesia), the propensity score is often calculated using logistic regression with general or neuraxial anaesthesia as the dependent variable and the baseline characteristic covariates entered as independent variables (in this case, sex, age, the presence of comorbidities, type of surgery, etc.) 5. A predicted probability is calculated for each patient, and is defined as the conditional probability of being assigned to general vs. neuraxial anaesthesia. The predicted probability is known as the propensity score, which falls between 0 and 1. The number itself is not interpretable, a high score is not good or bad, the score is just a summary of the effects of the variables in predicting group membership. By considering the propensity score, confounding factors that differ between the groups can be accounted for, so that any remaining difference is likely due to the variable of interest, and the investigator can then match patients based on similar scores. Propensity score matching is simple to conduct, but investigators must consider the choice of variables used to generate the propensity score. The rationale for propensity matching is that, once propensity scores have been accounted for, there would be no selection bias. But investigators are reminded that the score is generated according to which variables are entered into the equation. If important variables are omitted, the groups may still be unbalanced. When the study is retrospective and uses a pre-existing database, variables are limited to the ones which have already been collected, and these may not be the ones which are contributing to the prediction of group membership. In the current study, the primary outcome of interest is the one-year risk of epilepsy after surgery, and it is expressed as the rate ratio. The rate ratio is computed as the ratio of the incidence rate in an ‘exposed group’ (the general anaesthesia group) divided by the incidence rate in an ‘unexposed group’ (the neuraxial anaesthesia group) (see Box 1). Table 2 1 shows that out of the 235,066 who received general anaesthesia, 953 developed epilepsy within one year, whereas out of the 235,066 who received neuraxial anaesthesia, only 741 developed epilepsy. The incident rates of epilepsy after general anaesthesia would then be 953/(235,066 patients multiplied by the number of years each patient was followed up for) = 0.004054 (or 0.405 per 100 patients), and for neuraxial anaesthesia, 741/(235,066 patients multiplied by the number of years each patient was followed up for) = 0.003152 (or 0.315 per 100 patients). It is to be noted here that in this study, all patients were followed up for one year; therefore, the denominator for the above rate ratio calculation is the same as the number of patients, that is, 235,066. The rate ratio is, therefore, 0.405/0.315 = 1.286. This is interpreted as the group which underwent general anaesthesia had 1.286 times the rate of epilepsy compared with the group which underwent neuraxial anaesthesia. Since a rate ratio of 1 would denote the rate of developing epilepsy was the same in both groups, if the 95%CI for the rate ratio crosses 1, there would be no statistically significant difference between the two groups. When the investigators stratified the patients into covariates such as income and surgical type, you will notice that, once these covariates are taken into consideration (in particular digestive and peripheral vascular surgeries), the confidence interval spans from 0.78–1.19 to 0.22–1.75, respectively, and therefore, for these two surgical types, there is no difference in rate of epilepsy between general and neuraxial anaesthesia (Table 3) 1. Some readers might confuse the abbreviation RR, which is short for rate ratio in this study, for risk ratio, for which the abbreviation RR is more often used. The calculation of the risk ratio is very similar to that of the rate ratio, with the exception that the total number of patients in each group is the denominator (in this example, 235,066), rather than the number of person years. Since each patient was followed up for one year, the rate ratio and risk ratio in this example would be exactly the same. You may prefer to express your results using the risk ratio if your emphasis in on how different the long-term probability of developing epilepsy is for the general and neuraxial anaesthesia groups. The investigators have reported rate ratios since their emphasis was on whether the exposure to general anaesthesia is causal for the development of epilepsy. The rate ratios and confidence intervals were calculated by modelling the data using Poisson regression 6. Since the primary outcome development of epilepsy (or not) is binary, it would seem more appropriate to analyse the data using a logistic regression model. The authors chose to use Poisson regression as this model gives a direct estimation of rate ratios, with confidence intervals, rather than the odds ratios given by logistic regression. The rate ratio is preferred by many investigators since the odds ratio can exaggerate the true risk, especially if the incident rate of event is high in a particular population. The disadvantage with rate ratios is that its estimation is more difficult in a multivariable setting, and that is one reason why the stratification method was used in the current study. Although statistically sound from a methodological perspective, we must ask whether the findings of Chang et al. are clinically significant. If we apply these numbers to a hypothetical Taiwanese population, there would be one more case of epilepsy for every 1111 patients who underwent general rather than neuraxial anaesthesia. This compares with the reported incidence of between 1:24,000 to 1:54,000 of permanent injury due to central neuraxial blocks 7. Does this mean that, whenever possible, neuraxial anaesthesia should be the default anaesthetic technique? Maybe not just yet, as the authors pointed out there are a number of limitations to the study, most importantly, the plethora of important data that were not captured. Furthermore, there are documented geographical and ethnic variations in the prevalence of epilepsy in different countries 8 and even within the Taiwanese population 9 and therefore whether such data can be applied to other populations remains to be seen. Are we seeing just another example of association without causation 10? Based on the incident risk as identified in this study (0.405 vs. 0.315, risk difference 0.045 per 100 patients), it would take a prospective randomised trial involving 69,515 patients in each arm (with a power of 80% and p < 0.05) to prove causation (see Box 2 for power and sample size calculation) and that is without the complexity of stratification of total intravenous anaesthesia vs. volatile in the general group and with or without concomitant sedation in the regional group. With the impracticalities associated with conducting a randomised controlled trial, we are likely to be restricted to suboptimal quality data from such registries in the foreseeable future. However, it might be interesting to see whether such a pattern may be derived from comparable databases from other countries. If you have any questions on methodology, please direct them to Ms Method Matters at msmethodmatters@gmail.com. SC is statistical advisor to Anaesthesia. No other competing interests declared.