Apparent declining efficacy in randomized trials

Recent HIV prevention trials have given hope that a suite of interventions that effectively reduce individuals' risk of HIV infection will soon be widely available 1–6. In two studies, the RV144 and CAPRISA 004 trials, the relative risk of infection increased toward the null value of one over time. The RV144 and CAPRISA investigators interpreted these trends as evidence that the interventions' effects declined over the study period and suggested that their respective findings may be explained by waning vaccine efficacy and decreasing adherence 2,7,8. Here, we discuss these trends in the trials' results and note that, in addition to the possible mechanisms cited by the investigators, their apparent waning efficacy may be explained in part by selection bias due to heterogeneity in infection risk, an explanation that has not been considered previously. This bias arises when study participants vary in their susceptibility to infection, for example, because of differences in immune systems or exposure to infection. This can lead to increasing differences in the composition of the study population in each trial arm over time as those at highest risk become infected, and can occur despite comparability between arms at baseline. This issue is termed 'frailty' in statistics and demography, in which a large body of literature addresses the matter 9–14. We also discuss several methods that can improve understanding of the effects of infectious disease interventions and risk factors by assessing the impact of frailty on results. Variation in frailty among study participants likely creates trends in the incidence of infection over the course of a study. To understand this phenomenon, consider a theoretical placebo-controlled randomized trial, in which the risk of infection varies among participants. The highest risk individuals in such a trial are expected to become infected earlier, leaving a pool of lower risk individuals at later time points. If the intervention being tested is effective, the decline in the incidence of infection over time will be larger in the placebo arm because these individuals experience no direct protection from the intervention, and so those at high-risk will be quickly depleted, thereby lowering the infection rate over follow-up. However, high-risk individuals in the active arm may remain uninfected due to the protection conferred by the intervention, so the active arm's infection rate will be less affected. Consequently, the time-specific rate ratio for treatment vs. placebo will increase over time from a value of less than one initially to a value that may exceed one later. This phenomenon has also been termed 'survivor bias', 'survivor cohort effect', 'crossing of hazards' and 'depletion of susceptibles', and is observed in both chronic and infectious disease research 15–19. Many randomized trials report a weighted average of the time-specific rate ratio over the entire follow-up, but, because of the phenomenon described above, the average rate ratio will become increasingly attenuated as follow-up time increases. This attenuation will occur even if all risk factors were balanced between study arms at baseline, as expected in a large randomized trial, and if the biological effect of the intervention is constant over time, which can lead investigators to reject an efficacious intervention. Because the results of any rate analysis can be affected by frailty, it can also cause investigators to overlook a risk factor for infection that is in fact harmful. To illustrate these issues we assessed whether the results of the RV144 and CAPRISA trials are consistent with an attenuation of the average rate ratio due to frailty 1,2. Although we focused on the widely discussed rate ratio estimates, the RV144 team also presented Kaplan–Meier estimates 7. Tables 1 and 21,2 show average incidence rates of infection and rate ratios calculated from baseline up to several time points in each trial 1,2. Both studies show decreasing incidence rates in the placebo arms and generally increasing rate ratios. The incidence rate of infection in the intervention arm of the CAPRISA microbicide trial decreased slightly 2. The incidence rate of infection in the vaccine arm of the RV144 trial increased from 0–12 months to 0–24 months; it then decreased slightly but remained below that of the placebo arm.Table 1: RV144 HIV vaccine trial modified intent-to-treat data.Table 2: CAPRISA HIV microbicide trial intent-to-treat data.To test for variation in the time-specific rate ratios, we fit a Cox model with a product ('interaction') term between the intervention arm and time (on the logarithmic scale) on study 20. A positive coefficient for this product term would indicate that the time-specific rate ratio increases over time. In the context of prevention trials, in which the average rate ratio is less than 1, this implies that the average rate ratio becomes attenuated over time. Figure 2C of the RV144 publication provides information on the number of infections and loss to follow-up in each arm for various time points 1. Using these data we reconstructed an interval-censored dataset of the trial for analysis with a Cox model. The coefficient of the product term was 0.42, indicating that the time-specific rate ratio increased over time, but the 95% confidence interval was wide (−0.09, 0.93). However, as such tests have limited power, failure to reject the null hypothesis cannot be taken as strong evidence that there was no attenuation over time 21. Because any possible trend in estimates may not be linear, we conducted several other tests of the proportional hazards assumption (e.g., using quadratic forms of time), and each produced similar results. There was insufficient information to perform a similar test on the CAPRISA 004 data. The attenuation of the average rate ratio suggested by our analysis of the RV144 trial may have been due to frailty, waning efficacy, differential loss to follow-up, random variability, or any combination of the above. Furthermore, we note that frailty will have a large impact on results only when disease incidence is high in at least one subgroup. The RV144 investigators report that immune responses among the vaccinated waned during the first 6 months after vaccination; therefore, vaccine efficacy may have similarly waned 7. As waning efficacy and frailty are not mutually exclusive mechanisms of attenuation, both may have contributed to the increasing trend in the rate ratio. The CAPRISA investigators report that some of the increase in the rate ratio may be explained by decreasing adherence to microbicide use 2. Although waning vaccine efficacy and trends in adherence may help to explain the increase in the rate ratio observed over the course of both trials, neither mechanism can explain the decreasing incidence in the placebo arms. In contrast, frailty parsimoniously explains both observations. Because rate ratio estimates are less affected by frailty during early follow-up, this may partly explain why studies that stop early because a considerable benefit is found can give larger estimates than those that continue. This is an alternative and nonmutually exclusive explanation to random variation 22 that has been insufficiently addressed in the literature on biases involved in stopping trials early 22–25. Similarly, frailty has not been considered as a candidate explanation for null results in HIV prevention trials 26–30, or for variation in estimated effects across studies in meta-analyses 31,32. Three approaches for dealing with frailty have been suggested. First, data on known risk factors for infection, including direct or proxy data on exposure to infection, could be collected and used in sensitivity analyses. This approach typically adjusts for baseline risk factors only, and uses standard methods, such as Cox models 9,10,32,33. However, it may not be feasible to measure all risk factors for infection, and some risk factors may not even be known. When the goal is to estimate a biological measure of the direct effect of an intervention or risk factor, frailty effects due to heterogeneity in exposure to infection can be minimized by censoring individuals when they are no longer at risk of infection. Data on exposure to sexually transmitted infections can be obtained from serodiscordant couples studies, from partner report, or when participants report having no partners during a study period, and so could not have been exposed to HIV via sexual routes 4,34–38. Second, some have suggested a type of crossover design to reduce the magnitude of attenuation in trials that examine reversible interventions through limiting the depletion of those most susceptible to infection by alternately giving both study arms access to the protective intervention 30. Third, alternative analytical methods have been proposed, such as frailty models, which can account for heterogeneity by modeling the distribution of risk in the population, and which have been used to analyze several vaccine trials 39–41. Bayesian hierarchical models have been used to estimate the efficacy of a previous HIV vaccine candidate and permit the incorporation of data on contact patterns and the prevalence of infection 42. Frailty models have been widely used, particularly when the assumption of proportional hazards is considered unlikely to hold due to frailty effects 14. It should also be noted that survival curves, which use risks instead of rates, are not subject to the same frailty effects discussed above and can be estimated for observational studies as well as trials 15,43. In summary, accounting for frailty may improve our understanding of study results and facilitate decision-making on the use of interventions and avoidance of risk factors. Acknowledgements J.O.H. drafted the manuscript and conducted the analysis. All authors contributed to the interpretation of the data and critically revised the article for important intellectual content. Conflicts of interest and sources of funding J.O.H. and M.A.H. report no conflicts of interest. M.L. reports having received consulting fees from Novartis, Pfizer/Wyeth Vaccines, Roche and AIR Worldwide. R.P.W. reports accepting an invitation to present expert testimony for a law firm. J.O.H., M.L. and R.P.W. were supported by Award Number U54GM088558 from the National Institute of General Medical Sciences. M.A.H. was supported by NIH grant R01 AI073127. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institute of General Medical Sciences or the National Institutes of Health.