Unmodifiable Events, Heart Failure Research, and ‘Risk-Based Monitoring’ in Large Studies—The Unholy Triumvirate

A farmer had a donkey working the fields. One day the farmer forgot to feed the donkey, but the donkey worked the field well. The farmer decided the next day not to feed the donkey and the donkey worked well. After 5 days the donkey died. ‘Oh no, ’ thought the farmer, ‘now that I finally taught the donkey to work without eating, it goes and dies. ’ The development of new, effective interventions for the treatment of heart failure (HF) has been largely unsuccessful in the last decade. In contrast to major progress achieved in oncology, diabetes, and hyperlipidaemia therapies, with the exception of sacubitril/valsartan, no new medication or device has been approved for HF treatment in the last 10 years. 1, 2 This failure occurred not for lack of trying—many phase 2 and phase 3 studies have been conducted during this period. 3 However, we have many times witnessed the sequence of a phase 2 positive result not being replicated in phase 3, leading to abandonment of the intervention. In these cases we have concluded that the phase 2 positive result was a ‘chance finding’ or ‘false positive’, and the larger phase 3 has demonstrated that the intervention was not really effective. However, it is possible that, despite the phase 3 studies being larger, their results might represent ‘false negatives’. We explore the potential contribution of operational issues to the failure of phase 3 clinical studies in HF. Heart failure clinical studies struggle with lagging enrolment with most studies reporting enrolment rates of around 0. 5 patient/site/month. 3 The reasons for this low enrolment rate, in a disease that is extremely prevalent, are not well understood. Although it is not the scope of this manuscript to discuss the many reasons for this reduced enrolment rate, we note that enrolment rates declined starting in the 1990s, when in many US and western European institutions investigator payments were decoupled from enrolment, citing inappropriate incentives. As a result, enrolment in such institutions has dropped dramatically. In response, operational teams have shifted study conduct away from the US and western Europe into countries (particularly underdeveloped countries) or institutions where direct payment to study teams based on recruitment is possible and overall study conduct is significantly less expensive. 4, 5 Such places include countries that allow ‘split contracts’, i. e. contracts that enable some of the investigators' grant to be paid to the hospital but the remainder to be paid directly to the study team using a ‘split’ determined mostly by the site's principal investigator. The health care spending per capita tends to be lower and the investigators' grants, even though lower than in the US and western Europe, higher relative to the countries' per capita health spending and gross national product enabling brisk enrolment at a reduced overall cost. A second trend has emerged where more and more research is shifted to companies that specialize in recruiting patients into clinical studies—with names along the lines of ‘John Doe LLC. ’, ‘John Doe Research Associates, Inc. ’, ‘Mount Everest Research’ or ‘Great Research for America’ (all fictional names). These companies constitute neither the patient's primary care provider nor their HF treatment centre. They focus solely on the patient's study participation and both the company (and its owners and employees) and the patient are paid directly for study participation. Because the patients' payments for study participation are modest, and such specialized recruitment centres freely advertise their activities and patient payments on the internet (such online and/or social media advertising include statements like: ‘One way to earn extra money and help other people is to volunteer for a medical research study’, ‘Get paid to participate!’, ‘Compensation is available for up to 3900 for time and travel!’), enrolment in such centres may be biased towards poorer patients who may have less access to advanced care, especially in the US where health insurance is spotty. Taken together these trends have shifted clinical trials enrolment away from a more ‘real-life’ environment into an artificial one where patients are either treated differently (see below) or are enrolled in institutions that are not their primary care providers and whose relationship with the patient is primarily financially driven. These shifts in the places where clinical research is conducted may have a profound effect on which patients are enrolled in clinical studies. In ‘real-life’ clinical practice, new therapies are largely given to patients only after all established treatment pathways have been exhausted, and only after proven therapies such as revascularization, devices, beta-blockers and renin–angiotensin–aldosterone system (RAAS) blockers have been optimized. In the current study environment, patients are very commonly enrolled in studies where new medications are given on top of suboptimal care, and/or patients are enrolled into a clinical study and given a new, unproven intervention that had it been available by prescription probably would not have been given to them in similar circumstances. In the preceding decades, several effective therapies were developed for patients with HF and have become standard of care (SOC). These therapies include devices (automatic defibrillators and resynchronization therapy), revascularization (intravascular intervention or bypass surgery), and oral medications including RAAS blockers—angiotensin-converting enzyme inhibitors (ACEi), angiotensin receptor blockers (ARB), angiotensin receptor neprilysin inhibitors (ARNi), and mineralocorticoid receptor antagonists (MRAs) —and beta-blockers (BB). All of these interventions have been shown to improve morbidity and mortality in patients with HF, 6 but are only effective when actually prescribed and available to the patients. Regretfully, the use of these interventions is limited, especially in places where health care resources are limited. 7 Even simple medications such as RAAS blockers and BB—most of which are available as cheaper generic drugs—may not be widely available and, when prescribed and obtained, may be fake. 8 As a result, patients enrolled where the per capita health care spending is low may be treated for their HF mostly with diuretics, even when their HF is associated with low ejection fraction. Hence, these patients have very high morbidity and mortality rates—similar to those observed in HF cohorts decades ago. 4 In addition to lack of use of SOC therapy for HF, these patients are commonly not well treated for co-morbidities. Although we acknowledge that patients with HF are likely to have multiple co-morbidities, 9, 10 there is a distinction between having co-morbidities and having untreated co-morbidities—such as untreated hypertension or untreated diabetes mellitus, or unrevascularized active ischaemia. Very high event rates in these patients may not be modifiable by a novel experimental therapy administered in a clinical trial, especially those that specifically target HF processes such as remodelling, unless that new intervention has effects in addition to effects on HF that include effects on acute ischaemia, renal function, adherence, and arrhythmia. Furthermore, some of these unmodifiable events will occur very early after recruitment, as would be expected in untreated HF patients. One has to remember that short-term outcomes of HF patients 50 years ago, when diuretics were the only available therapy, were dire. Because the primary endpoints of most HF studies are defined as the time to the first event, a patient that has an early event, often before the study intervention might exert any effect, would count towards the endpoint. Any later events typically only count towards secondary endpoints and safety. Importantly, although we define eligibility criteria in study protocols to exclude patients with undertreated or unstable conditions, these proscriptions are sometimes ignored and these deviations not detected due to the reduced clinical monitoring implemented in recent years (see below). Although some would argue that a greater geographic distribution of clinical research sites is desirable, enabling better generalizability of findings and more equitable dispersion of new therapies, this approach may have certain drawbacks. First, in some places where it is documented that patients are substantially less well treated, 4 the risk of early events unmodifiable by the study intervention is increased. Second, experimenting with novel, untested therapies in places where patients are not treated with proven, routine, and largely inexpensive therapies seems ill advised. Such untested interventions may not only fail to help these patients but may actually harm them. For instance, one can easily imagine that some vasodilators may pose a risk to unrevascularized patients with severe ischaemia and active angina, or that interventions that may slightly increase the risk for some arrhythmias may harm patients with no automatic defibrillators. Clinical research is about assessing the risk–benefit ratio of a new intervention in a set environment. The risk–benefit ratio for patients who are not well treated may not be the same as in places where SOC is fully implemented. In recent years, some clinical trialists have discovered that patients had low adherence with the investigational product regimen. 11, 12 In some cases, these non-adherence patterns were geographically segregated. 13 The effect of such occurrences on clinical studies could be significant. Patients who are not given study drug definitely cannot benefit from the study drug and thus have a constant unmodified risk for the study primary endpoint. If the patient starts the study drug later, but an endpoint was reached before the patient has taken the study drug, then the endpoint was unmodifiable by the study drug and anything that happens to the patient after that moment does not count anymore towards the primary endpoint of the study. Clinical HF studies are usually designed to assess the effects of a new intervention on the time to first event of HF or cardiovascular readmission and/or death or cardiovascular death. Once an event has occurred, the patient has reached the primary endpoint and further evaluation of the patient may help determine some secondary endpoints and safety but the primary endpoint can no longer be affected. Most HF interventions take time to exert their effects and, hence, if many patients are enrolled in the study with a high early event rate, the study intervention may not be able to take effect before these events occur. For instance, if a patient with a history of ischaemic HF and a recent myocardial infarction (MI), with no revascularization and with 95% stenosis of the left anterior descending coronary artery and circumflex artery is enrolled in a study and within days develops a repeat MI and then dies, novel interventions directed at HF were unlikely to have saved the patient's life. Equally, a patient with untreated atrial fibrillation and a constant heart rate of 95 b. p. m. , who decompensates again within weeks, or a patient with end-stage renal failure and rapidly deteriorating renal function who is readmitted repeatedly early due to fluid overload and finally requires dialysis, would have events that are unlikely to be modified by new interventions targeting core HF processes such as remodelling. Additionally, a patient who won't take their medications and who has been admitted three times in the last 3 months is unlikely to comply with a new HF medication and is likely to be readmitted regardless. It is hard to expect a new therapy for acute HF to be effective in a patient admitted for pneumonia. For a new HF medication to be effective, the patient has to be allowed to take the medication for some time under reasonable conditions, while being reasonably treated overall. Let us imagine a simple 5000-patient, randomized, parallel-group, placebo-controlled HF study powered to a 20% relative risk reduction over a 2-year period with a cardiovascular death rate in placebo of 25%. The study is event-driven with a plan to stop enrolment when 900 cardiovascular deaths occur. Such a study would have about 85% power at the two-sided 0. 00125 significance level. Now let's assume that the study includes a subpopulation of patients who has a higher placebo event rate, say 45%, and that in this subpopulation the risk reduction with active therapy is only 5%. If the proportion from this subpopulation represented 10%, 20%, or 30% of the total patients enrolled, reducing the apparent treatment effect from 20% to 17. 2%, 14. 2%, and 11. 2%, respectively, the resulting power of the study is reduced from 85% to 64%, 38%, and 17% (Figure 1). Now let's assume that this subpopulation consists of high-risk patients, i. e. patients not treated with SOC or those who do not meet the letter or spirit of the protocol eligibility criteria, who have no benefit from the intervention. In such a study, the enrolment of 10%, 20%, or 30% of total patients from this subpopulation will reduce the treatment effect from 20% to 16. 3%, 12. 5%, and 8. 5%, and reduce the power to 57%, 25%, and 6%. Finally, one may envision cases where a subpopulation, for instance, patients with revascularized ischaemic cardiomyopathy with severe coronary artery disease not well beta blocked and given a vasodilator shortly after an acute MI, may actually derive harm from the study intervention. For a small negative effect (–3%), the observed overall relative treatment effects with 10%, 20% and 30% enrolment of such patients will be reduced to 15. 7%, 11. 2%, and 6. 5%. For a larger negative effect (–7%), relative treatment effects are reduced to 15. 1%, 9. 9%, and 4. 4%, respectively. Power declines concomitantly. Study power declines very quickly when 10–20% patients with high unmodifiable event rates are enrolled because our imaginary study (as is typical in HF outcome studies) is event-driven, meaning the trial is stopped when the necessary number of events have occurred. Hence, the more patients enrolled with fixed high event rates the sooner the study reaches its target number of events. Patients who have a lower placebo event rate and a more positive treatment effect contribute less to the study, and the events of the unresponsive subpopulation are overly weighted. We have assumed that this unresponsive patient subpopulation has contributed their events first, because in our experience patients of this type tend to be enrolled in centres that are the first to enrol and enrol quickly, and these events tend to occur soon after enrolment. The high event rates contributed by these unresponsive subpopulations may give the study management team, who are monitoring the blinded accumulating events during the study, a false sense that the study is on track. The combination of a high event rate in the placebo group and no or a reduced treatment effect in the active group creates a high blinded event rate. The study management team may even get complemented for being ‘on time’ and ‘on target’—until the data are unblinded and it becomes apparent that the high event rate was really associated with no treatment effect, at which time the study is finished, and the intervention is deemed ineffective and subsequently abandoned. Over the last 30 years, and until only a few years ago, industry-sponsored clinical research of investigational products was done in a certain way—data collection on each patient was extensive, and 100% of the data for 100% of the patients were verified against source documents on-site by clinical monitors. Although this system was expensive and had other drawbacks, it produced some positive results—for RAAS blockers, devices, BB and sacubitril/valsartan. In 2013, the US Food and Drug Administration published a documented entitled ‘Guidance for Industry Oversight of Clinical Investigations – A Risk-Based Approach to Monitoring’. 14 This document aimed at enabling the industry to reduce some of this burdensome monitoring (and cost) by focusing monitoring and clinical conduct on relevant and important data points, encouraging all stakeholders in clinical research to develop tools to identify areas of risk to human subjects, data quality, and trial integrity and focus their attention on these areas of risk. Furthermore, it suggested that in high-risk diseases (and HF is clearly such) one should temper this approach and continue to implement fuller monitoring. The European Medicines Agency followed suit with a reflection paper issued in 2013, 15 and the European Commission published in 2017 an expert group's opinion on the application of ‘Risk proportionate approaches in clinical trials’16 in light of the 2014 Clinical Trials Regulation (no. 536/2014). 17 However, some stakeholders in clinical research have interpreted this directive as suggesting that on-site monitoring can be immediately and substantially reduced without creating new, evidence-based ways to replace it, and have implemented these new approaches globally and equally in simple studies and complex HF studies. Without any evidence to support such a drastic shift, clinical study conduct has been changed dramatically such that only a small number of patients are now monitored on-site and, and in those patients, only a small part of the data is monitored. The dramatic reduction in on-site monitoring has been replaced with some combination of statistical-model-based flags and remote monitoring. Flags (‘risk identifiers’) typically identify sites that are at higher risk of audit by a regulatory authority and risks include such indicators as high enrolment, high proportions of missing data, lags in data completion, and a disproportionate adverse event rate. 18 Remote monitoring typically consists of reviews of data entered by investigators in electronic case report forms (and thus errors of omission cannot be easily detected), and focus almost entirely on reconciliation of data within the case report form, for example, the consistency between reported medical history and prior medications or between adverse events and study-defined endpoints. These processes are not geared specifically to HF studies and are implemented equally in non-cardiovascular, simple cardiovascular, and complex HF studies. The enrolment of inappropriate patients into a HF trial, or suboptimal treatment of enrolled patients, even when constituting a straight forward protocol violation, are not typically captured by these automated methods or simple remote monitoring and hence are commonly missed. This issue is especially important in HF studies where in many cases inclusion and exclusion criteria are subjective and hence cannot be easily ascertained. For instance, if patients are enrolled with high B-type natriuretic peptide caused by right HF secondary to chronic obstructive pulmonary disease, cirrhosis, non-adherence, acute congestion complicating or an acute MI, all those are unlikely to be captured by the current Hence, without evidence of or in the last several years, on-site monitoring has been reduced and replaced with new untested Such may have profound effects on the of of clinical studies. As the even if 10–20% of patients in the study are allowed to be enrolled under such and up having a high rate of unmodifiable the effect of a intervention may in the of fixed high event and no longer be these some people would that a less monitored environment better and if an intervention cannot be shown to be effective in such an environment, it will fail in However, as the artificial environment in which patients are enrolled into HF studies in the last decades is from patients are being enrolled in underdeveloped countries or in trial recruitment that are paid to enrol patients. the that of most of monitoring creates a study environment that better is at and at of many of or the of clinical studies conducted in HF. the these new of monitoring should have been before being However, to the of our not one study has been published that this approach to monitoring is to clinical and its is only by seems a and as should be as simple data and safety reconciliation and not a monitoring system for HF studies. It be of and report from and are by Research which research from and