Is intent‐to‐treat analysis always (ever) enough?

In a commentary written several years ago [1], Professor D. Rubin and I pointed out that one is usually advised to analyse clinical trials – the preferred modern strategy for empirical evaluation of medical therapies [2, 3]– for significance levels and estimates using only, or at least primarily, the intention-to-treat principle (see, e.g. [4]). We noted, however, that the intention-to-treat estimator (see below) estimates so-called ‘use-effectiveness’, the causal effect on outcome of prescribing the drug, rather than the medically more important ‘method-effectiveness’, the causal effect on outcome of actually taking the drug. We argued that studies should be designed so that they can yield valid estimates of method-effectiveness, although so long as protocols are not followed exactly, such estimates may depend heavily on additional assumptions (scientific models – see section on longitudinal data with dropout, below); ones that go beyond the data at hand. In this paper, I present a conceptual framework for thinking about causal estimands and estimators for both use and method effectiveness, with emphasis on the latter, focusing on the problems posed by deviations from protocol, notably noncompliance and dropout. The paper is a review and exposition, rather than a presentation of original material. Section 2 presents a conceptual framework, setting context, defining causal estimands (population quantities describing causal effects of treatments) for use effectiveness and method effectiveness, how these change in the face of noncompliance, and the problem created by dropout; section 3 discusses estimators of the causal estimands defined in section 2; section 4 considers the effect of dropout on the estimators; and section 5 discusses implications for study design. Although different in some details, the ideas, point of view, and notation presented here are essentially those presented in the statistical literature by D. Rubin and colleagues, based on the Rubin Causal Model. Rather than provide extensive references throughout this work, I limit further citations to important additional works which discuss specific points or present examples, and offer here a minimal set of references which provide background and an entree to the relevant literature [5, 6], and a more complete and technical discussion of noncompliance [1, 7-9], dropout [10, 11], and scientific modelling of drug dose-concentration-response [12, 13]. The context is a clinical trial in which n individuals are nominally assigned either to a control treatment, coded 0, or a test treatment, coded 1, and a univariate outcome is observed. The control and test treatments are assumed to differ. The random variable indicating nominal assignment for individual i is Zi, and the random variable denoting the individual's outcome is Yi. A key design feature of randomized clinical trials is ‘random assignment’ to nominal treatment. Since the validity of every estimator discussed in section 3 depends on this assumption, it deserves to be stated formally: it is assumed henceforth that the probability of being assigned to test (control) treatment is independent of pretrial prognosis and all other baseline patient characteristics. Two types of deviation from protocol will be of concern to this paper, noncompliance and dropout, and I introduce next the notation required to deal with them. Non-compliance recognizes that nominal assignment, the treatment assigned by the experimenter, may differ from actual assignment, the treatment taken by the patient. The random variable indicating actual assignment is Di; it can be written Di(Zi) to explicitly recognize that it is a function of assignment. Di can take the value 0, 1 or ‘other’, where the first two values correspond to values of Z– control (0) and test (1) – and the last indicates no particular treatment, but simply that actual treatment is unknown; some alternative is pursued. Thus, while recognizing only one ‘version’ of control and test treatments, I acknowledge the possibility of multiple versions of the alternative. In general, Y will be a function of D (and hence Z). Let Yi(d) denote the potential outcome (see below) for individual i actually receiving treatment Di=d. To maintain parallelism with dropout (below), I introduce the (redundant) random variable ), where I(·) is the indicator function, taking the value 1 if its logical argument is true, and 0 if it is not, to designate whether the intended outcome is observed (Ui=1) or not (Ui=0). I assume that the compliance tendency of each individual in the circumstances of the trial is a (unknown) baseline variable, Vi, which determines the values of Ui(z) according to Table 1. Dropout, as a problem for analysis and interpretation of a clinical trial herein refers to a data set that is incomplete due to the loss of the outcome datum from one or more subjects. The random variable indicating responder status is Ri. It takes the value 1 if the outcome of individual i is observed, and 0 if it is not. Just as with Di, Ri can be a function of assignment: Ri(z) is the potentially observable variable indicating whether or not individual i will drop out on assignment to treatment z. Similarly to Vi, the responder tendency of each individual in the circumstances of the trial is a (unknown) baseline variable, Wi, which determines the values of Ri(z), as in Table 2. For each individual in the trial, there are 11 potentially observable quantities, Zi, Vi, Wi, and Di(z), Ui(z), Ri(z), and Yi(z), for z=0,1, although there are only 7 independent choices, as Di(z) fully determines both Ui(z) and Vi, and Ri(z) fully determines Wi. I refer to the N×11 matrix of potentially observable data as the potential data matrix, or just the potential data. In any actual trial, we observe, for example, only one of Yi(Di(0)) or Yi(Di(1)) (these might or might not be different see Table 1), depending on whether zi is 0 or 1. The Rubin Causal Model claims that even if the control treatment was assigned and we observed Yi(Di(0)), nonetheless the test treatment could have been assigned, and in that case we would have observed Yi(Di(1)). Counterfactuals, that is unobserved potential outcomes, such as outcomes on unassigned treatments, are central in the Rubin causal model, as the model defines causal estimands (population quantities describing causal effects) using them. Although the idea of potential outcomes seems natural and intuitive to most persons, caution must be taken when they are used as a basis for causal reasoning. The only kinds of potential outcomes that may meaningfully feature in the definition of the causal relationships that a clinical trial might establish are those that would result from decisions made after study inception and before study termination11 A strict view would limit potential outcomes to only those outcomes consequent on actions assigned randomly by the experimenter, but I will not so limit the discussion herein. . Clearly, the ‘decision’ to assign test or control treatment to an individual could be altered after enrolment, so that the potential outcome on the control treatment for a type a complier assigned to the test treatment is certainly a meaningful concept. A bit less certainly so are the potential outcomes on both treatments for a type n complier: if through added inducements and reminders incorporated into a modified study design, one could imagine altering his ‘decision’ not to comply with any assigned treatment, then they may be meaningful, but if not, not. However, there is no possibility of altering a possible pre-existing correlation, perhaps a genetic-cultural link, between a propensity to behave as a type n complier and more rapid drug metabolism, leading to reduced sensitivity to the test treatment if it were taken. Thus, the type n complier's potential outcome on the test treatment if he were a type a complier is not a meaningful potential outcome, and hence causality in the sense of the Rubin causal model cannot be associated with complier type. Of the 11 potentially observable variables for any one individual, we always observe three, zi=Zi, di =Di(zi), and ri =Ri(zi). If ri = 1, then we also observe yi= Yi(di). The N×3 or 4 matrix of observed data is called the actual data matrix or the actual data. Note in particular that, contrary to usual practice, di is assumed to be observed: modern methods of monitoring compliance make this assumption realistic [14]. The characteristics of such methods, in fact, determine the definition of ‘actual assignment’ I have used in the face of noncompliance (and in this way, this work differs from that of the other authors cited previously): modern compliance monitoring methods record medicine container openings. They can therefore indicate that an individual is not taking the prescribed medication as instructed, but if they do so indicate, they do not indicate what other treatment he is taking. Thus, as noted above, the actual assignment category –‘other’ denotes some version of alternative treatment, but does not specify what it is. This view is particularly applicable, for example, when Di(z)=z is defined as >80% of prescribed doses taken correctly, as is commonly done. In that case an individual who takes less than 80% (but more than 0%) of the test treatment when Zi=1 and less than 80% (but more than 0%) of the control treatment when Zi=0 is of compliance type n, and for him by definition both Di(1)= ‘other’ and Di(0)= ‘other’; yet the two treatments are clearly not the same, nor, in general, would they be expected to lead to the same noncompliance and dropout can be as particular of data that were to be observed but were not in observed. those data are for those i as outcomes by an indicator as If noncompliance, but not dropout is i such that are if dropout, but not noncompliance is i such that are and if both are i such that ri A strict view would limit potential outcomes to only those outcomes consequent on actions assigned randomly by the experimenter, but I will not so limit the discussion herein. are data may be by the between the indicator and the of the potential data. on the data present of or are if the is independent of all other data. are if the depends only on other observed data and not at all on unobserved or potential the data This a technical data is a type of data. data are but also the assumption the of the no with the data The from the that with probability estimators and the model for the not in the probability model for the estimator to be valid and the can be We will not this is to estimator the assumption only data are and this is the that presents Note that it may be the case that depends on unobserved but not on such data after on the observed data. In that case the is not In such a of for the method of analysis of the data to be it must on the observed data. Thus, although data can be such are more to modelling assumptions than are based on complete data or data that are only noncompliance and dropout can and this is they present a problem for clinical trial A the problem with a randomized trial of a drug for a with outcome being a of after a on treatment. If drop out to on the test treatment of and if is associated with then the potential outcomes on the drug in the assigned to take it will be from the actual outcomes in that due to the dropout a natural estimator of drug the in between and those outcome data those on the assigned will be to the value it would have if dropout not been The are probability of being is to value with treatment that after for the observed data This discussion on causal that expected values of characteristics. However, for and it rather than I assume henceforth that the trial is so that I can on valid about this assumption, causal estimands defined on all of the potential data matrix are valid estimators of the and for this in an of no is made between the I some that will Let A and be the and The of the are in which case the the is in which case the assigned the denote the of the study for which is and denote the outcome with actual assignment those individuals in the study for is For I the logical A that all study in the to be the by the where it will no as an argument in the above, in an of I denote A and denote so that for example, . is the expected value of the between outcome with nominal assignment to the test treatment and with nominal assignment to It use effectiveness in the of the one from which the trial are method effectiveness is less an of clinical trials than use effectiveness, and the of this paper is to its and discuss how it can be I take the here to discuss at it is an important causal effectiveness – the expected outcome due to treatment with a drug a alternative – is certainly a more important for of individual than is use effectiveness – the expected outcome due to prescribing the drug the – for a of effectiveness is a rather than a of and one can more its from trial to – the for on clinical trials before the use of a drug – than that and is more on specific to a particular that may not a method effectiveness both and for example, if a trial the data do not a that use effectiveness is than the data may be with method effectiveness, indicating that the trial may have of noncompliance or dropout rather than of of method effectiveness for a specific that is with in values for that a to for drug The of method effectiveness is to the for drug is a a to use for A the expected and of a drug of compliance in a particular It is to see how this can be used to a for a patient. In an argument can be made that a for a patient is one that is to yield if taken as A the basis for such a emphasis on studies so that will be the to design that valid estimates of method I to this point to method effectiveness first that if all individuals were the method effectiveness would be however, the of actual treatments taken in to the two nominal can be different for all 4 types of 4 method effectiveness estimands may be one for each compliance Note that does not to the 4 types of does not actual treatment, so that this only concern with the method effectiveness estimands by Of least are and the is it does not outcomes on either actual treatment for the is of only if the control treatment is of and if the alternative treatment by individuals when assigned to the test treatment in differs from the control treatment. The two method effectiveness estimands and and are of the assumption that the alternative treatment taken by type when assigned to control is the test treatment to be true, for example, when to the test treatment is to those assigned to as is in drug – see both estimands expected outcomes the test treatment to some alternative. is of than as it the test treatment to the control treatment, which was for the trial for a the most relevant method effectiveness is a of and called the Causal It is the expected in outcome associated with a in nominal assignment in that the of the trial, will take the test treatment when assigned to as above, take the test treatment when they are assigned to the control then the that is of all those individuals who will take the test treatment trial if and only if they are assigned to I and as the method effectiveness estimands of are defined on potential estimators are to actual data. For I the assumption for the potential data made in the section to to of outcomes observed in each assignment the actual data. discussion henceforth can be to estimators that are for For I first valid estimators of and in the of and then the same estimators in its Just as for randomized assignment that were the of its e.g. and ), would be valid estimators of the causal and hence would be by estimators into The however, is that is not The of this considers how and might be . Note that the assumption it does for both and will in the type of clinical trial discussed by that the 1. with test treatment is all or 2. who does not take the test treatment takes the control The test treatment is not to are to be in to each all of the (1) is for a or a The control treatment is a defined of individuals do for the defining study and The test treatment is to and cannot be provide an The effect of is to make all alternative treatments the same and to the control treatment, and hence to make complier types and n a and even on potential only two types of types a and the always take and the always take control assigned to Since the so defined always take the same treatment of assignment, it is to imagine how outcome could depend on that assignment. If it does not, then the assumption the the assumption is as the This from the in which the estimator was first It is so the assumption an effect of compliance type on outcome as it determines the treatment actually . Note also that when the assumption there is no to observe so long as a valid of is perhaps from some other of test treatment in the assigned to test types a and If these same complier types could also be in the assigned to the control treatment, to just such would Of the key data – compliance unassigned treatments – are so that there is no to a and from n and in the into a and n and but a may be that is observable in all if not an indicator of compliance type at least a is at least with that is commonly used in this is the that it does not do the further provide an of such an assumption and an associated for the control to with the the the clinical trial data cannot provide that the assumption is If is used additional to to to this defines the to be discussed in the next in this is for method effectiveness if one can assume that the outcome on control is the same for and type so the which that control could be to take the control treatment. Of they but then and any additional the estimator can be a to use di when are not is to to if the two are it can be argued that it is that compliance and would be the same by perhaps If not, the estimator can be but more a between the two might be used to out use of the possibility is to all to the test treatment for a pretrial and to use the observed compliance in that as the it would be more to use the pretrial compliance as an for the rather than as a in the analysis Note that the data are if the outcome when assigned to control differs between and if is for even This is simply to that use effectiveness and method effectiveness are not even the of a The assumption may be to However, it may be less to of a that with of can outcome treatment only through a causal some other to the that are on such the will be Thus, a is to the A setting can be used to the same with a are central to the methods discussed in the next section for with dropout. it is possible to make in the of dropout on or For example, if dropout and noncompliance are the same either comply and or comply and and if or a can be then estimates Rubin that the assumption, and the additional assumption that dropout for is not to assignment, a valid estimator for can be are not always this section discusses a more than either of the It as on a to the data into just as a is used to data into data in the of the outcome so that the data between the and the are and one can as using the observed data by the This is here for a based analysis the probability model for the data in a more way, and the problems with for example, by Rubin . one cannot on a baseline to the (and hence in the data set as a In the data in longitudinal studies can in this and the of this discussion therefore on such data. A longitudinal study is one in which of are to be observed on each and the to be between assignment is a function of those In the and the one the outcome variable is simply the last value of the For example, if a is out and at after the outcome might be the at a then that there is a for each individual such that that all are to and that but The value of Ri then depends on the of to if the individual not drop if he The longitudinal design that the outcome is the last in an individual longitudinal Let the actually observed, and denote those that are to valid in the face of is to if a model for the is by and how such a model can is if we as at the of this that the will be with by in (but see the values with values and then with the analysis using as if it were the actual data Clearly, if each an of then the analysis using of it may be less than that will be The will depend on the that is the actually observed, and may as other can in principle be if depends on the e.g. on a and not on its observed If it depends on the observed value e.g. on the stated not on an model for then the model will be if the however, if are the of the can do is less than as the only data is the between the and the not the value of the is a It the at for individual i as the last observed value at this does not make the that a so that the last observed before is a of the last If the were a for a the would that a patient his at the last observed value the that it usually in with the natural of the For most there is no to that and would to the a patient out of a study were it out would be a The here certainly any causal estimator from the if the test treatment the of and if a point dropout, then more dropout will in the of is will these unobserved values to be the values last observed. this may make an analysis using more to than if it estimates would lead to of a the the of the can always be and in any the is not valid for method that scientific are the only ones that can They might be as as using the that is to values from a of the observed data on for each who or they might be as as a model, in an statistical model The key point is that, the of the model for the data cannot be on the observed for must therefore on the that them. This review with the that method effectiveness – outcome due to taking a test treatment a control – is an important causal on which to decisions whether to use the treatment, and if at what The to estimator of treatment effect estimates use effectiveness (but only if any data are an of value for decisions decisions and whether to the use of the not method if both with the same However, trials may be by deviations from protocol, notably some to comply with the prescribed treatment, and the same or other out before the study can be observed. of these deviations that the intention-to-treat estimator no estimates method effectiveness (and in the case of dropout, it may not even use on study other estimators may be that important method effectiveness It should therefore be a to design studies so that the such estimators to be valid are In general, other than the of randomized and assignment, and of complete the additional design the that one of the method effectiveness estimators discussed herein will be (1) treatment is the control treatment is of and the test treatment is only to those for it is is in all for the of the The study is and the is a function of the if those compliance with test are at baseline and In for data by deviations from protocol, scientific rather than of may be required to estimates of causal estimates will be to scientific assumptions that cannot be on the study data and assumptions should be and discussed in the study protocol to of