Therapist effects: Clever ways to make them (and everything else) disappear

Click to increase image sizeClick to decrease image size Notes 1. In this context in which cognitive–behavioral therapy was compared with interpersonal psychotherapy, we use the term treatment effect to denote the size of the effect that indexes the difference in outcomes between these two treatments. 2. It should be noted that using the last data point for the intent-to-treat sample is the typical manner in which such samples are analyzed and emphasizes the importance to the level of functioning of the patient when the patient terminated from treatment. As Crits-Christoph and Gallop (2005) correctly suggest, time is not considered, and these patients might have had a better score had they completed treatment. However, it should be realized that the longitudinal model assumes that the trajectory of improvement for patients terminating prematurely would have continued had they stayed in treatment, an assumption that is tenuous because many patients terminate when they have improved sufficiently (Brown & Jones, 2005 Brown, G. S. and Jones, E. R. 2005. Implementation of a feedback system in a managed care environment: What are patients teaching us?. Journal of Clinical Psychology, 61: 187–198. [Crossref], [PubMed], [Web of Science ®] , [Google Scholar]). See Figure I for the patient who improved quickly and terminated; the longitudinal analysis for this patient would have imputed a value extrapolated from the rate of change based on the first two measurements (a negative BDI score with a linear model!). 3. All of the authors in this section have remarked that the sample size in terms of therapists and patients is not ideal to test therapist effects, as the power to detect effects is relatively low. Crits-Christoph and Gallop (2005) suggest that naturalistic samples would be larger and more meaningful, a strategy used by Wampold and Brown (2006 Wampold , B. E. , & Brown , G. S. (2006) . Estimating therapist variability: A naturalistic study of outcomes in managed care . Journal of Consulting and Clinical Psychology , 16 , 184 – 187 . [Google Scholar]), who found statistically significant therapist effects. We recognized, as was noted in Kim et al. (2005), because of low power, that several therapist effects did not reach statistical significance, although many did. Nevertheless, the point estimates for therapist effects are sizable and always greater than the effect resulting from differences between treatments. There is absolutely no evidence, in these analyses of the NIMH data (by either group) or in any other data set, that the effect produced by comparing treatments is as large as the effect for therapists within these treatments. 4. Given the small number of therapists in this study (viz., 17), it is difficult to determine criteria that would objectively identify outliers. To eliminate the three therapists identified by Elkin et al. as outliers would be to eliminate approximately nearly 20% of therapists who were specially selected, trained, supervised, and monitored with regard to their delivery of cognitive–behavioral therapy and interpersonal psychotherapy.