Clinical Trials: A Surgeonʼs Perspective

Historically, decisions regarding the effectiveness of surgical interventions have been based on careful observations. Such accepted practices as appendectomy for acute appendicitis, colectomy for colorectal cancer, and fistulotomy for idiopathic simple fistula-in-ano are procedures that have never been evaluated with a randomized controlled trial, yet few would refute their role in the management of these conditions. What these interventions have in common is the magnitude of their treatment effect. For instance, prior to the introduction of appendectomy, appendicitis was almost uniformly fatal, whereas the mortality today is <0.1% (1). The 5-year survival rate without surgery in patients with rectal cancer is <10%, whereas following surgical extirpation, 5-year survival rates are 45-70% (2). Fistulotomy is usually effective in eliminating fistula-in-ano, whereas virtually no established fistulain-ano will disappear spontaneously (3). Thus, even taking into account that there may be differences in patient selection and ancillary care, it is likely that the improved outcomes are due to the treatments, and incorrect conclusions regarding their effectiveness are unlikely. In modern surgical practice, however, it is unusual that new surgical procedures or technologies will make such a dramatic difference in outcome. More often, new innovations result in a small improvement in survival, morbidity, or quality of life. For instance, there is controversy about whether laparoscopic appendectomy has advantages over open appendectomy. It is assumed that with both procedures the mortality is the same, but there may be differences in pain or time to discharge from hospital. Recently, local resection has been advocated instead of abdominoperineal resection for small cancers of the low rectum with the assumption that survival is equivalent but quality of life may be better. However, in both instances the difference in outcome is small and differences in patient selection, ancillary care, or other factors may, in fact, be responsible for the observed differences, rather than the treatment. Thus, it is in these situations where randomized controlled trials are necessary. Extraneous factors must be controlled to be certain that the observed difference is indeed due to the treatment. The randomized controlled trial is accepted as the best trial design for determining treatment effectiveness. There are several essential components of the randomized controlled trial. First, subjects are randomly allocated to two groups: usually a treatment group (in which the new treatment is being tested) and a control group (in which the standard therapy or placebo is administered). Thus, the control group is concurrent and subjects are randomly allocated to the two groups. Second, the interventions and follow-up are standardized and performed prospectively. Thus, it is hoped that both groups are similar in all respects except for the single factor being studied. Not only does this guard against differences in factors known to be important, it also ensures that there are not differences due to other factors that have not yet been identified. Thus, where differences in treatment effect are small, the randomized controlled trial may minimize the chance of reaching an incorrect conclusion about the treatment effectiveness. While there appears to be widespread acceptance by physicians of the need for randomized controlled trials, there has been much less enthusiasm for them by surgeons. In a review of three surgical journals, Solomon and McLeod (4) found that only 7% of published clinical articles were randomized controlled trials despite the fact that almost half of the articles addressed issues of treatment effectiveness. Furthermore, the proportion did not differ between 1980 and 1990. Similarly, Barnes (5) noted that only 5% of abstracts accepted at the annual joint meetings of the Society for Vascular Surgery and the International Society for Cardiovascular Surgery dealt with randomized controlled trials. Haines (6) reported that only 5% of articles in the Journal of Neurosurgery between 1973 and 1977 were controlled clinical trials. What clinical trials are being performed by surgeons? Solomon and colleagues (7) performed a survey of surgical randomized controlled trials published in 1990. Of the trials published by surgeons or from surgical departments, the majority (75%) compared two medical therapies, whereas trials comparing two surgical therapies comprised only 18% and trials comparing a medical to a surgical therapy comprised only 5% of all those published. Thus, trials comparing antibiotic prophylactic regimens and adjuvant chemotherapy regimens were not uncommon, whereas trials comparing two different operative procedures were infrequent. Furthermore, the published trials tended to be small: Almost two-thirds were single-center trials, and in half there was no significant difference detected probably because the sample size was small and the trial lacked adequate power. The quality of the trials tended to be poor, especially if they contained one or two surgical arms. In the field of inflammatory bowel disease, there is a notable paucity of surgical trials. There are a few trials comparing surgical technique in performing the pelvic pouch procedure and comparing anastomotic techniques in Crohn's disease (8,–11). Recently, there have been several trials assessing the effectiveness of prophylactic therapy in the prevention of recurrent Crohn's disease (12,–17). Although some of these latter trials have been performed by surgeons, all compare two medical therapies. Thus, it appears that there are relatively few randomized controlled trials being performed, and of those being performed, most compare medical therapies rather than surgical treatments. Why is there a paucity of surgical trials? Certainly it is not for want of controversy. The role of segmental colectomy and the value of strictureplasty in Crohn's disease and the need for a mucosectomy in performing a pelvic pouch procedure are but a few controversies. Various reasons have been cited for the lack of trials. Some have argued that methodological issues preclude the performance of surgical randomized controlled trials (18,–21). Others have suggested that surgeons' concerns about randomizing patients and patient preferences are major obstacles (22,–24). What is apparent is that there are issues related to the performance of surgical clinical trials that must be considered. The basic elements of a randomized controlled trial are similar whether the effectiveness of a medical therapy, psychosocial intervention, or a surgical procedure is being assessed. However, certain issues are unique to, or at least must be considered more frequently in, randomized controlled trials assessing surgical procedures. In some instances, these issues may preclude the performance of randomized controlled trials. In medical trials, the dosage can be standardized and compliance can be measured. Furthermore, the prescribing physician is not a variable. In surgical trials, however, standardization of a procedure may be difficult because surgeons vary in their experience with and their ability to perform a surgical technique, there may be individual preferences in performing the procedure, and there may be technical modifications as the procedure evolves (18,25). Furthermore, differences in peri- and postoperative care may impact on the outcome. There are several strategies that may be instituted to ensure that a certain surgical standard has been achieved and that the effect of the surgeon as a variable is minimized. First, one can ensure that there is a consensus among participating surgeons on how to perform the critical aspects of the procedure. Prior to the trial, teaching sessions may be held to improve standardization. During the trial, measures can be instituted (e.g., pathological examination of resection margins in Crohn's disease) to ensure that the operation is performed adequately. Where results are suboptimal, performance may be improved by providing feedback to surgeons. In addition, patients may be stratified according to surgeon to ensure that there is not an imbalance. Finally, participation may be limited to a small number of surgeons or only to those who have documentation of performing a satisfactory number of procedures. The obvious advantage of this strategy is that standardization is more likely. As well, it increases the likelihood of observing a treatment effect, since only experts are included. The disadvantage to this strategy is that the results may be less generalizable. Standardization is an important issue in designing a surgical trial, and a certain minimum is mandatory. The amount of standardization, however, may vary depending on the question being asked in a similar way that the compliance of patients may vary in a medical trial. Thus, a trial where surgical technique is highly controlled is analogous to the medical trial where only compliant patients are randomized. This may be viewed as an efficacy trial. Where the surgical maneuver is less controlled, it may be viewed as an effectiveness trial. In other words, the question is: Is this surgical procedure effective when performed by many different surgeons without special expertise with this procedure? However, no matter what strategy is employed, it is essential that the investigator describe how the procedure was performed and by whom so it can be replicated by the reader. Chalmers (26) has argued that the first patient in whom a surgical procedure is performed should be randomized. Most surgeons would argue, however, that there is a learning curve in any procedure and that modifications to the technique are made frequently at its inception. By including these early patients, one would almost certainly bias the results against the new procedure. It might also inhibit modifications being made as experience with the procedure increases. Alternately, if modifications are not made, the procedure may be obsolete at the completion of the trial, and thus the results may have little impact. Delaying institution of a trial until the procedure has been perfected also has its hazards. A trial may be difficult to initiate when the procedure has “matured” and is widely accepted not only by the surgical community but also by patients. The paucity of randomized controlled trials testing surgical therapies may be due in large part to this dilemma related to the timing of the trial. However, the EC-IC Bypass Study is an example of a trial that was performed even though the procedure had wide acceptance (27). Bias may occur due to lack of blinding of patients and/or investigators in any clinical trial whether it is assessing medical or surgical therapies. This is of great concern in surgical trials where it is known that surgery often has a significant placebo effect (21,28). In trials comparing medical and surgical therapies, blinding may be impossible since sham operations are generally unethical. Even in trials comparing two surgical operations, there may be difficulty blinding patients and investigators if the incisions differ or the operations differ in other ways. The lack of blinding may be minimized by choosing a “hard” outcome measure such as recurrence of disease or death. However, more often the important outcome is a change in symptoms or quality of life, and outcome may be biased by knowing which treatment group the subject is in. In these situations, the potential for bias may be minimized if a hard outcome measure is also measured and it correlates with the patient's subjective assessment. For example, recurrence of Crohn's disease may be based on subjective complaints in conjunction with endoscopic evidence of disease. Assessments may also be performed by an independent assessor who is unaware of the treatment group that the patient is in. Finally, if the criteria used to define an outcome are explicitly specified a priori, it may minimize or eliminate bias (e.g., criteria to diagnose a flare-up of perianal disease). Investigators may also choose, in this situation, to have a blinded panel review the results of tests to ensure they meet these criteria. Patient recruitment is often difficult for researchers undertaking a randomized controlled trial. Since not all patients with a certain condition may require surgery, it may be even more of a problem for surgical than medical trials. Furthermore, the surgical procedure performed may vary depending on the site and manifestations of the disease. For instance, strictureplasty is indicated only in patients who have multiple small bowel skip lesions. These constitute only a small proportion of patients requiring surgery for Crohn's disease and even a smaller proportion of patients who have the disease. As a result, while Crohn's disease is not an uncommon condition, the number of patients who would be suitable for a strictureplasty is small and a trial comparing strictureplasty to resection is probably not feasible because of the relatively few patients one might expect to accrue. One obvious solution to this problem is to conduct multicenter trials. In recent years, several surgical groups interested in inflammatory bowel disease have been formed, and one would hope that more surgical trials will be performed. However, even with multiple centers, there are some questions that probably cannot be answered with a randomized controlled trial because of inadequate number of patients. In these situations, clinical decisions may have to be based on results from less rigorous studies (29). Patient issues may severely affect recruitment into surgical trials (30,–34). In medical trials, patients may be agreeable to participate, knowing that at the conclusion of the trial, they may be able to receive the treatment that is found to be more effective. Surgical procedures, however, are usually permanent, and therefore there is no opportunity for the patient to receive the more effective procedure at the conclusion of the trial. For example, there is uncertainty whether recurrence of Crohn's disease is similar after colectomy and ileorectal anastomosis and total proctocolectomy in patients with Crohn's colitis. Unfortunately, should a trial find that recurrence rates are equal after both procedures, those randomized to receive an ileostomy would not be candidates for reconstructive surgery. Knowing that there is this uncertainty about the superiority of total proctocolectomy, it is likely that the patient would refuse participation in such a trial and would probably opt for having a colectomy and ileorectal anastomosis to avoid a permanent stoma. It has also been suggested that patients requiring surgery are less inclined to participate in a trial and leave an important decision such as the choice of the surgical procedure to chance. Other issues such as patients feeling like a “guinea pig” are common to both medical and surgical trials, but again may be of greater significance in surgical trials. Unlike the release of new medicines, surgery is not controlled by a regulatory board such as the Food and Drug Administration or Health Protection Branch (24). Whereas all medications must undergo extensive investigation prior to their release, new surgical procedures can be performed often with little or no constraint even from the hospital or local ethics committee. Indeed, there may be a number of disincentives to surgeons wishing to undertake a trial to assess a new procedure. First, if a trial is instituted, approval must be obtained from the local ethics committee, a much more detailed consent form would probably be required, the surgeon would have to discuss randomization with the patient, and follow-up for the patient would probably be more intensive. On the other hand, if the surgeon were simply to perform the operation, while a consent form would be required, it would almost certainly be less inclusive. Economic disincentives may make participation in trials unattractive. While a physician would be paid equally whether s/he entered a patient into a trial, the surgeon would not be paid equally if s/he were to randomize a patient to a medical treatment. A surgeon might also fear loss of her/his referral base if s/he did not embrace a new procedure even though there was concern about its effectiveness. Such was the case with the introduction of laparoscopic cholecystectomy when general surgeons felt their practices were threatened if they did not quickly adopt this new technology. Finally, funding of surgical trials is often a more difficult issue. Pharmaceutical companies, because of pressure from regulatory agencies to perform trials prior to release of medication, are a major source of funding for medical trials. Generally, industry funding is less accessible for surgical trials. Finally, it must be admitted that surgeons seem to place greater reliance on the results of case series. Whether this is because of the historical role of these studies, skepticism of the need for randomized controlled trials, or the perceived difficulty in performing surgical clinical trials, it is difficult to know with certainty. Some of the first surgical trials were performed in gastrointestinal surgery in the early 1960s by Goligher and colleagues (35) in evaluating peptic ulcer disease. More recently there have been some notable surgical trials that have assessed the value of various surgical procedures (lumpectomy for breast cancer, extracranial-intracranial bypass, and carotid endarterectomy) and had a major impact on the practice of surgery (27,36,37). However, as has been discussed, there are many other situations where randomized controlled trials have not been performed. Methodological issues may be cited as the reasons, but generally these do not preclude the performance of clinical trials. It seems more likely that a reluctance on the part of surgeons to perform trials and a greater acceptance of the results from case series is the more likely reason. In addition, recruitment of patients may be difficult for some of the reasons already stated. In the future, it is likely that surgeons will have to become more involved in performing clinical trials and evaluating procedures more rigorously. It is unlikely that regulatory bodies will impose this. Instead, those who control health care resources will demand more evidence of the effectiveness of new surgical innovations before allowing their adoption. Surgical technology tends to be expensive and to consume large amounts of resources. Only those that are cost effective are likely to be adopted in the future. Already in the United Kingdom there is a push to prevent adoption of new technologies without adequate assessment with clinical trials. It will be interesting to see whether the same pressures are seen in the U.S.A. and Canada. Surgical clinical trials may be seen to be a cost-effective investment. While there is agreement about the relative paucity of published surgical clinical trials, the reasons for this are less clear. There are methodological issues that make it difficult to perform surgical clinical trials. However, it is unusual that these issues preclude their performance, and it is hoped that more surgical interventions will be rigorously tested in well conducted trials in the future.