Quality of Reports of Randomized Clinical Trials in Plastic Surgery

The randomized clinical trial is defined as a study design in which patients are allocated at random either to an intervention group undergoing a particular intervention, whether for diagnostic, preventive, or therapeutic purposes, or to a control group.1,2 Health care systems are expected to satisfy a growing demand for services with limited resources. To ensure that resource allocation is efficient, increasing interest is being placed in evidence-based research.3 Among research methods, the randomized clinical trial is considered the best source of available scientific evidence and the only scientifically reliable method for assessment of the efficacy (and risks) of an intervention.1,2 Regardless of whether or not the result of a randomized clinical trial reaches statistical significance, the design, conduct, and published report should be of high quality. High-quality trials and their reports should lead to better and more realistic estimates of treatment effects, more accurate estimates of treatment efficacy, and greater acceptance of these results within the health care community.4,5 Despite their scientific merit, randomized clinical trials are uncommon in most medical disciplines, particularly in surgical disciplines, including plastic surgery.6–8 In this Editorial, we present a survey of randomized clinical trials performed by plastic surgeons and published in the English language. A computerized search was conducted to identify the maximum number of articles published as randomized clinical trials in plastic surgery from 1966 to 2003. We elaborated strategies of electronic search for each database consulted: Latin-American and Caribbean Literature in Sciences of Health (LILACS), MEDLARS—Medical Literature Retrieval System, on-line (MEDLINE), Excerpta Medica Database (EMBASE), and Cochrane Controlled Trials Register (CCTR). Abstracts of randomized clinical trials in plastic surgery were identified and selected. Trials that had not been carried out by plastic surgeons or with the participation of at least one plastic surgeon were excluded. One author conducted the searches and the selection of trials. Whenever there was any doubt about the relevance of the study, the full text was assessed. After selection, the trials were classified into four categories, according to allocation concealment9: A, allocation concealment is appropriately described; B, allocation concealment is not described, but the authors mention in the text that the trial is a randomized one; C, allocation concealment is inappropriate; or D, the study is not a randomized clinical trial. Two reviewers independently classified the trials. Disagreements were resolved at a consensus meeting. Our sample was constituted by the trials classified into category A. These randomized clinical trials were then evaluated for their quality. The assessment was made independently by two raters and cross-checked. Two validated tools were used to assess the quality of the trials: the Delphi List10 and the quality scale described by Jadad et al.3 The Delphi List is a generic criteria list for quality assessment in randomized clinical trials; it should be used alongside other instruments.10 It does not provide scores, and it is used to evaluate topics such as allocation concealment, similarity of the groups at baseline regarding the prognostic indicators, eligibility criteria, and blinding of assessor, care provider, and patient. The quality scale3 assesses three items: randomization, double blinding, and description of withdrawals and dropouts. Each item is given one point, if described, or zero points, if not described. For the first two items (randomization and double blinding), if these methods are described and are appropriate, one additional point is given for each item. Conversely, if the methods used to generate the randomization sequence or create blinded conditions are described but inappropriate, the relevant item is given zero points. Thus, the scale produces scores from 0 to 5. A trial could be judged as having poor quality if it is awarded two points or less. The kappa statistic11 was used to study agreement between the two reviewers in the assessment of trials’ allocation concealment, in data collection for the Delphi List,10 and in data collection for the quality scale.3 The electronic search identified 4927 reports in the four consulted databases. One reviewer selected 516 reports, excluding the 4411 studies that clearly were not randomized clinical trials or that had not been carried out by plastic surgeons or with the participation of at least one plastic surgeon. After examining the full text of the 516 reports, the reviewer excluded 204 of them for the following reasons: 11 studies had not been published in the English language, 102 studies had not been carried out by at least one plastic surgeon, and 91 studies were not randomized clinical trials. Of the 312 remaining studies, 107 were common to two or more databases or they repeated inside of the same database. Because of this, the final selection had 205 studies. Two reviewers independently classified the 205 selected studies according to allocation concealment.9 After a consensus meeting, they had the following: 34 studies (16 percent) in category A, 114 studies (56 percent) in category B, 24 studies (12 percent) in category C, and 33 studies (16 percent) in category D. The agreement coefficient (kw) between the two reviewers was 0.69 (p < 0.001). When assessed by the Delphi List,10 14 of the 34 trials with allocation concealment appropriately described had similar groups at baseline regarding the most important prognostic indicators. The eligibility criteria were specified and the outcome assessor was blinded in 17 trials. The care provider was blinded in 11 trials and the patient was blinded in 20 of the randomized clinical trials. Point estimates and measures of variability were presented for the primary outcomes measures in 15 trials, and the analysis included an intention-to-treat analysis in 13 of the 34 trials. The agreement coefficient (kw) between the raters was 0.76 (p < 0.001). After the consensus meeting, the mean score on the quality scale3 was 2.4. Twenty (59 percent) of the 34 randomized clinical trials with successful concealment had a score of 2 or less. The agreement coefficient between the raters was 0.62 (p < 0.001). Many authors have reported difficulties in conducting a randomized clinical trial.12,13 In the surgical area, particularly, difficulties in obtaining an appropriate sample, in keeping both surgeon and patient blinded, and in standardizing surgical procedures and technical differences among surgeons have been emphasized.6,13,14 Although carrying out of a randomized clinical trial in the surgical area is more difficult, the difficulties do not disable its accomplishment.13 The results of randomized clinical trials with inappropriate randomization are not reliable. Therefore, authors of randomized clinical trial reports should meticulously describe the process of randomization, particularly the allocation concealment.15,16 We found only 34 trials with appropriately described allocation concealment, but 56 percent (114/205) of the selected trials were classified into category B (despite the authors reporting that the trial was a randomized one, they do not describe the allocation concealment). We believe that in many of these trials the process of randomization was appropriately designed and conducted, but the authors failed to report it. Randomization does not mean alternating between treatment and control; rather it involves a random device (e.g., a coin or a die) or a series of random numbers (a random numbers table or a computer-generated sequence of random numbers) for assigning patients in a pattern that is inherently unpredictable.17 In the majority of the 34 randomized clinical trials with successful concealment, allocation concealment was undertaken by opening a sealed envelope and randomization was determined by the use of random number tables or computer-generated random numbers. Perhaps the most difficult topic is the double-blinding. Among the 34 randomized clinical trials with successful concealment, only four succeeded in double-blinding patients, surgeons, and assessors.18–21 However, in none of these trials was the primary variable studied a surgical procedure: one trial studied the use of topical steroids and vitamin E to prevent scarring of burn wounds and grafts, another trial tested the effectiveness of desmopressin in decreasing operative blood loss in major flap reconstructions, and two trials were about the use of local anesthetics for postoperative pain relief in reduction mammaplasty. In all of these studies, double blinding was feasible because the pharmacy or a physician not involved in the procedure opened the envelope and prepared the solution to be administered. Thus, neither the surgeons, assessors, or patients knew who received what. In some areas, particularly in surgical trials, double-blinding is not feasible or appropriate, since the surgeon performing the surgery cannot be blind.22 Even trials that cannot be double-blind can, however, still be awarded more than two points on the quality scale3 if they were conducted and reported properly, thus avoiding the category of poor trials. Trials could be awarded three points if they included a description of appropriate methods to generate the randomization sequence (two points) and a detailed account of withdrawals and dropouts (one point).18 This is generally feasible, even in surgical trials. Fifteen of the 34 randomized clinical trials with successful concealment could have scored three points (and then been removed from the poor trial category) if they had described withdrawals and dropouts. Then we would have 85 percent of the randomized clinical trials with a score of 3 or more points, and we could conclude that the randomized clinical trial reports in plastic surgery were high-quality ones. The quality assessment of randomized clinical trials is an important issue, since the observation of mistakes and flaws may allow authors to avoid them in the future. It may be important to distinguish between assessing the quality of a trial and assessing the quality of its report. A trial designed with several biases but that is well reported can receive a high quality score. On the other hand, a well-designed and well-conducted trial that is poorly reported can receive a low quality score. Finally, when a physician or a scientist has to evaluate the quality of a trial, she or he normally has to rely on the reporting.23 The quality of randomized clinical trials has been evaluated by many authors in many medical areas, and most have concluded that the trials are of low quality.23–25 In our assessment of 34 randomized clinical trials in plastic surgery using the quality scale,3 59 percent (20/34) received a score of 2 points or less. On the basis of these scores and the answers to the Delphi List items,10 we conclude that the randomized clinical trial reports in plastic surgery are also of low quality. However, if plastic surgeons attempt to provide some details when reporting their trials, such as clearly describing their method of allocation to treatment, eligibility criteria, and withdrawals and dropouts and including an intention-to-treat analysis, we could change this conclusion in the near future.