Common Fallacies in Designing a Research Project: Guidance Principles

The manuscript acceptance rate for biomedical journals varies across publications and specialties. Most journals accept 20 to 30 percent of manuscript submissions.1 The acceptance rate for high-quality biomedical journals such as the Journal of the American Medical Association for research papers is as low as 4 percent, and the rate for the New England Journal of Medicine is 5 percent.2,3 The lower acceptance rates of the Journal of the American Medical Association and the New England Journal of Medicine are attributed to the journals’ stringent review process, accepting only manuscripts with the most scientific or clinical merit. In addition, these journals receive thousands of submissions annually, as more and more investigators are striving to publish in journals with high impact factors. In recent years, Plastic and Reconstructive Surgery has been receiving an increasing number of manuscripts across study modalities, including survey studies, systematic reviews, meta-analyses, large database research, and education and training studies. The reasons for instant rejection are divided into five categories: information (neither new nor innovative), size (too small), article type (case reports or historical article), goal (lack of clearly defined purpose), and commonality (similar well-executed study published recently in the Journal). A few investigators have reported on the rates of manuscript rejection either instant or following peer review. For example, a recently published study analyzed the number of total submissions to the journal Headache over 3 years and revealed that 336 (42.9 percent) of the manuscripts were instantly rejected and that the overall rejection rate was 62.6 percent.1 Hesterman et al. determined that the number one fatal flaw leading to rejection was poor methodology and study design.1Figure 1 illustrates the most common reasons for manuscript rejection before the peer-review process and after the peer-review process. In a step-by-step guide to preparing a manuscript, Balch et al. delineated reasons for rejection of manuscripts that have also been described in other articles (Table 1).4–6 Table 1. - Common Reasons for Rejection of Manuscripts Category Reasons General Not of sufficient interest to readership, manuscript does not conform to journal requirements, subject not appropriate for journal readership Methods Retrospective design, case study, small sample size, follow-up too short, invalid statistics, or control (inadequate or lack of control group) Results Nothing new, data previously presented elsewhere, data not supportive of conclusions, or outcomes worse than those with the current standard of care Discussion Poorly written (grammar/content), conclusions vague or not supported by the data, found in another article from same author, or references incomplete Fig. 1.: Common reasons for manuscript rejection before and after peer review.Similar to conducting surgical procedures, the design and execution of quality research projects require skills that can be trained through repetition and experience. Articles that are published often originate from a select group of investigators who have mastered this skill set.7 In this article, we discuss common flaws seen in manuscripts submitted to Plastic and Reconstructive Surgery, including survey studies, systematic reviews and meta-analyses, large database research, and education and training studies. We hope to help investigators gain insight into the review process, develop the necessary skills to succeed, and improve the quality of their research. COMMON FLAWS AND DISCUSSION Survey Studies Survey studies are commonly used research tools capable of answering questions about large populations that are unobtainable through other research methods. In hand surgery, survey research has been used to determine residents’ attitudes toward a career in hand surgery, incidence of postoperative complications, and disease incidence and prevalence.8–11 The major limitation in survey studies is participants’ willingness to respond. In manuscript submissions to Plastic and Reconstructive Surgery, the most common flaw is low response rate (<20 percent of the sample size). Response rate has been used frequently to judge the validity of the study.12 For survey studies to have an impact, investigators must be able to generalize the responses to a broader population. Although there is no defined “sufficient” response rate, many biomedical journals use 60 percent as a minimum response rate, whereas others consider 50 percent as adequate, 60 percent as good, and 70 percent as a very good response rate.13,14 Methods of survey administration are important factors that affect response rate. In the field of hand surgery, telephone interviews have resulted in a higher response rate (80 percent) when compared to mailed (34 percent) and e-mailed surveys (24 percent).15Table 2, adapted from Jones et al.,16 discusses the advantages and disadvantages of different methods of data collection, including personal, telephone, postal, and electronic means. Table 2. - Advantages and Disadvantages of Survey Methods* Method of Data Collection Advantages Disadvantages Personal Complex questions can be asked; allows clarification of questions; visual aids can be used; higher response rates Expensive; time inefficient; training necessary to avoid bias Telephone Allows clarification of questions; able to target larger radius than personal; less expensive or time consuming (compared to personal method); higher response rates Visual aids cannot be used; difficult to develop rapport with interviewee; training necessary to avoid bias Postal Able to target larger population; visual aids can be used (although limited) Unknown nonresponse population; time for data compilation; lower response rates Electronic Able to target larger population; visual aids can be used; quick responses; quick data compilation; lower costs Unknown nonresponse population; not all subjects accessible; potential exclusion of those without Internet access; lower response rates *Adapted from Jones TL, Baxter MA, Khanduja V. A quick guide to survey research. Ann R Coll Surg Engl. 2013;95:5–7. In survey studies, poor response rates increase several types of biases, including nonresponse bias, when those who respond differ in significant ways from those who do not respond.8 With a higher response rate, there is a lower chance of nonresponse bias. However, lower response rates can be equally as informative if the respondents represent the greater population. To determine whether the respondents represent the population of interest, investigators can perform a nonresponder analysis to compare known variables between those who did and those who did not respond. This will indicate whether a nonresponse bias is present. For example, Bot et al. investigated the factors associated with responsiveness to surveys in patients seen for hand surgery or other hand-related abnormality.17 One hundred four patients were sent the Disabilities of the Arm, Shoulder and Hand questionnaire and satisfaction survey 6 months after their visit; 35 patients (34 percent) responded to the survey.17 After analyzing the characteristics of nonresponders, Bot et al. determined that male sex, increased pain, and young age were important factors associated with not responding to surveys. A nonresponder analysis in this example was simple to conduct because the information about nonresponders was collected and easily accessible, in contrast to a survey to the general public or advertised by means of flyer in which virtually no information is known about nonresponders. In these latter cases, it is recommended to compare the demographic of the sample to published demographic information for the population to ensure similarity between these two groups. There are various methods to optimize survey response. For example, in a Cochrane systematic review of randomized controlled trials investigating ways to increase responses to postal or electronic questionnaires, investigators found that adding monetary or nonmonetary incentives, using a teaser on the envelope, an interesting topic, and shorter electronic questionnaires all increased the odds of response by more than half.18 Tips for overcoming the common flaws regarding survey studies are listed in Table 3. Table 3. - Key Points for Overcoming Common Fallacies in Survey Studies Formulate a clear study question Acquire sufficient response rate (>20%) Optimize response rate with incentives Check nonresponder bias Use short validated surveys Have a strong, impactful conclusion Systematic Reviews and Meta-Analyses The publication of systematic reviews and meta-analysis has increased rapidly across all medical disciplines, including plastic surgery. For example, the total numbers of studies tagged as “systematic review” or “meta-analysis” in PubMed-indexed articles increased from 50 review articles in 1986 to 28,959 systematic reviews and 9135 meta-analyses in 2014.19 Between 1990 and 2014, there was an increase in publication rate of 153 percent among all articles in PubMed, whereas the publication rate for systematic reviews and meta-analyses was 2728 percent and 2635 percent, respectively. The strength of systematic review lies in the well-established research protocol, clear research question, explicit research plan, and comprehensive compilation of existing data. However, many studies labeled as “systematic reviews” do not follow the methodologic rigor of the systematic review protocol. For example, Moher et al. conducted a systematic review of the epidemiology and reporting characteristics of systematic reviews, resulting in over 1000 citations.20 Of these, only 300 studies were included; reasons for exclusion of initial results included lack of intent to review literature or explicit methods, literature reviews, review of reviews, and methodology papers. The Preferred Reporting Items for Systematic Reviews and Meta-Analyses checklist is a 27-item checklist that serves to improve reporting quality of systematic reviews and meta-analyses.21 Many journals, including Plastic and Reconstructive Surgery, endorse the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines and recommend that authors adhere to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses checklist and use a Preferred Reporting Items for Systematic Reviews and Meta-Analyses flow chart when reporting systematic reviews or meta-analyses.22 Reasons for noncompliance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses checklist have not been determined; some investigators suggest that it is attributable to either a lack of awareness of the checklist or a lack of understanding of the items within the checklist.23 For example, in an analysis of the reporting quality of systematic reviews and meta-analyses in plastic surgery, Lee et al.24 focused on systematic reviews published over a 2-year period in five major plastic surgery journals with the highest impact factor, including Plastic and Reconstructive Surgery. The investigators revealed that the median score was 16 of 27 (59 percent) and the average compliance of an article was 68 percent. Compliance for each item on the checklist ranged from 5 to 99 percent. The two items with lowest compliance on the checklist were “indication of review protocol and registration information” and “present data on the risk of bias of each study.” In addition to reporting quality and compliance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses checklist, common flaws in manuscripts submitted to Plastic and Reconstructive Surgery include poor and heterogeneous data. “Poor data” refers to systematic reviews or meta-analyses with a limited review of studies, often fewer than 10. Heterogeneity can be categorized into clinical, methodologic, or statistical heterogeneity. Clinical heterogeneity is defined as differences in participants, interventions, or outcomes; methodologic heterogeneity is the variability in study design and risk of bias; and statistical heterogeneity is the variability in intervention effects.25 Heterogeneity in systematic reviews and meta-analyses leads to nondefinitive conclusions that cannot answer the study question. Recommendations to overcome the common flaws discussed in this section are listed in Table 4. Table 4. - Key Points for Overcoming Common Fallacies in Systematic Reviews and Meta-Analyses Formulate a clear study question Use the PRISMA checklist and flow chart Review ample number of studies (>10) Have a strong, impactful conclusion PRISMA, Preferred Reporting Items for Systematic Reviews and Meta-Analyses. Large Database Research Large population-level databases contain aggregated health information that serves as a useful tool for answering a variety of questions. Database research is also advantageous because it decreases inherent bias in case series from single institutions and increases external validity.26 There has been a rapid increase in large-database publications per year in surgical journals. For example, fewer than 100 large-database studies were published in 1995, whereas 1538 were published in 2014.26 Although there is increased accessibility of large databases, these databases can be misused and result in erroneous conclusions.27 Common flaws seen in large-database studies submitted to Plastic and Reconstructive Surgery include poor methodology, lack of originality, and lack of involvement from a trained statistician. Methodologic flaws include poorly formulated questions or hypotheses and inappropriate database selection. Rather than being “hypothesis-driven,” many investigators are “data-driven” and purely exploratory, mining for the purpose of finding statistically significant data. Formulating questions using Feasible, Interesting, Novel, Ethical, and Relevant criteria or the Patient, Population, or Problem; Comparison or Intervention; Outcome format can help frame a clear research question. These frameworks also assist the investigator in defining the population of interest to adequately answer the question.7 Another common flaw is choosing the appropriate database. There are multiple databases available with varying information. JAMA Surgery recently published a 13-part series to provide an overview of the 11 most commonly used data sets along with their strengths and weaknesses.7 In the field of plastic and reconstructive surgery, Adkinson et al. analyzed 19 databases and characterized their differences by size, type of information, data collection methods, acquisition costs, and variable availability.26 Understanding these differences helps investigators choose the best database to answer their research question. For example, in JAMA Surgery, attempting to address a research question inadequately with the proposed data set is a common reason for rejection before the manuscript is sent out to peer-reviewers, whereas major methodologic flaw is a common reason during the peer-review stage. Lack of originality refers to large-database studies that are repeats of studies that have been conducted in other fields. In these manuscripts, investigators recycle the methodology of an existing study and change it to plastic surgery. Involvement of a trained biostatistician early in the process is useful for evaluating the methods and analytic strategy, and for performing appropriate data analysis. This is essential because a common reason for manuscript rejection before peer review is a flawed data-analytic technique.7 Other common flaws in large-database studies include failure to acknowledge the limitations of the data set or to address any missing data.7,28 Acknowledgment of limitations of data demonstrates critical assessment of the research project and current understanding of the literature. In addition, limitations are important considerations for readers when interpreting the validity of the research and credibility of conclusions.29 It is particularly important in large-database studies to recognize and discuss the proportion of missing data, because this can impact analytic techniques and conclusions drawn from them. For example, if patients of low socioeconomic status are more likely to miss follow-up visits, it would be difficult to analyze the effect of socioeconomic status on postoperative complications or surgical outcomes. Recommendations for overcoming the common flaws seen in large-database studies are listed in Table 5. Table 5. - Key Points for Overcoming Common Fallacies in Large-Database Research Formulate a clear study question Involve a trained biostatistician early on Choose the appropriate database Avoid data mining Avoid recycling the methodology of a study published in another field Acknowledge limitations of data set Address any missing data Have a strong, impactful conclusion Education and Training Studies Technological advancements have revolutionized medical learning and practice. For example, cadavers and animal models were the only simulators available for surgical skill training, but now various simple inanimate models are easily accessible to hone basic surgical skills. Other technological advances include computer simulation and training, computer-assisted preoperative training, computer-assisted surgery, telemedicine, and medical robotics.30,31 These developments have moved a part of surgical education from the operating room into the classroom; surgical trainees can begin refining their skills before even entering the operating room. Kazan et al.31 noted that simulators can be used as a prerequisite to increase surgical competencies for “high-technical-skill” procedures or increase familiarity of “low-exposure” procedures. Education and training studies implementing these technological advancements are important in the education of surgical trainees and established surgeons. The quality of education and training manuscripts submitted to Plastic and Reconstructive Surgery regarding education have some most common flaws, including poor methodology and predictable conclusions. For example, consider two groups of participants where one will receive training and the other will not (“control group”). As expected, the group that received training will do better compared with the control group that did not receive training. Table 6 contains the recommendations for overcoming the flaws discussed in this section. There needs to be better, more reliable studies to optimize education and training in plastic surgery. Table 6. - Key Points for Overcoming Common Fallacies in Education and Training Studies Formulate a clear study question Effective methodology Avoid predictable conclusions Have a strong, impactful conclusion The “So What?” Factor The foundation of any research project begins with a clearly formulated question. A common flaw that can be applied to all the study modalities mentioned above stems from being unable to answer one question: “So What?” Submissions that lack novelty or originality, are convoluted by unclear impact statements or conclusions, or express low scientific or clinical significance are a challenge to acceptance in Plastic and Reconstructive Surgery. Manuscripts that lack novelty or originality are those topics that have been well studied previously, use information found in standard textbooks, present no new perspectives, or have an answer that is obvious. For example, in a retrospective review of 1000 medical and dental manuscripts submitted to the Journal of Clinical and Diagnostic Research, “commonality,” defined similar to our definition of lack of novelty or originality, was the most common reason for manuscript rejection.32 These common flaws are similar to those cited by other investigators as reasons for manuscript rejection.1,4,16,32 High-quality manuscripts need to have a clear take-home message to help readers understand the scientific and clinical implications of this project, such as how the study can impact science, eliminate or bridge gaps in current knowledge, introduce a new procedure or or current clinical practice. The “So What?” is an essential in the of and AND Survey Survey studies with low response rates (<20 percent) have a high of being rejected without review. For those that the minimum of response rate, Plastic and Reconstructive Surgery the investigators to provide data and statements that the sample most likely the broader population of Systematic review and Plastic and Reconstructive Surgery will systematic reviews or meta-analyses with fewer than articles manuscripts with more than articles but with conclusions will be rejected fatal heterogeneity of the data or the conclusion that the data are limited that better studies need to be which is not Database studies with unclear impact with clinical or scientific or that to answer the “So What?” question will be In addition, manuscripts that recycle the methodology of a project in another field are not to the Plastic and Reconstructive Surgery Education and training with poor methodology and that lack a control or be rejected without review. These guidelines are to increase in the process and guide investigators who to publish in Plastic and Reconstructive Surgery. In addition, the information in this article can help investigators avoid the common seen in rejected manuscripts and the for high-quality research projects to be published in journals. Research reported in this publication was supported by the of and and of the of number The is the of the authors and does not represent the of the of