Surveying Emergency Medicine

Academic Emergency Medicine (AEM) frequently receives manuscripts that include data from surveys. In this commentary, we present an overview of survey methodology, items to consider when designing a study that involves questionnaires, and best practices for reporting survey methodology and presenting findings in manuscripts for AEM. Two survey-based studies published in this issue are used to illustrate some of these points. This commentary is not meant to be a detailed exploration of the nuances of survey methodology, but rather an overview to guide the emergency medicine (EM) researcher on topics to contemplate when considering utilizing survey tools in research. Readers are encouraged to review additional resources on survey methods for researchers.1-3 A survey can be a study's only research tool or can be used as an adjunct to other data collection in allowing researchers to answer their study questions. An important tenet when contemplating a study is that the research question is defined first, and then the researcher decides whether using a survey is an appropriate research tool to answer that question. Data collection using questionnaires provide opportunities to measure respondents' attitudes, behaviors, characteristics, knowledge, observations, opinions, and perspectives. Questionnaires can be used in qualitative, quantitative, and mixed-methods research. Although respondents typically report on behalf of themselves, the respondent can be a proxy for others (e.g., parents responding for children) or an observer. Decisions on what type of data are collected and who responds affect questionnaire design, development, and reporting of results. A common element of survey research is that the collected data are generally “subjective,” even if quantitative data are gathered. Data are not just recordings from a monitor, but responses are affected by respondent's understanding of the questions and responses, recall, and their affect and current perspective. In addition, quality of the questionnaire, methods of administration, participant selection, sampling methods, and analytic techniques also affect the collected data and its interpretation. Data accuracy from survey research is highly affected by the quality of the instructions, questions, and responses. Creation or selection of a high-quality questionnaire is paramount, and a researcher's overarching goal is to prepare a questionnaire that is valid (measures what intends to) and reliable (gathers data in a consistent manner). Strategies need to be used to reduce bias, including from inability of participants to understand the instructions, questions, and responses; inappropriate questions (cultural, situational errors); response fatigue; or respondent sabotage (purposely providing false data). If available and appropriate to the study, employing previously used study instruments can be more efficient than de novo questionnaire development. Questionnaires that have been rigorously developed and evaluated and have known psychometric properties demonstrating their validity and reliability in populations similar to those for the intended use are optimal. However, publication and prior use of study instruments are not equivalent to validation.4 An example of a validated questionnaire that most emergency physicians (EPs) are clinically familiar with is the CAGE screening questions for alcohol abuse that has established validity and reliability metrics.5 Researchers need to be critical when reviewing potential survey instruments for inclusion in their studies, and evaluate the development process of these instruments. Further, questionnaires from previous studies might be outdated or otherwise might not be appropriate for the study or participants (e.g., not age, culturally, or developmentally appropriate). Many studies require creating new questionnaires to answer the study's research question. Questionnaires can be modified or adapted from previous studies with the understanding that this could affect their psychometric properties. Response options for questionnaires, such as binary, categorical, open-ended, scalar (e.g. numerical, visual analog, or Likert), word scale, symbols, and mixed response options, must also be chosen carefully in question selection, adaptation, or construction. Whether constructing new or adapting or using previously used questionnaires, the questions and responses should 1) be commensurate with the intended constructs and concepts that they intend to measure, 2) support the goals of the research, 3) match the hypotheses of the study, and 4) be appropriate to the population under study, the study design and setting, and administration of the questionnaire. Response burden needs to be considered and respondent fatigue must be balanced against the study's requirements to collect meaningful and comprehensive data. Strategies exist to improve engagement through breaks, use of pictures, single-screen questions and responses, and the appropriate use of skip patterns (with clear instructions, or integrated computer-based logic). Although a comprehensive review of good instruction, question, and response design is beyond the scope of this commentary, there are some common pitfalls that researchers should avoid. Be careful to avoid long, conditional, or multiple components (e.g., “and”/ “or” statements); double negatives; slang and idiomatic or colloquial expressions; medical vernacular; value-laden biases of the researcher; and absolutes (e.g., always, never) in instructions, questions, and responses. Using questions that cannot be answered by the respondent (e.g., views on what another person is feeling, attempting to assess knowledge about a topic for which he or she has no knowledge, hypothetical scenarios that cannot be accurately or reliably considered, abstract or ambiguous concepts) yields uninformative data. Design of response options also needs equal attention to avoid incomplete response options, overlapping response options, responses that do not fit the nature of questions (e.g., using scales instead of “yes”/”no”/”don't know” responses), including options that respondents will choose as defaults instead of answering or addressing the question, and not including an option for respondents to use when none fit for them. Instructions should be clear, concise, and easy to follow and should also provide context for questions and responses (e.g., time frames for the questions and responses, indicating whether a single response option is required or more than one option can be chosen). Many studies require surveying a population that is language and culturally heterogeneous. Questionnaires should be translated to languages used by study participants using accepted translation and back-translation techniques by qualified personnel to ensure translation accuracy.6 Requiring study personnel to translate questionnaires during administration can introduce variability and lack of assurance in the accuracy of the translation and can threaten the survey instrument reliability and validity within and across populations studied. Questionnaire content should be reviewed carefully for slang and colloquial or idiomatic expressions that can be directly translated, but might not be valid in other languages. Cultural appropriateness is a different imperative than translation accuracy and reflects the importance of ensuring that the questionnaire and study methodology chosen is acceptable to, and can meet the study objectives for, the populations studied. Survey questions, responses, and instructions, as well as the study methods, need to be verified to ensure that they are appropriate to the cultures of respondents. This process generally requires members of the cultures under study to assist in the study design and in the questionnaire development and evaluation process.7 In addition, the readability of the questionnaire should be commensurate with the literacy abilities of the study population. Language and overall readability can be measured with online programs or word processing software programs (e.g. Flesch-Kincaid grade level, Coleman Liau index, Automated Readability Index, SMOG, Flesch reading ease, Fernandez-Huerta Readability Index Spanish). Questionnaire administration and delivery also can affect collected data. Researchers should carefully consider whether questionnaires should be interviewer- or self-administered in light of study feasibility, goals, and the nature of the data collection. Electronic delivery (e.g. e-mail, Web based, text message) of questionnaires to respondents can have advantages over written questionnaires distributed in person or by mail in the efficiency of delivery and collection of data, but has barriers in potentially excluding those with limited access to these delivery methods. Electronically administered questionnaires have the advantage of being able to incorporate skip patterns and visual media to enhance the survey experience. Written surveys are labor-intensive to administer and to transfer the collected data into electronic databases, but negate the need for use of and access to electronic delivery systems. Electronic and written questionnaires are subject to the literacy capabilities of the study population and limitations of the visually impaired. However, electronic-based questionnaires can be enhanced with voice-enhanced administration methods (interviewer or audio-computer self-administered). The amount and veracity of data collected can be greatly enhanced if an interviewer helps guide respondents and uses reflective or follow-up questions (e.g., qualitative research), but interviewer administration has the potential to inhibit accuracy (e.g., reduces true responses to sensitive topics) or create bias (e.g., encourages respondents to answer in a way to please the interviewer). After developing a draft questionnaire, the researcher first needs to critically review the questionnaire content, format, and construct, to assure confluence with the study goals. It is frequently helpful that the questionnaire and study protocol are reviewed by a survey methodologist. A cognitive-based assessment of the draft questionnaire among a sample similar to the intended study population is recommended. These assessments consist of interviewing participants regarding the content, format, style, and intent of instructions, questions, and responses and are helpful to ensure that participants interpret the questions and responses as presented and intended. The questionnaire can be revised based on the results of these assessments. In addition, pilot testing of the questionnaire and study methods is useful to examine survey sampling, administration, response rate, reactions, and data collection methods and provides researchers a period to refine these aspects, based on observations and experiences. Occasionally studies only pilot test questionnaires with a few colleagues, but this should instead be done among a sample similar to the study population and in an environment or with a delivery method similar to the proposed study's methodology. Pilot testing also permits evaluation of respondents during participation (e.g., unexpected behaviors, nonverbal responses, confusion, and inability to follow or understand instructions) and assessment of response time and burden and allows collection of a sample data set, which can be examined for quality and ability of the data to meet study goals. Consultation with a biostatistician or psychometrician can be useful for psychometric analyses of the questionnaire to examine its reliability (e.g., test–retest) and validity (content, construct face, etc.), perform factor and subscale analyses, and measure internal consistency (Cronbach's alpha). Most importantly, the biostatistician's or psychometrician's continuous involvement in developing an analysis plan in parallel with questionnaire development will allow the questionnaire to have greater success in answering the study objectives. Most studies do not survey the entire population of interest but rather a selected sample. Selection of a sample from the population of respondents should be designed to maximize external validity (“generalizability”). Sampling in survey studies can utilize many of the same sampling techniques used by other types of studies. Random sampling is the preferred method for most research but can be a challenge to perform adequately in the ED environment. Methods of random sampling can vary, but ultimately should reflect an attempt to select an unbiased and representative sample of the underlying population of interest. Other sampling methods can be employed (e.g., quota-based sample, snowball sample, time–space sample, location sample) and decisions on which method to use are based on meeting research goals and objectives. Sample size calculations are needed prior to conducting to the study for anticipated comparisons involving quantitative data to ensure adequate power to detect differences. Once a sample is selected, the goal is to achieve a high response rate from that sampled group, and techniques exist to maximize this rate.8 Response rates affect the generalizability of findings and a rate of greater than 70% to 75% is desirable,9, 10 although a response rate with physician surveys of 10 percentage points less has been acceptable in self-administered postal surveys.9, 10 The use of questionnaires in research protocols requires some special attention when describing the methods and findings in manuscripts.9-12 We have summarized in Table 1 and in this section some best practices for manuscripts submitted to AEM that supplement the journal's standard guidelines for authors. The manuscript should state the research question(s) being addressed and why the use of a survey was the appropriate method to answer those questions. The methods section, as in other research reporting, requires sufficient detail to allow others to replicate the study. Study protocols with questionnaires require additional information, including details on questionnaire development and pilot testing. A copy of the study's questionnaire should be included when the manuscript is submitted to the journal for review. This allows the editors and journal reviewers to evaluate the questions employed and not just the results. If the manuscript is accepted for publication, the editor can discuss with the author inclusion of the questionnaire in either the print or the electronic version of the journal. A description of the study location, sampling techniques, participant selection and assessment, inclusion and exclusion criteria, consent procedures, administration and delivery, and psychometric properties are necessary to allow reviewers and readers to independently assess the internal and external validity of study findings. Although there are no specific generally accepted guidelines for reporting of study sampling and enrollment for survey-based research, it is helpful to include information that conforms to recommendations in the CONSORT (http://www.consort-statement.org) or STROBE (http://www.strobe-statement.org) guidelines. The response rate, as well as a discussion of how that was calculated, is necessary. In addition, data are needed on those refusing survey participation or exiting the study and how the sample differed (or not) from the population being studied. Sample size calculations and assumptions, analytic techniques, and methods of handling missing data (e.g., imputation) should be specified. The analytic methods should be appropriate to the study goals and objectives, type of data, questionnaire, and survey methodology. In discussing study findings, authors need to explain how the survey findings answer the study's research question and how this contributes to advancing knowledge on the study topic. Acknowledging limitations specific to the survey methods should include any limitations due to questionnaire development or administration, the population being studied, sampling methods, response rates, completion rates, and missing data. All of these need to be discussed in reference to their effect on the study findings and its external validity. This issue of AEM contains two studies using surveys of clinicians that add to EM's knowledge about clinical decision rules and clinical decision tools. Ballard et al.13 used a questionnaire to examine knowledge, attitudes, and behaviors of using a pediatric clinical decision rule across a large health care network. The authors describe well their methodology of questionnaire development including the theoretical basis for their questions and having incorporated previously published research into their question development. Although the study did pilot test the survey, more description of how that pilot testing was incorporated into the final survey instrument or its delivery would have been useful. The researchers could have also provided some descriptive information on how respondents were similar to, or different from, nonrespondents. The article could have benefitted from a participant flow diagram similar to the one in Figure 1 of the second study, by Kinlin et al.14 This second article has some additional examples of good methodology that warrant mentioning. It describes how the study sample was identified and recruited and provides data describing both respondent and nonrespondents. The study was strengthened by its use of objective data to collaborative questionnaire data and pretesting of its questionnaire (although providing more details of that process is needed). Each study also included a copy of the survey instrument with the manuscript submission to assist in the review process. The use of questionnaires in research can contribute needed data to answer a research question. Like other research methods, survey-based methods have some unique attributes that necessitate attention in study development, study conduct, and reporting of study findings. The journal welcomes manuscripts describing research that includes questionnaires that have sound methodology and when their reporting conforms to the journal guidelines.