Methods for Testing and Evaluating Survey Questions

An examination of survey pretesting reveals a paradox. On the one hand, pretesting is the only way to evaluate in advance whether a questionnaire causes problems for interviewers or respondents. Consequently, both elementary textbooks and experienced researchers declare pretesting indispensable. On the other hand, most textbooks offer minimal, if any, guidance about pretesting methods, and published survey reports usually provide no information about whether questionnaires were pretested and, if so, how, and with what results. Moreover, until recently there was relatively little methodological research on pretesting. Thus pretesting’s universally acknowledged importance has been honored more in the breach than in the practice, and not a great deal is known about many aspects of pretesting, including the extent to which pretests serve their intended purpose and lead to improved questionnaires. Pretesting dates to the founding of the modern sample survey in the mid-1930s or shortly thereafter. The earliest references in scholarly journals are from 1940, by which time pretests apparently were well established. In that year Katz reported, “The American Institute of Public Opinion i.e., Gallup and Fortune i.e., Roper pretest their questions to avoid phrasings which will be unintelligible to the public and to avoid issues unknown to the man on the street” (Katz 1940, p. 279). Although the absence of documentation means we cannot be sure, our impression is that for much of survey research’s history, there has been one conventional form of pretest. Conventional pretesting is essentially a dress rehearsal, in which interviewers receive training like that for the main survey and administer the questionnaire as they would during the survey proper. After each interviewer completes a handful of interviews, response distributions may be tallied, and there is a debriefing in which the interviewers relate their experiences with the questionnaire and offer their views about the questionnaire’s problems. Survey researchers have shown remarkable confidence in this approach. According to one leading expert, “It usually takes no more than 12–25 cases to reveal the major difficulties and weaknesses in a pretest questionnaire” (Sheatsley 1983, p. 226). This judgment is similar to that of another prominent methodologist, who maintained that “20–50 cases is usually sufficient to discover the major flaws in a questionnaire” (Sudman 1983, p. 181). This faith in conventional pretesting is probably based on the common experience that a small number of conventional interviews often reveal numerous problems, such as questions that contain unwarranted suppositions, awkward wordings, or missing response categories. However, there is no scientific evidence to justify the confidence that this kind of pretesting identifies the major problems in a questionnaire. Conventional pretests are based on the assumption that questionnaire problems will be signaled either by the answers that the questions elicit (e.g., “don’t knows” or refusals), which will show up in response tallies, or by some other visible consequence of asking the questions (e.g., hesitation or discomfort in responding), which interviewers can describe during debriefing. However, as Cannell and Kahn (1953, p. 353) noted, “There are no exact tests for these characteristics.” They go on to say, “The help of experienced interviewers is most useful at this point in obtaining subjective evaluations of the questionnaire.” Similarly, Moser and Kalton (1971, p. 50) judged, “Almost the most useful evidence of all on the adequacy of a questionnaire is the individual fieldworker’s i.e., interviewer’s report on how the interviews went, what difficulties were encountered, what alterations should be made, and so forth.” This emphasis on interviewer perceptions is nicely illustrated in Sudman and Bradburn’s (1982, p. 49) advice for detecting unexpected word meanings: “A careful pilot test conducted by sensitive interviewers is the most direct way of discovering these problem words” (emphasis added). Yet even if interviewers were extensively trained in recognizing problems with questions (as compared with receiving no special training at all, which is typical), conventional pretesting would still be ill suited to uncovering many questionnaire problems. Certain kinds of problems will not be apparent from observing respondent behavior, and the respondents themselves may be unaware of the problems. For instance, respondents can misunderstand a closed question’s intent without providing any indication of having done so. Moreover, because conventional pretests are almost always “undeclared” to the respondent, as opposed to “participating” (in which respondents are informed of the pretest’s purpose; see Converse and Presser 1986), respondents are usually not asked directly about their interpretations or other problems the questions may have caused. As a result, undeclared conventional pretesting seems better designed to identify problems the questionnaire poses for interviewers, who know the purpose of the testing, than for respondents, who do not. Furthermore, when conventional pretest interviewers do describe respondent problems, there are no rules for assessing their descriptions or for determining which problems that are identified ought to be addressed. Researchers typically rely on intuition and experience in judging the seriousness of problems and deciding how to revise questions that are thought to have flaws. In recent decades a growing awareness of conventional pretesting’s drawbacks has led to two interrelated changes. First, there has been a subtle shift in the goals of testing, from an exclusive focus on identifying and fixing overt problems experienced by interviewers and respondents to a broader concern for improving data quality so that measurements meet a survey’s objectives. Second, new testing methods have been developed or adapted from other uses. methods interviews, response respondent and The of these methods issues of how they be in as well as whether they in lead to in survey In the of of poses special for pretesting, as do of special such as and questionnaires in more than one of which have in recent these on the research in the to testing and questionnaires interviews focus on to the interviews, by focus on providing a of the by the or are to reports of the respondents have either as they the survey questions or The is to reveal the thought in a and at an are to problems with the Although is not with a of this approach. In the designed interviews to questions respondents been asked the during a by a were of the exact and the they to The interviewer were asked that what the After to what the to the respondent, interviewers how from that for as only this time for to how the was interviewers about aspects of the the questions and were to test about problems to each of the to the respondents were to how they would their from the one they the they were asked to interviewers, who two of conducted interviews with a sample of to the and the of problems. interviews in a way from interviews, which the and interviews in a with one that respondents their from an conducted the which is with what we know about the of However, in many such as and which one or two to respondent interpretations of survey is that little on pretesting an to so The leading to a for interviews in pretesting not until two decades with the of Survey was of how respondents survey questions about in which on the by and to problem and a from at the to both research on the for to survey questions and and the at the for in of the the be to pretest questionnaires (e.g., and were by other and survey The not to pretest questionnaires. by special from of and survey pretesting for the and the the and their in training (e.g., or for pretesting (e.g., Although there is about the of no has about such as whether to whether to or and how to and evaluate results. In this is to the of methodological research these is to a of to the for interviews to survey pretesting. As and that reports are more to be if they information a has in (as opposed to and if the not thought (e.g., not Thus some survey instance, of information may be well suited to in a However, the of report methods to in survey questions is to for as that do not the for that the in interviews may a assumption by and for of the has problems with the methods typically to for that kinds of difficulties that respondents would not of a of interviews that respondents who for an little an who for from the with interviews, such as were up with or to for than of all interviewer of the respondent in a for for such as more about or of the including and (e.g., what to or know what Thus to be by the which may not be well in that the As one way to deal with this that interviewers be trained to and the in which each ought to be and that report quality should be in of problem and problem which are the goals of They designed an of two the of experienced the more experienced interviewers trained to only when there were the respondent was a The conventional interviews identified many more problems than the As in more problems not results. and the of problem in two by a of trained who of the interviews, and by and was with the of some other researchers about the of data and was for the interviews than for the conventional may be to the interviewers having training in what should be a compared to the conventional interviewers who were no of what a Furthermore, as interviewers much often than conventional interviewers, more of their were in cases with the of a Thus we to what interviewers do in The importance of this is by and who conducted an in which were to have two interviewers each interviews of the questionnaire methods were for the and a report identifying problems in the questionnaire as well as a questionnaire the problems. In of the questionnaire were from who were not in the another of interviews was conducted by a to test the questionnaires. The on many aspects of the interviews, including research interviewer interviewer research and data of interviewer and This is with the of and Presser is even more the many in which some of have that in the the cannot be to any one was the in both the number of questions identified as having problems and the number of problems Moreover, there was only the in the problems the interviews and the much more in identifying which questions problems than in identifying what the problems The that identified the problems the with the This was the only one that not the in the which that on interviewer and to However, the from the tests of the questionnaires not identify one as better or than the In research on interviews has to reveal how the methods to the interviews the data Yet much more is to provide a for interviews, which are from conventional other testing methods that have been developed may be as to conventional pretests as to a survey response respondent and was developed in the by Cannell and at the of Survey and can be to evaluate both interviewers and were almost on interviewers, so no on pretesting In the and a researchers to was not to pretesting in the until the and as one of two for pretesting interviews or interviews for a of the and in the asking and by of (e.g., the interviewer not the or the respondent are as and describe an of that on the of interviewer and respondent not the of the individual on the of a question’s an is as either interviewer the the respondent one of the and the interviewer the was the problem was the respondent asked for and one of the or was and the problem was not with a of are identified as and compared the from this in a survey of the with the from and from methods not data a by by the by two questionnaire and the developed by and which we describe in the The two methods based on similar as of the the two of methods identified problems. As and the methods point what go with the the and reveal what go testing based on observing the of the time takes a respondent to a most questions are the kind of of a that is almost without Thus was not until the of survey in the that the of response was as a testing and and response to evaluate the of answers and, to evaluate the questions The that respondent and they this by the of and answers by information from another In they compared the of response to that of other of In a both response and the of about their answers were with research and reports no in some an confidence or and the of their Thus to a more of the in which of respondent are useful in response the that the of response is than that of other of problems may careful as opposed to the to This is so as the of information means be in at The of data many would improved of the and of response and of how be with other testing methods, such as to the of questionnaire problems. and response which are “undeclared” testing methods, respondent are a “participating” which the respondent about the purpose of the have been as a to conventional pretest interviews p. they most have been conducted as by how in a can reveal both the of questions and the respondents have to the In how can be to the extent to which questions lead to or that respondents may be in either undeclared or well suited to how about test whether interpretations of are with that are the of and other problems. evidence of by on evaluations of questionnaire on the of the The research we have that the to conventional pretests in the kinds of problems they are suited to their for the of a problem and for the of their and the to for instance, that respondent and are more than and response to identify of problems. Yet we do not have of many of the the methods The is not only that we research designed to such that testing methods are probably in many cases to that respondents the questions and are and to of see and and and methods to conventional pretests and interviews identify questionnaire problems and lead to designed to the problems. whether the are there is no for of the and are of two First, the and can be compared the testing that identified the if interviews respondents with an the and can be in another of interviews in to that the such problems than the The of from this kind of is usually there is no that will have any on survey Second, and can be to if any, they for a survey’s The from this kind of is such have a in pretesting. they were the of one of the earliest to pretesting the with the on to of such as questionnaire of the of and questionnaire of the to evaluate the of that in response a and as for instance, by how interviews and are the identify problems and and the test the of the As evidence is in whether survey and if so, by how on of that in only one can be to test of questionnaires that in as by researchers the Survey of and questionnaire to meet major to response and both and to and to questions about new to the of the to the new an was conducted in which respondents were to either the new or and were with the new and, with one the of and in the not On the other hand, were not (in in was for the and the new questionnaire to that these may have been by two of the First, experienced interviewers were for both the and new The with the questionnaire to in their may have to their more than the new questionnaire and more respondents to with Second, the of new to the may have more than the that were to the that the that many to of as in the the of questionnaire the research with a of the of the which is to the not the of individual which is to an of the of questionnaire to of other to of not only about the of about the of as of (as well as more of for which because they are usually much than is and are usually thought of as Researchers who in questionnaires to have and who in have little for This is as the two should in for survey research to Moreover, the problem is not In the of survey and their to both questionnaire and (e.g., Thus is that one recent to evaluate questionnaires on a in how may be to the with questions when the questions have been asked of the respondents two or more as on an and there is usually no direct way to evaluate the that than on a for methods ought to be research like on individual survey have on the of of many of test have little information about the of the and describe how response can the to which respondents who have the on a The of to identify the of researchers to that do a better of can be by methods to tests a respondent is a the of the and an of is based on is based on that and the is each the of the is and when the is no more are and response of cases and are relatively to no new data is to of a by the and to the and of a sample of questionnaire and the be to the and of and a data of that of and for questionnaire They developed a to the to the of their of response in the data sample and the in which these were the and the or were the provide of the on the or of the can be by the and the to their Although more data are to the more to be in such a for The of has many aspects of questionnaires. On the one hand, the of new with to and our to a of more and with improved data quality On the other hand, the and the many for questionnaire The of data has at for the and testing of survey is the that answers to survey questions may be by the in which the questions are testing methods the is that survey of much more than their and and and the aspects of the and to of these to be and their on is that survey are more and for The methods that on to test and are no methods be to the testing of from the of individual In the of testing questionnaires has the growing of survey and the of for has an and of the testing often with no of of this testing can be done and only by is often not up to the and The of information to the as well as the of by the interviewer in are to Thus testing for can be as as testing for As and questionnaires interviewers to two one with the and another with the respondent, and the of be to help interviewers both to data and provide of the in which testing in this focus on is even in the and how interviews may be adapted to the aspects of in questionnaires. They show how the of interviews can in the of testing of aspects to that such as the of and of the that the survey In to testing methods in other and evaluations based on data that are during (e.g., response and and and the importance of testing and, with and and and they that the testing of questionnaires focus on their of and that questionnaires in special problems. pretesting is still more in these cases than is for of with questionnaires in a pretesting has been even for such than for As a result, the methodological on pretesting is even for these cases than for of describe of that have questionnaire pretesting to be to rely on and to of a which are with and many which are to have to the respondent by In because are designed to questionnaire are to rely on to to the and testing of questionnaires. In to of conventional methods, they like focus and with and other They better of quality evaluations and as well as more documentation of to provide of questionnaire problems. they that tests be in so that can be without the In for and and of for guidance about the kinds of questions and that can be asked of of The evidence that is about the earliest at which can be with the to kinds of questions (e.g., is only The how pretesting methods, including focus interviews, and can be adapted to of and they provide of pretests that these methods with has always been to and recently has for as (e.g., and administer in by are questionnaires as a of growing of In the for instance, the and that a at other than from in to in Moreover, by of in in which all the a other than and reports that of survey research has been to than of in questions and the problems in response or so they are to problems from asking questions about a (e.g., with in each (e.g., so that problems with a do not in for all the questions that are and as well as that are and special to and offer guidance on and for and They a more of than the common to most They the for of the do not is and they that the quality of as well as the of the questions as survey be they that of to on the and they to the of and documentation of which is for a survey is pretesting lead to better know of only one that this and whether pretesting data problems and survey The and identify problems in a pretest of a questionnaire of The questions most by the methods as having problems were to the problems. a was conducted to the and The interviews were and the interviewers were asked to evaluate the questionnaires the The of the questions identified in the pretest as to problems for interviewers were more to show problems in the and to be identified by interviewers as having problems for Similarly, the questions identified by the pretest as problems for respondents in more respondent problems, to both the and the interviewer was for questions identified by the pretest as either or problems than for questions not identified as having problems. Thus the of pretesting methods was a of the problems the would in the However, the questions not to the The no on the of interviewer and respondent problems. interviewers the as respondent problems, they as more interviewer problems. The for this including their of only questions as most which often problems that to In the questions were not to another of testing the methods that identified the problems to that the were the are as they that we have much better for questionnaire problems than for fixing pretesting methods, and of the the and of problems methods is often and the for this more is in their some of the methods are two other are First, of may because the methods are suited for identifying problem For problems that with no in the asking and are to be up by we should probably only in the problems identified by pretesting Second, may a of interviewers, or about what is as a For is a problem if a is awkward to or is only a problem if the The kinds and of problems that a questionnaire pretest methodological to identify are not always and this of may to the that have been In such the by and see and not only of leading to and to to our of which methods are in and for is that problem not point to problem in any or direct For instance, and and and pretesting to identify problems that were by only to in that the either not in or new problems. The that we are better to identify problems than to the of testing questionnaires have been to for questionnaire testing and the problem and the of testing methods on survey the of research and in the of and the of a data to First, we that the problem and major of testing is to we know little about the to which is by the problem at the of the testing and and and are in of in this research has an by the problem and response missing or on the assumption that or are more based on be more than based on direct (e.g., designed we be better to for methods, evaluate the of methods for kinds of problems, and how much pretesting is that data are and are themselves to Thus another for research is to of that can be to testing Second, we information about the of testing methods on survey The of testing may be or even more than the survey on whether the testing lead to the of problems that survey interviewer and data have for Although we know about the direct of testing methods, we know almost about how the methods in their on Thus a for research is to how testing methods in identifying the kinds of problems that survey improved methods for problems are useful to the extent that we can the problems, we more guidance in As a result, in pretesting on in the of asking questions and Presser a research the and that is and and and this is a On the one hand, pretesting should be by research the and On the other hand, research and of the and should be by both the of pretesting and in the testing methods the or in questionnaire and and the kind of by and In pretesting’s focus on aspects of the response that can for respondents to ought to of the response and the and we improved to This will to what is from pretests of individual questionnaires. of the at the on the of Survey p. that survey in a the interviews they including the the methods and the As that this would research issues such of questions are identified by as testing are with discovering problem of are in response to of this should to all methods of pretesting. a pretesting on the would not only research on questionnaire would serve as an for researchers questionnaires for new This is a of from Presser the methods in this data to test a questionnaire. do not focus and or interviews which are most at an there is an to be do we evaluations by and or and of which data from respondents. research to evaluate questionnaires was at the the 1983, Although by research than this in which respondents to as they and and describe a to which should even pretest reports are at and many other pretest reports may be in the of the American Survey and the American for Public Opinion at is and the reports often contain information about the