Approaches for Improving Validity in Quantitative Research Articles

Approaches for Improving Validity in Quantitative Research Articles Andrew D. Asher (bio) Introduction Validity, or the assessed likelihood that a research design represents and measures the concepts it purports to study, is an essential component of evaluating the quality and efficacy of research manuscripts prior to publication. Since there is no single method to determine validity, readers must make a comprehensive judgment based on both the conceptual and technical elements of a research design, making it imperative that authors provide sufficient contextualization and discussion of data collection and analysis decisions that might affect the interpretation of their findings and conclusions. Especially when using well-established statistical procedures, manuscripts presenting quantitative analysis often gloss over information that is fundamental to understanding the underlying validity of the research. A three-step evaluation heuristic that considers the sample, the statistic and significance, and the effect size is a useful tool to help correct for this problem and to ensure that key elements are present, appropriate for the research design, and contain sufficient information to assess validity (See Figure 1). [End Page 209] Click for larger view View full resolution Figure 1. A "4S" evaluation workflow for quantitative analysis: Sample, Statistic & Significance, and (Effect) Size. Using this heuristic as a starting point, this editorial discusses common problems found in quantitatively focused manuscripts with the goal of providing guidance to prospective portal authors prior to review. Many of these errors and omissions stem from the types of quantitative research designs that are frequently employed in LIS research, such as methods involving surveys or rubrics, but the principles discussed here are broadly applicable to other research approaches as well. Sampling Design: A Study's Foundation Draft manuscripts often include insufficiently described sampling methods. The quality of sampling procedures and how well a sample represents the population under study provide the foundation for the validity of all subsequent inferences based on the data collected. Clear and detailed discussion of the sampling approach and its limitations is [End Page 210] therefore necessary as a first step to enable reviewers and readers to evaluate the quality of the research presented. Probability-based sample designs are usually required to ensure that analyses involving inferential statistics are valid, and so that concepts such as uncertainty, confidence, and risk can be quantitatively represented. Probability-based sampling methods range from simple to complex, but in all cases the chance of a member of the target population being included in the sample is known, can be expressed as a probability, and is a result of a randomization procedure.1 However, while probability-based design is almost always a best practice, practical constraints such as cost, time, or data availability often make these designs difficult to implement, resulting in researchers relying on non-random or inadequately randomized samples. Survey-based methods can be particularly vulnerable to this issue, and a frequent problem in LIS studies is the presentation of survey results as if they are based on probability samples when they are not. Surveys sent to listservs are often an example of this error. LIS surveys routinely use listservs as a proxy for the population of library professionals in place of randomized sample designs. This practice results in several potential problems. First, without detailed information about the membership of a listserv and associated demographic data, it is not possible to accurately estimate how well it represents the population under study. This produces a high potential for selection bias—members of the listserv may differ in some systematic way from the population at large—as well as non-response bias—people who answer surveys on listservs may differ from people who do not.2 Second, since the overall membership of a listserv is often unknown to the researchers, it is impossible to calculate common measures of survey quality such as response rates and margin of error, which are essential to evaluating both validity and confidence in the survey outcomes. At best, responses from a survey sent to a listserv can only represent the population of the listserv itself—without additional information we cannot interpolate how well any group of respondents potentially represents a wider population. Utilizing a listserv for gathering survey responses is an example...