Missing data in primary care research: importance, implications and approaches

Missing data are a common occurrence in all study designs used to conduct primary care research (pragmatic randomized trials, observational studies, quality improvement efforts, etc.). We define missing data as values not available to the investigator that would have contributed to the final analysis had they been observed. Examples of missing data include patients lost to follow-up, partially filled-out surveys or incomplete medical records. Missing data can compromise the validity of study findings (e.g. risk for bias increases, subgroups of the population may be underrepresented, loss of information, reduced statistical power, etc.) (1–5). Multiple studies have shown that high rates of missing data may negatively impact conclusions in primary care studies (6–8). A holistic approach for addressing missing data should include approaches in the design phase (prior to the study), conduct phase (during the study) and in the analytic phase (after data collection is complete). This article presents key concepts regarding types of missing data, and discusses good practices to observe when conducting primary care research with missing data. Whatever approach is used to address missing data, all require making assumptions about the mechanism that resulted in missing data. Determining these mechanisms can be imprecise and impossible to verify; thus, taking steps to prevent missing data in the first place is just as important as missing data analytic approaches. There are numerous and unique randomized trial designs in primary care; these designs use administrative, electronic health record (EHR) and patient-reported data. In the design of randomized trials, there is extensive literature that provides guidance on how to avoid missing data (3,9). Generally, these include: (1) identifying target populations underserved by interventions that may have incentives to remain in the study; (2) under appropriate research questions, utilizing patient-centered (10) and adaptive interventions (11) to meet the specific needs of different clinics, their patients and avoid patient/clinic dropout by motivating patients/physicians to stay engaged throughout the study; (3) utilizing designs that add interventions to existing accepted interventions; (4) selecting primary outcomes that are less susceptible to extensive missing data (e.g. easily defined, standardized quality indicators or clinical outcomes, biomarkers that have widely available standardized protocols for collection, and brief, validated, language and literacy-appropriate survey instruments). For observational studies that utilize administrative and clinical data such as EHRs, a common data source in primary care research, several considerations can be utilized to reduce missingness including: (1) focusing on variables that are consistently collected as part of the routine clinical practice, (2) designing clear data collection procedures focused on only meaningful data to minimize burden on those extracting data, (3) identifying and linking several data sources for better capture of important variables (12,13) (e.g. race/ethnicity fields in administrative and EHR data can be linked and used to fill in missing values (14)). The STROBE guidelines provide great guidance on what to consider as one designs observational studies (15) and provide suggestions for how to document missing data. Avoiding missing data in survey research has been extensively detailed (16,17) and we refer the reader to those citations. In short, there is an emphasis on reducing survey burden (e.g. limiting survey length) and pilot testing surveys before large-scale implementation to ensure capture of elements that address the main research question, as well as information that might be useful in approaches to address missing data. For study designs with prospective data collection, missing data can also occur during the study (prior to analysis). Table 1 provides ideas based on published research (9,18,19) on how to minimize missing data during this phase. Suggestions for reducing missing data in the conduct phase of a study Suggestions for reducing missing data in the conduct phase of a study Even with much care during the study, missing data will be unavoidable. It is good practice to plan one’s analytic methods to address missing data in the design phase. There is no single best method to address missing data that is appropriate for all circumstances as study designs and reasons for missing data may vary. Prior to selecting a method, it is important to identify the potential mechanism leading to missing data. Missing data can be classified into three types: missing completely at random (MCAR), missing at random (MAR) and missing not at random (MNAR). When data are MCAR, the probability of being missing is equal for all observations. If data are MAR, this suggests that the probability of being missing is equal only within groups defined by observed data. When data are MNAR, the probability of being missing varies for unobservable reasons (20). To illustrate these concepts, consider a scenario where a researcher is interested in studying if there are disparities in colorectal cancer screening by race/ethnicity in their health system. If a sample of patient faecal occult blood tests (FOBT) mail-in kits were lost in the mail due to a random glitch, we would say that the data are MCAR. Missing data scenarios that have nothing to do with the patient being studied (e.g. weight scale breaks down, lost mail, etc.) are typically considered MCAR. Most simple solutions for missing data (e.g. complete case analysis, missing-indicator method, single-value imputation, sensitivity analyses with best/worst case scenarios) only provide unbiased results under the MCAR assumption, though estimates may be less precise (21–23). Data are MAR if the missingness does depend on the patient but the missing component can be predicted from other information about the patient. For example, if a patient does not mail in the FOBT kit because the instructions are in English and they speak Spanish and you know the patient’s preferred language, and the reason that the patient did not send in the kit was not related to the outcome of screening, this missing data are MAR. In MAR settings, one can leverage other available data to address missing data; most statistical methods that address missing data assume MAR. Scenarios that do not meet MCAR or MAR fall into MNAR. When data are MNAR, it is because the missing data are specifically related to what is missing. For example, if a patient does not mail in the FOBT kit because they were uncomfortable with specimen collection, we would say that these missing data are MNAR; anytime the missing data are related to what is missing is an indication of MNAR. This setting is the most problematic and is the most complex situation to handle. Within a study, data may be both MAR and/or MNAR, and rarely is it MCAR. There are four common types of methods that address missing data (9): (1) complete case analyses, (2) single imputation, (3) inverse probability weighting and (4) multiple imputation. Complete case analysis (i.e. listwise deletion) is a method that removes all patients with any missing data from the analysis. This is typically the default approach to handle missing data in many statistical software programs and while it is convenient to implement, it relies on the often-unrealistic assumption that the data are MCAR which can result in biased estimates and reduced statistical power (3). Single imputation methods are methods that fill in missing data with a single value. Examples include last observation carried forward (e.g. in a longitudinal study, fill in missing values with the last value you observed), mean imputation (e.g. if missing body mass index [BMI], estimate the mean from the patients with observed BMI and use that mean to fill in the missing values), among others. Inverse probability weighting is a method that utilizes only complete cases (i.e. patients who had fully observed data). These complete cases are weighted by the inverse of their probability of being a complete case (24–26). Intuitively, complete cases may be different than those with missing data and thus may not be representative of the broader population. Thus, one can assign larger weights to patients who are underrepresented in the sample and lower weights to those that are overrepresented with the goal to make the sample look more like the population. Multiple imputation (27), applicable to cross-sectional and longitudinal designs (28), is an extension of single imputation that fills in missing data, typically using standard regression methods, with multiple plausible values that account for the uncertainty of the imputed values (29). Figure 1 shows the main three stages involved in multiple imputation: (1) generate m imputed data sets according to the a priori analytic study plan, (2) analyse the m imputed data sets, (3) pool the results from the m analyses using multiple imputation combining rules to yield a single estimate (30). Starting with observed, incomplete data, multiple imputation creates several versions of the data by replacing missing values with plausible ones. The parameters of interest are estimated for each of the imputed data sets. Then the multiple parameter estimates are pooled into one estimate. While there are no universal rules for how many imputations to perform, it is recommended that the number of imputations should be similar to the percentages of incomplete cases (31,32). Stages of multiple imputation with an illustrative example where the mean haemoglobin A1c (HbA1c) among patients is the parameter of interest and m = 3 imputations were performed. Note: Multiple imputation begins with an incomplete data set. Multiple imputed data sets are created by replacing missing values with plausible values based on a random draw from a distribution that is specifically modelled for the missing values, typically using regression models. Analysis is done on each imputed data set to estimate a parameter of interest. These estimates are pooled into one single estimate of the parameter and its variance. Stages of multiple imputation with an illustrative example where the mean haemoglobin A1c (HbA1c) among patients is the parameter of interest and m = 3 imputations were performed. Note: Multiple imputation begins with an incomplete data set. Multiple imputed data sets are created by replacing missing values with plausible values based on a random draw from a distribution that is specifically modelled for the missing values, typically using regression models. Analysis is done on each imputed data set to estimate a parameter of interest. These estimates are pooled into one single estimate of the parameter and its variance. In most cases, single imputation methods are considered an improvement to complete case analyses but they do not reflect the uncertainty in the imputations and thus can lead to standard errors that are too small and have the potential for incorrect conclusions (33). Inverse probability weighting and multiple imputation approaches are generally recommended as they often assume a MAR missing data mechanism and thus can use supplementary information about the missing data in the final analysis. In practice, a MAR assumption is more likely to be valid as compared to a MCAR assumption, especially when all relevant important variables are collected. There are no universal guidelines for the amount of missing data that make statistical inference is valid. Several characteristics play a role including the amount of missingness (e.g. percentage of data missing), the correlation between cause of missingness and variable containing missingness and the correlation between cause of missingness and missingness itself (3). In general, the lower percentage of missing data, the better. Missing percentages of ≤5% are thought to be trivial (27). One study found that an analysis is likely to be biased if 10% or more of the data are missing (34). Another suggests that if more than 40% of data on important variables are missing, the results should be considered hypothesis-generating rather than confirmatory (35). Lastly, one simulation study showed that the proportion of missing data should not be used as a guide to whether to use a method like multiple imputation or not. Instead, they encouraged the use of methods such as multiple imputation to reduce bias and improve efficiency at any proportion of missing data (36). In survey research, acceptable response rates vary. One ‘rule of thumb’ is 60% (37). Some agencies may ask for the use of survey methods that result in a 75% response rate yet response rates lower than 30% are seen in published studies (38). Response bias—the idea that respondents are different than non-respondents and not representative of the whole survey population—is more likely in surveys with lower response rates (38). Even in settings with minimal missingness, it is generally recommended that an analytic method that addresses missing data should be employed. Though outside the scope of this primer, sensitivity analyses that assume MNAR should be performed to check the robustness of study findings, such as pattern-mixture models (39) and selection models (40,41). Primary care studies are multifaceted and will encounter missing data at the patient, clinician or clinic level that could negatively impact conclusions. Investigators, in collaboration with statisticians, should include missing data approaches or their mitigation in the design phase (prior to the study), conduct phase (during the study) and in the analytic phase (after the study has finished data collection). Understanding the missing data mechanism (MCAR/MAR/MNAR) is important to guide the researcher in understanding study limitations, and identifying the appropriate analytic approach to employ. Several methods are available to address missing data and one or more should always be used. By ensuring that missing data are properly addressed, primary care research studies will continue to increase rigour and inform evidence-based practice. Funding: This work was supported by grant numbers R01MD011404 and R01MD014120 from the US National Institute for Minority Health and Health Disparities and by grant number R01AG056337 from the US National Institute for Aging. Ethical approval: none. Conflict of interest: none.