Research Using “Big Data” in Orthopaedic Trauma

This is the authors' abstract. We don't add key points for this paper.

When I consider the sheer volume of orthopaedic research (in the field of orthopaedic trauma and elsewhere) currently produced through the use of large data sets like the Nationwide Inpatient Sample (NIS), the National Trauma Databank (NTDB), and the National Surgical Quality Improvement Program (NSQIP), I am reminded of the experience of Mixter and Barr in the first half of the last century.1 Two physicians working at Massachusetts General, Mixter and Barr published the work that popularized lumbar discectomy surgery and were credited with ushering in what was called the Dynasty of the Disc.1 Anecdotally, legend has it that both surgeons eventually came to regret their publication around lumbar disc surgery. I often wonder if the same may be said in the future of the first scholars who introduced the use of “Big Data” to the orthopaedic field as well as other surgical disciplines. Clearly, the use of large scale data sets, national registries, and claims-based data have a number of advantages. Foremost, the volume of encounters associated with “Big Data” means that clinical samples provide a great deal of statistical power and enable robust methodologic approaches that would not be possible with small cohorts derived from single centers.2 The most important feature in this regard, particularly from a clinical perspective, is the capacity to account for patient, medical, or hospital level factors that might otherwise confound study results.2,3 In addition, the national scope of some data sets and registries render findings potentially more translatable to the average patient that may be treated in any particular practice, anywhere across the United States.2–7 Many of these databases are also readily accessible to researchers in ways that such patient-centered information was not in decades past.2 Thus, the sheer egalitarian nature of “Big Data” combined with the ability to identify even small statistically significant differences between study groups can serve as irresistible temptations to authors considering these study sets as source material for clinical, health services, and policy-based work. Indeed, in recent years the amount of orthopaedic research produced from studies conducted using NIS,6 NTDB,4 and NSQIP3,5,7 data has increased at an exponential rate. A recent systematic review (unpublished) of the orthopaedic literature derived from the NSQIP alone identified more than 170 articles produced in the last 5 years. In many ways, each of these contributions could be seen as a valuable addition to the Dynasty of Big Data in orthopaedic surgery. But I am not nearly so sanguine. As with most things in science and medicine, the numerous advantages associated with “Big Data” also come at a cost, and there are many potential pitfalls that researchers and the interested reader should be aware of. One must keep in mind that statistical findings are not always tantamount to clinically valid or translatable results, especially when the research has an exploratory (as opposed to a hypothesis-driven) bent along the lines of “Risk factors for outcome X in orthopaedic procedure Y in the Z data set.” It should also be appreciated that although used extensively in research associated with countless medical and surgical disciplines, the popular NIS, NTDB, and NSQIP data sets, not to mention health insurance claims or Medicare data are not primarily intended to support clinical research.2,3,8,9 By and large, new versions of these data sets come out once a year and are not necessarily compatible with, or representative of, previous versions.3 For example, the sampling strategy and clinical variables associated with NIS data may change between versions, particularly within the products produced since 2012.2 In a similar vein, newer versions of the NSQIP contain additional variables (eg, hospital readmission metrics) and also report on patients treated at participating hospitals outside the United States.2,3,8,9 New data are not being collected or published at even close to the same rate that orthopaedic research is being produced and, viewed in this light, such data sets may seem closer to a relatively finite resource that must be respected, and used judiciously and appropriately with well considered methodological techniques. At the editorial level, some of the biggest issues encountered among studies using “Big Data” include the absence of hypothesis-driven work, inconsiderate approaches to missing data, and frequent misunderstanding of the inclusion criteria associated with the data sets used. In an effort to properly approach the resource of “Big Data,” particularly when it comes to research in the field of orthopaedic trauma, suggested guideposts are proposed in the appended Table 1.TABLE 1: Limitations Associated With the Use of Popular Orthopaedic Databases With Some Suggested Approaches to Minimize ProblemsLack of hypothesis-driven research has already been addressed above to some extent but, especially when dealing with a finite number of patients with a particular outcome (say mortality), statistical models can quickly become overfit if too many predictor variables are tested. Even in a large data set of thousands, if only a few patients have the outcome of interest, characteristics unique to those individuals could be seen as statistically significant predictors, but such findings might not be generalizable to the experience of other patients in different clinical settings. The way to avoid this is to define the variables that would be tested, or adjusted for, before commencement of the study using conceptual models or empiric understanding of the factors that come to influence the metrics under study. Researchers should also consider the number of patients with the outcome of interest first and then support their models using the most important covariates instead of data dredging that considers every possible variable, some with no clinical relevance to the topic at hand. Missing data are a concern in every kind of research from prospective randomized studies to level IV retrospective reviews at single centers.10 It is not unique to research conducted using “Big Data” but can easily confound study findings, especially if patients with missing data are somehow different from the population that has accurate information imparted for all variables under consideration.10 For example, in a study evaluating outcomes among patients treated with hemiarthroplasty versus total hip arthroplasty for hip fracture, if important confounders such as insurance status or medical comorbidities are not available for all the patients treated with total hips, these factors cannot be controlled for and study estimates may be inaccurate as a result. Concerns with data integrity abound, and there are entire monographs on the appropriate means to assess and redress missing data.10 The most effective means of addressing missing data include multiple imputation (where missing data for patients are estimated based on other characteristics)10 or sensitivity analyses (where the analyses are rerun with and without patients with missing data). Statistically unsupported approaches should not be used including dropping all patients with missing data or treating missing variables as a null value (eg, patients with missing data regarding their smoking status are assumed to be nonsmokers). Last, the design of the data set used should be taken into account when considering the optimal database to test the hypotheses already developed.2,3,8,9 The subset of data selected by the authors should parallel the population whose care the study is intended to inform. For example, an investigation looking to examine factors relevant to geriatric hip fractures might elect to limit inclusion to those individuals 50 and older found to have isolated hip injuries. Those with polytrauma, injuries from gunshots or high-velocity trauma, and patients in their 20s and 30s would probably not generate useful data that could direct care for individuals with fragility fractures involving the hip. NSQIP actively excludes most typical trauma cases from their data collection processes,8 although a parallel product, the Trauma Quality Improvement Program does not.2 NIS bases its national representation on weighted samples and, therefore, rare types of traumatic injury (such as open pelvic fractures or atlanto-occipital dissociation) may still not be nationally representative even after appropriate use of the data set's weighting protocol.2 The same may be said for the NTDB, as individuals treated at nontrauma centers for orthopaedic injuries likely do not make their way into the data set unless they are transferred early in the care process.2 Regardless of the camp that a particular investigator may be in, whether available data sets are viewed as precious commodities or an unlimited cornucopia of potential articles (the truth is probably somewhere in between), I think we can all agree that the most useful clinical research is methodologically sound, patient-centered, and oriented toward changing practice or policy. If such goals are to be achieved using “Big Data,” the approaches outlined above should be considered when fashioning studies and crafting an analytic approach. To do otherwise in the service of the proverbial data dynasty certainly might lead to the exhaustion of these critical resources for research. ACKNOWLEDGMENTS The opinions expressed in this work are the author's own and do not necessarily reflect those of the Department of Defense or the United States Government.