Editorial Comment: Learning From Large-Scale Orthopaedic Databases

Advances in research and technology offer new possibilities and challenge us to think in innovative ways. Where social media, smart phones, and tablet computers are concerned, such changes in perspective and behavior have been embraced enthusiastically. Clinging to old ways of doing things, either by rejecting the new tools or by failing to use them appropriately, can be costly. At the same time, placing unrealistically optimistic expectations on novel technologies can be a mistake.FigureFigureThese issues are particularly relevant when examining the evolution and subsequent use of large national datasets during the last 15 years. The collection of information for such datasets has been driven by billing needs as well as efforts aimed at improving quality of care. Increasingly accessible electronic records systems and pooled data have facilitated these efforts, as have improved analytic techniques and software. Orthopaedic surgeons have embraced these new sources of data, and the number of publications employing them is on the rise [5]. Currently, there are a number of large national datasets that track patients with orthopaedic conditions. Some extract patient information from administrative billing codes such as the Nationwide Readmissions Database, while others use prospectively-collected or chart-abstracted data like the National Trauma Data Bank. Additionally, several datasets only track patients while they are in the hospital (like the National Inpatient Sample), while others offer varying lengths of followup (for example, the National Surgical Quality Improvement Program tracks patients for 30-days postoperatively). Large databases offer many opportunities beyond those provided by traditional single-institution or even regional collaborative studies for orthopaedic research. The sizable patient samples, often tens of thousands of patients or more, facilitate the study of large, representative groups of patients to help us arrive at generalizable results and allow for the study of rare variables or outcomes that smaller samples would not be able to assess. The large volume of information available in these databases also facilitates the creation of tools to evaluate a patient's risk of complications [2], or the testing of their utility for a specific orthopaedic population [1]. Together, these features allow researchers to assess many different populations and to strive for clinically meaningful conclusions, with the ultimate goal of enhancing patient care. Utilizing such large national datasets necessitates statistical methodologies and considerations that had previously not been familiar to most orthopaedic surgeons. The sheer quantity of data available greatly increases the statistical power of studies, yet requires clinical discretion for appropriate interpretation [9]. As the sample sizes increase, more statistically substantial associations can be identified, even though such correlations may not be clinically important. As the number of datasets available grows, it is important to note that they often incorporate differing subsets of the population. There are concerns that the same analysis conducted in multiple datasets can result in differing conclusions and clinical implications [4]. Even within a dataset, there may be mixed indications, patient characteristics, and procedural differences. Controlling and accounting for potentially confounding variables with multivariate analysis, propensity matching, or other analytic tools becomes crucial. Finally, large data is constantly evolving over time. Definitions are modified and new variables are added while outdated variables are eliminated. ICD-9 coding has been replaced with ICD-10 coding. Chart-abstracted data require precise definitions that get revisited over time. The NSQIP occasionally changes fields from an annual update when it determines the field is not reliable, but the fields are still in prior versions of the database. Disease definitions based on ICD-9 coding may see a jump across years as reporting criteria change. Established research models must continue to adapt with such changes or else they will become obsolete [8]. Clearly, consideration needs to be given to the interpretation of national database research. Two relevant quotes from the mathematician John Tukey come to mind when considering database research. The first: “The combination of some data and an aching desire for an answer does not ensure that a reasonable answer can be extracted from a given body of data” [6], applies even if the data contain millions of patients. The second quote: “Far better an approximate answer to the right question, which is often vague, than an exact answer to the wrong question, which can always be made precise” [7] recognizes that just because an answer can be calculated to a high number of decimal places or based on a large number of subjects does not mean that it is a useful insight. Overall, there is great potential value in data from large national datasets. Entwined in such power, however, are limitations that must be considered. A recent Clinical Orthopaedics and Related Research® editorial on large-database studies tried to define areas of greatest potential impact for such studies [3]. The authors of the papers in this month's symposium use a variety of large datasets to answer clinically relevant questions; we hope the answers and points contained in these studies can help surgeons, healthcare systems, and policymakers make better choices. However, readers of these studies should be aware, as in any study, of the potential limitations in both data quality and analysis. Database studies are powerful, but not necessarily definitive.