Methods for developing and analyzing clinically rich data for patient‐centered outcomes research: an overview

The Agency for Healthcare Research and Quality, through its Effective Health Care program, sponsored a third invitational symposium entitled “Research Methods for Comparative Effectiveness (CER) and Patient-Centered Outcomes Research (PCOR)” on 6 and 7 June 2011 in Rockville, Maryland. The conference focused on original research that highlights methods development or provides methodological insights that arise from the conduct of the research. This focus represents emerging, innovative approaches to CER/PCOR and reflects the challenges of practice-based research that are directly applicable to CER as well as other types of research. This supplement to Pharmacoepidemiology and Drug Safety represents some of the presentations from the symposium, which were further developed after the symposium and independently peer reviewed. Although all methods have limitations, the supplement is intended to be a resource for researchers interested in learning, applying, and improving different methodological approaches to the design and analysis of CER/PCOR studies. Decision makers need valid information on the comparative effectiveness of medical products, interventions, diagnostics, and delivery systems in routine care. Selecting the most appropriate data sources, research design, and analytic strategy to address a CER question that will inform specific healthcare decisions requires the alignment of numerous perspectives. Often a gap exists between the capabilities of a data source (even with the application of sound research and analytic methods) and the CER question for which an answer is sought. The symposium examined new and emerging methods for CER and PCOR that either use clinically rich data sources or enrich existing data sources with supplemental data, thus helping to fill this gap between available data and data needs specific to studying a question. Secondary data may often be the most representative information source for describing routine care, and there are a growing number of electronic databases available for supporting CER/PCOR studies. Administrative data sources capture healthcare encounters longitudinally and serve as a comprehensive data backbone reflecting routine care. However, they often lack clinical detail or patient-centered outcomes (such as laboratory test results, left ventricular ejection fraction, or a depression scale, pain scale, assessment of functional status, etc.), which may limit their utility for research intended for decision makers. The aim of the symposium was to explore methodological options for enhancing secondary data sources or prospectively designed registry studies and randomized trials that reflect routine care as much as possible, increasing their utility for generating evidence that can inform CER questions and influence healthcare decisions. In the opening session, Dr. Louis B. Jaques, the Director of Coverage and Analysis Group at the U.S. Centers for Medicare and Medicaid Services, reminded us about the importance of incorporating the stakeholder perspective in planning CER/PCOR studies. Stakeholders can assist researchers by articulating what evidence is needed to inform their decisions (design considerations, clinical measures, etc.). The more CER can address this evidentiary need, the more useful it will be because it permits the decision maker to act with increased certainty. When to use a randomized controlled trial to address a CER question is among the many decisions a stakeholder has to make. A clear understanding of factors that influence this decision will make it easier for stakeholders to make informed decisions. Also, providing clarity on which patient-centered outcome measure to pick among the many available and how to reach consensus on the most appropriate measures are needed for decision making. The article by John Concato provides a perspective that randomized trials offer strong internal validity for comparisons and offer many things to the CER consumer by addressing numerous competing hypotheses.1 However, the advantages of clinical trials come with limitations, some obvious and some subtle. From an overarching perspective, randomized studies are relevant to activities and decision making in scientific as well as policy settings, while also encompassing a wide range of intersecting and overlapping topics. Representative situations include the following: How does an individual investigator decide that a randomized trial is warranted to study a particular research question? When should funding agencies commission randomized or non-randomized research? Do regulators always need evidence from randomized trials to approve new drugs? Specifically, an argument is presented, and representative examples are cited, which focuses on comparative effectiveness research (CER) and promotes “medicine-based evidence.” This framework identified the dimensions to be considered: ethical, practical, logistical, financial, and the ranges of each dimension that favors one over the other. Researchers struggle with the limitations of administrative healthcare databases and have begun linking them with supplemental data from a wide range of sources to improve assessment of study exposures, subgroup identifiers, outcomes, and confounding factors. Toh et al.2 provided a useful demonstration of a variety of approaches for dealing with missing data in electronic medical record (EMR) database studies. The example begins to shed light on the underappreciated issue of missing data in EMR databases. As EMR databases become more commonly used in CER, understanding the application and performance of the methods to deal with missing confounder information described in this article will be increasingly important. The existing methods for analyzing supplemental confounding data require different assumptions and may provide disparate results, so these results may serve to guide future researchers. Accurately identifying a patents' health state at baseline or during follow-up is particularly important for studying the comparative effectiveness of cancer treatments. The article by Nordstrom et al.3 developed algorithms to identify metastatic cancer in claims data, using tumor stage information from an oncology EMR data warehouse. The contribution of this article to the rapidly growing field of linking insurance claims data with detailed EMRs is to provide insight into challenges and rewards. Most of the algorithms for metastatic cancer had reasonable specificity and acceptable positive predictive value, but there was a trade off with sensitivity that prevented any of the models from having good predictive ability on all measures. Accurate ascertainment of metastatic status may require going beyond purely administrative data sources and accessing medical records or other data sources. Similarly, Dore et al.4 described a study successfully linking health insurer claims data with detailed clinical oncology information on stage at diagnosis, current clinical status, histology, and other clinical markers. Enriched data such as these have the potential to extend the horizons of what questions can be answered with secondary data and to reduce the resource barriers to answering them, with important implications for CER. Studies using such data will need to appropriately account for the varied nature of the data collection mechanisms. The article by Aplenc et al.5 described the merging of the cooperative oncology group data with an administrative data source funded by the National Cancer Institute to develop a more effective platform for clinical trial analysis and CER. The article describes innovative data linkage between clinical trials and administrative data, creating an integrated data set that is likely to have numerous uses, some of which may not yet be envisioned. Their primary focus on identifying unintended treatment effects is intriguing and may offer efficiencies for clinical trials in the future. Studying the effectiveness of in-hospital interventions challenges the utility of the many data routinely collected by hospitals. Lagu et al.6 showed how incorporating initial treatment choices improves the performance of a mortality using in-hospital administrative data. The results provide researchers with an option for addressing concerns about confounding that is relatively easy to implement and has been shown to provide an incremental benefit in terms of mortality prediction in patients with sepsis. Until administrative data sets are available that include clinical information (such as vital signs and laboratory results), such proxies as initial treatment choice may allow researchers to conduct non-randomized effectiveness studies in sepsis patients. Paxton et al.7 described the functioning of a total joint replacement registry. They describe the data infrastructure build from electronic health records and administrative data throughout a large integrated healthcare system that was supplemented with primary data collection. The article emphasized some of the challenges associated with orthopedic device classification and evaluation, including the handling of missing data. The registry permits evaluations of device choice and outcomes associated with total hip arthroplasty that depend on refined exposure, confounder, and outcome measures only available through such a linkage of data sources. Confounding adjustment at baseline or during follow-up is a challenge in non-randomized studies of intended treatment effects, whereas confounding adjustment during follow-up challenges both observational and randomized studies alike. Several innovative approaches or applications to CER questions are presented in the subsequent articles. Prior event rate ratio (PERR) adjustment has been introduced several years ago as a type of self-controlled design with the hope to more fully adjust for potential confounder, measured or unmeasured. Yu et al.8 presented a simulation study together with theoretical derivations on the performance of PERR adjustment in comparative studies of intended treatment effects. It serves to clarify some misconceptions about the proper application of the method and positions the method correctly so that future users will be able to use it in a more targeted manner. The simulations show that PERR-based methods can reduce bias from unmeasured confounders in situations when the exposure effect is relatively large in comparison with confounder–exposure interaction or when time interval effect is rather modest. As propensity score (PS) matching is increasingly used, matching techniques need to be understood more fully. Rassen et al.9 evaluated several methods of PS matching in cohort studies through simulation and empirical analyses and concludes that the 1:n variable ratio matching can be used to increase precision in cohort studies over 1:1 matching at a small cost in bias. The article by Hammill et al.10 used a simulation study to examine the performance of PS-based methods for estimating relative risks when exposed, and comparison subjects are selected from different data sources. In CER, researchers are likely to encounter situations similar to the one described by this simulation, and the results of the simulation will largely be seen as encouraging and are consistent with theory. Under various scenarios evaluated in the simulation, the PS methods often yielded estimates of relative risk that were close to the true effect. The delineation of situations that diverged from true effects further strengthens this analysis. Polinski et al.11 explored the specific challenges in PS adjustment in evaluating the effectiveness of healthcare intervention using the effect of the Medicare Part D doughnut hole on health outcomes as an example. They compare pre-defined PS and high-dimensional PS approaches for their ability to address how patient characteristics influence plan choice and whether coverage gap entry is associated with cardiovascular events or death. The high-dimensional PS approach confirmed the hazards associated with coverage gap entry among exposed patients and provides a solution that produces more clinically plausible results. Although these approaches try to deal with population imbalances at study baseline, Neugebauer et al.12 involved the application of inverse probability–weighted estimation to fit dynamic marginal structural models in observational studies to address pragmatic CER questions and to properly adjust for time-dependent confounding including informative loss to follow-up. Focusing on data on treatment intensification in patients with diabetes provides this article with greater reach than a purely theoretical or simulated study would have. The results are concordant with randomized clinical trials (RCTs), which provide some external calibration and make this approach a viable alternative to inadequate standard modeling approaches in CER problems in which time-dependent confounding is expected. Cai et al.13 presented a simulation study of instrumental variable (IV) analyses using the antidiabetic effect of bezafibrate as an example. They describe a method of IV implementation that may have considerable utility for binary outcomes that are often of primary interest in CER. The geometric structural mean model (GSMM) IV approach outlined in the article provides approximately unbiased adjustment for unmeasured confounding on binary outcomes when a valid IV is available. Weiss et al.14 demonstrated methods to assess the applicability of trial evidence to a target group in the presence of heterogeneity of treatment effects. The article deals with the important issue of extrapolating trial findings to patient groups in the presence of treatment effect heterogeneity and provides a new approach using suitable data sources that will be helpful in structuring researchers' thinking around this issue. It may become an important stepping-stone for a new line of investigation that is very relevant for decision making in health care. The application involves a drug therapy to prevent mortality in patients with heart failure. The analysis uses individual-level data from a trial and a registry. Differences between the studies are explored using a principal components analysis with biplots. The analysis graphically illustrates heterogeneity while also providing quantitative assessment of the applicability of evidence from one population to another. CER evidence is often most critical shortly after a new therapy or medical product is introduced. However, in this early marketing phase, channeling of patients to the new therapy may be particularly strong and information on users is still sparse, a combination that is particularly challenging.15 The article by Signorovitch et al.16 on CER using matching-adjusted indirect comparison applied weighting/matching strategies to indirect comparisons of RCTs. Indirect comparisons of RCTs are often the earliest evidence on the comparative effects of two active therapies when results of more thorough comparative effectiveness investigations are not yet available. Therefore, any way to improve these methods is welcome and important. The finding that matching-adjusted indirect comparison can adjust for baseline difference between trial populations enables timely and reliable comparison for treatments. In the absence of head-to-head trials, a matching-adjusted indirect comparison of treatment outcomes across separate randomized trials can provide valuable comparative evidence. Confounding adjustment can be challenging if only few patients are exposed to a new therapy. Robert Glynn et al.17 explored the role of disease risk scores for CER with emerging therapies. The article considers settings where a disease risk score may have theoretical advantages as a balancing score over PS and traditional multiple regression models in CER in settings of evolving therapies. Balancing on a disease risk score provides an attractive alternative to a PS in settings such as newly marketed drugs and provides an important axis for evaluation of potential effect modification. The article provides an overview of settings where the different methods offer advantages and the potential for joint consideration of propensity and disease risk scores. Decision makers request information on causal relationships between therapies and health outcomes that are of high validity and also generalizable to routine care. Can randomized trials ever mimic routine care or are findings from RCTs never generalizable to routine care? Large streamlined trials or cluster randomized trials have been proposed as possible solutions. Greevy et al.18 developed an innovative method and software to aid in the design of cluster randomized trials. The reweighted Mahalanobis distance (RMD) incorporates user-specified weights and imputed values for missing data. Matching using RMD achieved better balance than simple randomization. RMD matching provides an easy-to-use tool that incorporates user knowledge and missing data. Most of the reports in this supplement advance our understanding on how to combine information from various data sources and use the best aspects of each to better inform decision makers. As much as these methodological developments and considerations will be helpful, they also show that there is still a wide range of methodological gaps that need to be filled to improve the results from CER studies. This project was funded under Contract No. 290-05-0016I from the Agency for Healthcare Research and Quality, US Department of Health and Human Services as part of the Developing Evidence to Inform Decisions about Effectiveness (DEcIDE) program. The authors of this report are responsible for its content. Statements in the report should not be construed as endorsement by the Agency for Healthcare Research and Quality or the US Department of Health and Human Services.