Building electronic data infrastructure for comparative effectiveness research: accomplishments, lessons learned and future steps

Comparison of EHR Data-Completeness in Patients with Different Types of Medical Insurance Coverage in the United States

Prior studies have demonstrated that misclassification of study variables due to electronic health record (EHR)-discontinuity can be mitigated by restricting EHR-based analyses to subjects with high predicted EHR-continuity based on a simple algorithm. In this study, we compared EHR continuity in populations covered by Medicare, Medicaid, or commercial insurance. Using claims-linked EHRs from a multicenter network in Massachusetts, including Medicare (MA EHR-Medicare cohort) and Medicaid (MA EHR-Medicaid cohort) claims data; and TriNetX (TriNetX cohort) claims-linked EHR data from 11 US-based healthcare organizations, we assessed (1) EHR-continuity quantified by proportion of encounters captured by EHR (capture proportion (CP)); (2) area under receiver operating curve (AUROC) of previously validated model to identify patients with high EHR-continuity (CP ≥ 0.6); (3) misclassification of 40 patient characteristics, quantified by average standardized absolute mean difference (ASAMD). Study participants were ≥ 65 years (Medicare) or ≥ 18 years (Medicaid, TriNetX) with ≥ 365 days of continuous insurance enrollment overlapping with an EHR encounter. We found that the mean CP was 0.30, 0.18, and 0.19 and AUROC of the prediction model to identify patients with high EHR-continuity was 0.92, 0.89, and 0.77 in the MA EHR-Medicare, MA EHR-Medicaid, and TriNetX cohorts, respectively. Restricting to patients with predicted EHR-continuity percentile of top 20%, 50%, and 50% in MA EHR-Medicare, MA EHR-Medicaid, and TriNetX cohorts resulted in acceptable levels of misclassification (ASAMD < 0.1). Using a prediction model to identify cohorts with high EHR-continuity can improve validity, but cutoffs to achieve this goal vary by population.

The Impact of Longitudinal Data-Completeness of Electronic Health Record Data on the Prediction Performance of Clinical Risk Scores

The impact of electronic health record (EHR) discontinuity (i.e., receiving care outside of a given EHR system) on EHR-based risk prediction is unknown. We aimed to assess the impact of EHR-continuity on the performance of clinical risk scores. The study cohort consisted of patients aged ≥ 65 years with ≥ 1 EHR encounter in the 2 networks in Massachusetts (MA; 2007/1/1-2017/12/31, internal training and validation dataset), and one network in North Carolina (NC; 2007/1/1-2016/12/31, external validation dataset) that were linked with Medicare claims data. Risk scores were calculated using EHR data alone vs. linked EHR-claims data (not subject to misclassification due to EHR-discontinuity): (i) combined comorbidity score (CCS), (ii) claim-based frailty score (CFI), (iii) CHAD2 DS2 -VASc, and (iv) Hypertension, Abnormal renal/liver function, Stroke, Bleeding, Labile, Elderly, and Drugs (HAS-BLED). We assessed the performance of CCS and CFI predicting death, CHAD2 DS2 -VASc predicting ischemic stroke, and HAS-BLED predicting bleeding by area under receiver operating characteristic curve (AUROC), stratified by quartiles of predicted EHR-continuity (Q1-4). There were 319,740 patients in the MA systems and 125,380 in the NC system. In the external validation dataset, AUROC for EHR-based CCS predicting 1-year risk of death was 0.583 in Q1 (lowest) EHR-continuity group, which increased to 0.739 in Q4 (highest) EHR-continuity group. The corresponding improvement in AUROC was 0.539 to 0.647 for CFI, 0.556 to 0.637 for CHAD2 DS2 -VASc, and 0.517 to 0.556 for HAS-BLED. The AUROC in Q4 EHR-continuity group based on EHR alone approximates that based on EHR-claims data. The prediction performance of four clinical risk scores was substantially worse in patients with lower vs. high EHR-continuity.

The SHOnet learning health system: Infrastructure for continuous learning in pediatric rehabilitation

Introduction

To describe the development and implementation of learning health system (LHS) infrastructure for a pediatric specialty care health system to support LHS research in pediatric rehabilitation settings.

Methods

An existing pediatric common data model (eg, PEDSnet) of standardized medical terminologies for research was expanded and leveraged for this stud, and applied to SHOnet, a clinical research data resource consisting of deidentified data extracted from the electronic health record (EHR) from the Shriners Hospitals for Children speacialty pediatric health care system. We mapped EHR data for laboratory, procedures, drugs, and conditions to standardized vocabularies including ICD-10, CPT, RxNorm, and LOINC to the common data model using an established extraction-transformation-loading process. Rigorous quality checks were conducted to ensure a high degree of data conformance, completeness, and plausibility. SHOnet data elements from all sources are de-identified and the server is managed by the SHC Information Systems Department. SHOnet data are refreshed monthly and data elements are continually expanded based on new research endeavors.

Interventions

Not applicable.

Results

The Shriners Health Outcomes Network (SHOnet) includes data for over 10 000 distinct observational data elements based on over two million patient encounters between 2011 and present.

Conclusion

The systematic process to develop SHOnet is replicable and flexible for other pediatric rehabilitation research settings interested in building out their LHS capabilities. Challenges and facilitators may arise for building such LHS infrastructure for rehabilitation in areas of (a) data capture, curation, query, and governance, (b) generating knowledge from data, and (c) dissemination and implementation of new institutional knowledge. Further research studies are needed to evaluate these data resources for scalable system-learning endeavors.SHOnet is an exemplar of an LHS for rehabilitation and specialty care settings. The success of an LHS is dependent on engagement of multiple stakeholders, shared governance, effective knowledge translation, and deep commitment to long-term strategies for engaging clinicians, administration, and families in leveraging knowledge to improve clinical outcomes.

Impact of longitudinal data-completeness of electronic health record data on risk score misclassification

BACKGROUND

Electric health record (EHR) discontinuity, that is, receiving care outside of a given EHR system, can lead to substantial information bias. We aimed to determine whether a previously described EHR-continuity prediction model can reduce the misclassification of 4 commonly used risk scores in pharmacoepidemiology.

METHODS

The study cohort consists of patients aged ≥ 65 years identified in 2 US EHR systems linked with Medicare claims data from 2007 to 2017. We calculated 4 risk scores, CHAD2DS2-VASc, HAS-BLED, combined comorbidity score (CCS), claims-based frailty index (CFI) based on information recorded in the 365 days before cohort entry, and assessed their misclassification by comparing score values based on EHR data alone versus the linked EHR-claims data. CHAD2DS2-VASc and HAS-BLED were assessed in atrial fibrillation (AF) patients, whereas CCS and CFI were assessed in the general population.

RESULTS

Our study cohort included 204 014 patients (26 537 with nonvalvular AF) in system 1 and 115 726 patients (15 529 with nonvalvular AF) in system 2. Comparing the low versus high predicted EHR continuity in system 1, the proportion of patients with misclassification of ≥2 categories improved from 55% to 16% for CHAD2DS2-VASc, from 55% to 12% for HAS-BLED, from 37% to 16% for CCS, and from 10% to 2% for CFI. A similar pattern was found in system 2.

CONCLUSIONS

Using a previously described prediction model to identify patients with high EHR continuity may significantly reduce misclassification for the commonly used risk scores in EHR-based comparative studies.

Methodological Issues in Analyzing Real-World Longitudinal Occupational Health Data: A Useful Guide to Approaching the Topic

Ever greater technological advances and democratization of digital tools such as computers and smartphones offer researchers new possibilities to collect large amounts of health data in order to conduct clinical research. Such data, called real-world data, appears to be a perfect complement to traditional randomized clinical trials and has become more important in health decisions. Due to its longitudinal nature, real-world data is subject to specific and well-known methodological issues, namely issues with the analysis of cluster-correlated data, missing data and longitudinal data itself. These concepts have been widely discussed in the literature and many methods and solutions have been proposed to cope with these issues. As examples, mixed and trajectory models have been developed to explore longitudinal data sets, imputation methods can resolve missing data issues, and multilevel models facilitate the treatment of cluster-correlated data. Nevertheless, the analysis of real-world longitudinal occupational health data remains difficult, especially when the methodological challenges overlap. The purpose of this article is to present various solutions developed in the literature to deal with cluster-correlated data, missing data and longitudinal data, sometimes overlapped, in an occupational health context. The novelty and usefulness of our approach is supported by a step-by-step search strategy and an example from the Wittyfit database, which is an epidemiological database of occupational health data. Therefore, we hope that this article will facilitate the work of researchers in the field and improve the accuracy of future studies.

Longitudinal Data Discontinuity in Electronic Health Records and Consequences for Medication Effectiveness Studies

Electronic health record (EHR) discontinuity (i.e., receiving care outside of the study EHR system), can lead to information bias in EHR-based real-world evidence (RWE) studies. An algorithm has been previously developed to identify patients with high EHR-continuity. We sought to assess whether applying this algorithm to patient selection for inclusion can reduce bias caused by data-discontinuity in four RWE examples. Among Medicare beneficiaries aged >=65 years from 2007 to 2014, we established 4 cohorts assessing drug effects on short-term or long-term outcomes, respectively. We linked claims data with two US EHR systems and calculated %bias of the multivariable-adjusted effect estimates based on only EHR vs. linked EHR-claims data because the linked data capture medical information recorded outside of the study EHR. Our study cohort included 77,288 patients in system 1 and 60,309 in system 2. We found the subcohort in the lowest quartile of EHR-continuity captured 72-81% of the short-term and only 21-31% of the long-term outcome events, leading to %bias of 6-99% for the short-term and 62-112% for the long-term outcome examples. This trend appeared to be more pronounced in the example using a nonuser comparison rather than an active comparison. We did not find significant treatment effect heterogeneity by EHR-continuity for most subgroups across empirical examples. In EHR-based RWE studies, investigators may consider excluding patients with low algorithm-predicted EHR-continuity as the EHR data capture relatively few of their actual outcomes, and treatment effect estimates in these patients may be unreliable.

External Validation of an Algorithm to Identify Patients with High Data-Completeness in Electronic Health Records for Comparative Effectiveness Research

Objective

Electronic health records (EHR) data-discontinuity, i.e. receiving care outside of a particular EHR system, may cause misclassification of study variables. We aimed to validate an algorithm to identify patients with high EHR data-continuity to reduce such bias.

Materials and Methods

We analyzed data from two EHR systems linked with Medicare claims data from 2007 through 2014, one in Massachusetts (MA, n=80,588) and the other in North Carolina (NC, n=33,207). We quantified EHR data-continuity by Mean Proportion of Encounters Captured (MPEC) by the EHR system when compared to complete recording in claims data. The prediction model for MPEC was developed in MA and validated in NC. Stratified by predicted EHR data-continuity, we quantified misclassification of 40 key variables by Mean Standardized Differences (MSD) between the proportions of these variables based on EHR alone vs the linked claims-EHR data.

Results

The mean MPEC was 27% in the MA and 26% in the NC system. The predicted and observed EHR data-continuity was highly correlated (Spearman correlation=0.78 and 0.73, respectively). The misclassification (MSD) of 40 variables in patients of the predicted EHR data-continuity cohort was significantly smaller (44%, 95% CI: 40–48%) than that in the remaining population.

Discussion

The comorbidity profiles were similar in patients with high vs low EHR data-continuity. Therefore, restricting an analysis to patients with high EHR data-continuity may reduce information bias while preserving the representativeness of the study cohort.

Conclusion

We have successfully validated an algorithm that can identify a high EHR data-continuity cohort representative of the source population.

Applying a Commercialization-Readiness Framework to Optimize Value for Achieving Sustainability of an Electronic Health Data Research Network and Its Data Capabilities: The SAFTINet Experience

CONTEXT

Sustaining electronic health data networks and maximizing return on federal investment in their development is essential for achieving national data insight goals for transforming health care. However, crossing the business model chasm from grant funding to self-sustaining viability is challenging.

CASE DESCRIPTION

This paper presents lessons learned in seeking the sustainability of the Scalable Architecture for Federated Translational Inquiries Network (SAFTINet), and electronic health data network involving over 50 primary care practices in three states. SAFTINet was developed with funding from the Agency for Healthcare Research and Quality to create a multi-state network for comparative effectiveness research (CER) involving safety-net patients.

METHODS

Three analyses were performed: (1) a product gap analysis of alternative data sources; (2) a Strengths-Weaknesses-Opportunities-Threat (SWOT) analysis of SAFTINet in the context of competing alternatives; and (3) a customer discovery process involving approximately 150 SAFTINet stakeholders to identify SAFTINet's sustaining value proposition for health services researchers, clinical data partners, and policy makers.

FINDINGS

The results of this business model analysis informed SAFTINet's sustainability strategy. The fundamental high-level product needs were similar between the three primary customer segments: credible data, efficient and easy to use, and relevance to their daily work or 'jobs to be done'. However, how these benefits needed to be minimally demonstrated varied by customer such that different supporting evidence was required.

MAJOR THEMES

The SAFTINet experience illustrates that commercialization-readiness and business model methods can be used to identify multi-sided value propositions for sustaining electronic health data networks and their data capabilities as drivers of health care transformation.

Identifying Patients With High Data Completeness to Improve Validity of Comparative Effectiveness Research in Electronic Health Records Data

Electronic health record (EHR)-discontinuity, i.e., having medical information recorded outside of the study EHR system, is associated with substantial information bias in EHR-based comparative effectiveness research (CER). We aimed to develop and validate a prediction model identifying patients with high EHR-continuity to reduce this bias. Based on 183,739 patients aged ≥65 in EHRs from two US provider networks linked with Medicare claims data from 2007-2014, we quantified EHR-continuity by mean proportion of encounters captured (MPEC) by the EHR system. We built a prediction model for MPEC using one EHR system as training and the other as the validation set. Patients with top 20% predicted EHR-continuity had 3.5-5.8-fold smaller misclassification of 40 CER-relevant variables, compared to the remaining study population. The comorbidity profiles did not differ substantially by predicted EHR-continuity. These findings suggest that restriction of CER to patients with high predicted EHR-continuity may confer a favorable validity to generalizability trade-off.

Reporting Data Quality Assessment Results: Identifying Individual and Organizational Barriers and Solutions

Introduction:

Electronic health record (EHR) data are known to have significant data quality issues, yet the practice and frequency of assessing EHR data is unknown. We sought to understand current practices and attitudes towards reporting data quality assessment (DQA) results by data professionals.

Methods:

The project was conducted in four Phases: (1) examined current DQA practices among informatics/CER stakeholders via engagement meeting (07/2014); (2) characterized organizations conducting DQA by interviewing key personnel and data management professionals (07-08/2014); (3) developed and administered an anonymous survey to data professionals (03-06/2015); and (4) validated survey results during a follow-up informatics/CER stakeholder engagement meeting (06/2016).

Results:

The first engagement meeting identified the theme of unintended consequences as a primary barrier to DQA. Interviewees were predominantly medical groups serving distributed networks with formalized DQAs. Consistent with the interviews, most survey (N=111) respondents utilized DQA processes/programs. A lack of resources and clear definitions of how to judge the quality of a dataset were the most commonly cited individual barriers. Vague quality action plans/expectations and data owners not trained in problem identification and problem-solving skills were the most commonly cited organizational barriers. Solutions included allocating resources for DQA, establishing standards and guidelines, and changing organizational culture.

Discussion:

Several barriers affecting DQA and reporting were identified. Community alignment towards systematic DQA and reporting is needed to overcome these barriers.

Conclusion:

Understanding barriers and solutions to DQA reporting is vital for establishing trust in the secondary use of EHR data for quality improvement and the pursuit of personalized medicine.

Using Certification to Promote Uptake of Real-World Evidence by Payers

Most randomized controlled trials are unable to generate information about a product's real-world effectiveness. Therefore, payers use real-world evidence (RWE) generated in observational studies to make decisions regarding formulary inclusion and coverage. While some payers generate their own RWE, most cautiously rely on RWE produced by manufacturers who have a strong financial interest in obtaining coverage for their products. We propose a process by which an independent body would certify observational studies as generating valid and unbiased estimates of the effectiveness of the intervention under consideration. This proposed process includes (a) establishing transparent criteria for assessment, (b) implementing a process for receipt and review of observational study protocols from interested parties, (c) reviewing the submitted protocol and requesting any necessary revisions, (d) reviewing the study results, (e) assigning a certification status to the submitted evidence, and (f) communicating the certification status to all who seek to use this evidence for decision making. Accrediting organizations such as the National Center for Quality Assurance and the Joint Commission have comparable goals of providing assurance about quality to those who look to their accreditation results. Although we recognize potential barriers, including a slowing of evidence generation and costs, we anticipate that processes can be streamlined, such as when familiar methods or familiar datasets are used. The financial backing for such activities remains uncertain, as does identification of organizations that might serve this certification function. We suggest that the rigor and transparency that will be required with such a process, and the unassailable evidence that it will produce, will be valuable to decision makers.

Clinical Research Informatics for Big Data and Precision Medicine

OBJECTIVES

To reflect on the notable events and significant developments in Clinical Research Informatics (CRI) in the year of 2015 and discuss near-term trends impacting CRI.

METHODS

We selected key publications that highlight not only important recent advances in CRI but also notable events likely to have significant impact on CRI activities over the next few years or longer, and consulted the discussions in relevant scientific communities and an online living textbook for modern clinical trials. We also related the new concepts with old problems to improve the continuity of CRI research.

RESULTS

The highlights in CRI in 2015 include the growing adoption of electronic health records (EHR), the rapid development of regional, national, and global clinical data research networks for using EHR data to integrate scalable clinical research with clinical care and generate robust medical evidence. Data quality, integration, and fusion, data access by researchers, study transparency, results reproducibility, and infrastructure sustainability are persistent challenges.

CONCLUSION

The advances in Big Data Analytics and Internet technologies together with the engagement of citizens in sciences are shaping the global clinical research enterprise, which is getting more open and increasingly stakeholder-centered, where stakeholders include patients, clinicians, researchers, and sponsors.

A Harmonized Data Quality Assessment Terminology and Framework for the Secondary Use of Electronic Health Record Data

Objective:

Harmonized data quality (DQ) assessment terms, methods, and reporting practices can establish a common understanding of the strengths and limitations of electronic health record (EHR) data for operational analytics, quality improvement, and research. Existing published DQ terms were harmonized to a comprehensive unified terminology with definitions and examples and organized into a conceptual framework to support a common approach to defining whether EHR data is ‘fit’ for specific uses.

Materials and Methods:

DQ publications, informatics and analytics experts, managers of established DQ programs, and operational manuals from several mature EHR-based research networks were reviewed to identify potential DQ terms and categories. Two face-to-face stakeholder meetings were used to vet an initial set of DQ terms and definitions that were grouped into an overall conceptual framework. Feedback received from data producers and users was used to construct a draft set of harmonized DQ terms and categories. Multiple rounds of iterative refinement resulted in a set of terms and organizing framework consisting of DQ categories, subcategories, terms, definitions, and examples. The harmonized terminology and logical framework’s inclusiveness was evaluated against ten published DQ terminologies.

Results:

Existing DQ terms were harmonized and organized into a framework by defining three DQ categories: (1) Conformance (2) Completeness and (3) Plausibility and two DQ assessment contexts: (1) Verification and (2) Validation. Conformance and Plausibility categories were further divided into subcategories. Each category and subcategory was defined with respect to whether the data may be verified with organizational data, or validated against an accepted gold standard, depending on proposed context and uses. The coverage of the harmonized DQ terminology was validated by successfully aligning to multiple published DQ terminologies.

Discussion:

Existing DQ concepts, community input, and expert review informed the development of a distinct set of terms, organized into categories and subcategories. The resulting DQ terms successfully encompassed a wide range of disparate DQ terminologies. Operational definitions were developed to provide guidance for implementing DQ assessment procedures. The resulting structure is an inclusive DQ framework for standardizing DQ assessment and reporting. While our analysis focused on the DQ issues often found in EHR data, the new terminology may be applicable to a wide range of electronic health data such as administrative, research, and patient-reported data.

Conclusion:

A consistent, common DQ terminology, organized into a logical framework, is an initial step in enabling data owners and users, patients, and policy makers to evaluate and communicate data quality findings in a well-defined manner with a shared vocabulary. Future work will leverage the framework and terminology to develop reusable data quality assessment and reporting methods.

Statewide Hospital Discharge Data: Collection, Use, Limitations, and Improvements

OBJECTIVES

To provide an overview of statewide hospital discharge databases (HDD), including their uses in health services research and limitations, and to describe Agency for Healthcare Research and Quality (AHRQ) Enhanced State Data grants to address clinical and race-ethnicity data limitations.

PRINCIPAL FINDINGS

Almost all states have statewide HDD collected by public or private data organizations. Statewide HDD, based on the hospital claim with state variations, contain useful core variables and require minimal collection burden. AHRQ's Healthcare Cost and Utilization Project builds uniform state and national research files using statewide HDD. States, hospitals, and researchers use statewide HDD for many purposes. Illustrating researchers' use, during 2012-2014, HSR published 26 HDD-based articles on health policy, access, quality, clinical aspects of care, race-ethnicity and insurance impacts, economics, financing, and research methods. HDD have limitations affecting their use. Five AHRQ grants focused on enhancing clinical data and three grants aimed at improving race-ethnicity data.

CONCLUSION

ICD-10 implementation will significantly affect the HDD. The AHRQ grants, information technology advances, payment policy changes, and the need for outpatient information may stimulate other statewide HDD changes. To remain a mainstay of health services research, statewide HDD need to keep pace with changing user needs while minimizing collection burdens.