Review: electronic health records and the reliability and validity of quality measures: a review of the literature

A basic model for assessing primary health care electronic medical record data quality

Background

The increased use of electronic medical records (EMRs) in Canadian primary health care practice has resulted in an expansion of the availability of EMR data. Potential users of these data need to understand their quality in relation to the uses to which they are applied. Herein, we propose a basic model for assessing primary health care EMR data quality, comprising a set of data quality measures within four domains. We describe the process of developing and testing this set of measures, share the results of applying these measures in three EMR-derived datasets, and discuss what this reveals about the measures and EMR data quality. The model is offered as a starting point from which data users can refine their own approach, based on their own needs.

Methods

Using an iterative process, measures of EMR data quality were created within four domains: comparability; completeness; correctness; and currency. We used a series of process steps to develop the measures. The measures were then operationalized, and tested within three datasets created from different EMR software products.

Results

A set of eleven final measures were created. We were not able to calculate results for several measures in one dataset because of the way the data were collected in that specific EMR. Overall, we found variability in the results of testing the measures (e.g. sensitivity values were highest for diabetes, and lowest for obesity), among datasets (e.g. recording of height), and by patient age and sex (e.g. recording of blood pressure, height and weight).

Conclusions

This paper proposes a basic model for assessing primary health care EMR data quality. We developed and tested multiple measures of data quality, within four domains, in three different EMR-derived primary health care datasets. The results of testing these measures indicated that not all measures could be utilized in all datasets, and illustrated variability in data quality. This is one step forward in creating a standard set of measures of data quality. Nonetheless, each project has unique challenges, and therefore requires its own data quality assessment before proceeding.

Electronic supplementary material

The online version of this article (10.1186/s12911-019-0740-0) contains supplementary material, which is available to authorized users.

Primary Care Practices' Abilities And Challenges In Using Electronic Health Record Data For Quality Improvement

Federal value-based payment programs require primary care practices to conduct quality improvement activities, informed by the electronic reports on clinical quality measures that their electronic health records (EHRs) generate. To determine whether EHRs produce reports adequate to the task, we examined survey responses from 1,492 practices across twelve states, supplemented with qualitative data. Meaningful-use participation, which requires the use of a federally certified EHR, was associated with the ability to generate reports-but the reports did not necessarily support quality improvement initiatives. Practices reported numerous challenges in generating adequate reports, such as difficulty manipulating and aligning measurement time frames with quality improvement needs, lack of functionality for generating reports on electronic clinical quality measures at different levels, discordance between clinical guidelines and measures available in reports, questionable data quality, and vendors that were unreceptive to changing EHR configuration beyond federal requirements. The current state of EHR measurement functionality may be insufficient to support federal initiatives that tie payment to clinical quality measures.

Adverse events in prehospital emergency care: a trigger tool study

Background

Prehospital emergency care has developed rapidly during the past decades. The care is given in a complex context which makes prehospital care a potential high-risk activity when it comes to patient safety. Patient safety in the prehospital setting has been only sparsely investigated. The aims of the present study were 1) To investigate the incidence of adverse events (AEs) in prehospital care and 2) To investigate the factors contributing to AEs in prehospital care.

Methods

We used a retrospective study design where 30 randomly selected prehospital medical records were screened for AEs each month in three prehospital organizations in Sweden during a period of one year. A total of 1080 prehospital medical records were included. The record review was based on the use of 11 screening criteria.

Results

The reviewers identified 46 AEs in 46 of 1080 (4.3%) prehospital medical records. Of the 46 AEs, 43 were classified as potential for harm (AE1) (4.0, 95% CI = 2.9–5.4) and three as harm identified (AE2) (0.3, 95% CI = 0.1–0.9). However, among patients with a life-threatening condition (priority 1), the risk of AE was higher (16.5%). The most common factors contributing to AEs were deviations from standard of care and missing, incomplete, or unclear documentation. The most common cause of AEs was the result of action(s) or inaction(s) by the emergency medical service (EMS) crew.

Conclusions

There were 4.3 AEs per 100 ambulance missions in Swedish prehospital care. The majority of AEs originated from deviations from standard of care and incomplete documentation. There was an increase in the risk of AE among patients who the EMS team assessed as having a life-threatening condition. Most AEs were possible to avoid.

Electronic supplementary material

The online version of this article (10.1186/s12873-019-0228-3) contains supplementary material, which is available to authorized users.

Assessment and Stratification of High-Impact Data Elements in Electronic Clinical Quality Measures: A Joint Data Quality Initiative Between CancerLinQ® and Cancer Treatment Centers of America

PURPOSE

A joint data quality initiative between the Cancer Treatment Centers of America and the ASCO big data health technology platform CancerLinQ® was initiated to document and codify the steps taken to evaluate, stratify, and determine the potential effect of data elements used for electronic clinical quality measures as captured within structured fields in electronic health records.

METHODS

The processes involved the identification of clinical concepts required in measure population criteria and then to map these to the corresponding components of the CancerLinQ data model. A quantitative assessment of mappings between electronic clinical quality measure clinical concepts and attributes from the CancerLinQ clinical database was performed. In parallel, a qualitative analysis of high-impact data elements from the Cancer Treatment Centers of America clinical measures was made using local, expert consensus.

RESULTS

An impact assessment was derived using a count of the data elements across measures and the specific population criteria affected.

CONCLUSION

A list of putative high-impact data elements can provide guidance for clinicians to facilitate specific data element capture related to quality metrics in an electronic environment.

Measuring the quality and completeness of medication-related information derived from hospital electronic health records database

Objective

Electronic Health Records (EHRs) database is a great source for pharmacoepidemiological research as thousands of patients' clinical and medication information is stored in the database. However, the use of EHRs database for research purposes depends greatly on the accuracy and completeness of the data being used. This study mainly aimed to assess the completeness of EHRs patients' medication-related information.

Design

A retrospective cross-sectional study using data extracted from the EHRs database was conducted.

Setting

The EHRs data was obtained from a single tertiary hospital in Saudi Arabia.

Main outcome measures

The completeness of data was measured considering if a patients' record contains all desired types of data (i.e., patients' demographics, clinical diagnosis, and medication-related information).

Results

A total of 23,411 unique individuals were identified after extracting the data from the EHRs. The study found that 89.9% of the patients had a complete data (i.e., age, gender, marital status, nationality, encounter type, and clinical diagnosis). Further, 83.1% of the patients had complete medication-related information. Subgroup analysis by the encounter type indicated that the data was 91.0% complete for outpatient encounter and 93.2% complete for inpatient encounter.

Conclusion

The study findings indicate that the completeness of the data varies by the desired types of data. EHRs can be a potentially great resource to conduct research to assess medication use. Further studies focusing on the content and completeness of EHRs for a specific patient population and evaluate other dimensions of EHRs data quality are needed.

Can we understand population healthcare needs using electronic medical records?

INTRODUCTION

The identification of population-level healthcare needs using hospital electronic medical records (EMRs) is a promising approach for the evaluation and development of tailored healthcare services. Population segmentation based on healthcare needs may be possible using information on health and social service needs from EMRs. However, it is currently unknown if EMRs from restructured hospitals in Singapore provide information of sufficient quality for this purpose. We compared the inter-rater reliability between a population segment that was assigned prospectively and one that was assigned retrospectively based on EMR review.

METHODS

200 non-critical patients aged ≥ 55 years were prospectively evaluated by clinicians for their healthcare needs in the emergency department at Singapore General Hospital, Singapore. Trained clinician raters with no prior knowledge of these patients subsequently accessed the EMR up to the prospective rating date. A similar healthcare needs evaluation was conducted using the EMR. The inter-rater reliability between the two rating sets was evaluated using Cohen's Kappa and the incidence of missing information was tabulated.

RESULTS

The inter-rater reliability for the medical 'global impression' rating was 0.37 for doctors and 0.35 for nurses. The inter-rater reliability for the same variable, retrospectively rated by two doctors, was 0.75. Variables with a higher incidence of missing EMR information such as 'social support in case of need' and 'patient activation' had poorer inter-rater reliability.

CONCLUSION

Pre-existing EMR systems may not capture sufficient information for reliable determination of healthcare needs. Thus, we should consider integrating policy-relevant healthcare need variables into EMRs.

The Reach and Feasibility of an Interactive Lung Cancer Screening Decision Aid Delivered by Patient Portal

OBJECTIVE

Health systems could adopt population-level approaches to screening by identifying potential screening candidates from the electronic health record and reaching out to them via the patient portal. However, whether patients would read or act on sent information is unknown. We examined the feasibility of this digital health outreach strategy.

METHODS

We conducted a single-arm pragmatic trial in a large academic health system. An electronic health record algorithm identified primary care patients who were potentially eligible for lung cancer screening (LCS). Identified patients were sent a patient portal invitation to visit a LCS interactive Web site which assessed screening eligibility and included a decision aid. The primary outcome was screening completion. Secondary outcomes included the proportion of patients who read the invitation, visited the interactive Web site, and completed the interactive Web site.

RESULTS

We sent portal invitations to 1,000 patients. Almost all patients (86%, 862/1,000) read the invitation, 404 (40%) patients visited the interactive Web site, and 349 patients (35%) completed it. Of the 99 patients who were confirmed screening eligible by the Web site, 81 made a screening decision (30% wanted screening, 44% unsure, 26% declined screening), and 22 patients had a chest computed tomography completed.

CONCLUSION

The digital outreach strategy reached the majority of patient portal users. While the study focused on LCS, this digital outreach approach could be generalized to other health needs. Given the broad reach and potential low cost of this digital strategy, future research should investigate best practices for implementing the system.

A New Insight Into Missing Data in Intensive Care Unit Patient Profiles: Observational Study

Background

The data missing from patient profiles in intensive care units (ICUs) are substantial and unavoidable. However, this incompleteness is not always random or because of imperfections in the data collection process.

Objective

This study aimed to investigate the potential hidden information in data missing from electronic health records (EHRs) in an ICU and examine whether the presence or missingness of a variable itself can convey information about the patient health status.

Methods

Daily retrieval of laboratory test (LT) measurements from the Medical Information Mart for Intensive Care III database was set as our reference for defining complete patient profiles. Missingness indicators were introduced as a way of representing presence or absence of the LTs in a patient profile. Thereafter, various feature selection methods (filter and embedded feature selection methods) were used to examine the predictive power of missingness indicators. Finally, a set of well-known prediction models (logistic regression [LR], decision tree, and random forest) were used to evaluate whether the absence status itself of a variable recording can provide predictive power. We also examined the utility of missingness indicators in improving predictive performance when used with observed laboratory measurements as model input. The outcome of interest was in-hospital mortality and mortality at 30 days after ICU discharge.

Results

Regardless of mortality type or ICU day, more than 40% of the predictors selected by feature selection methods were missingness indicators. Notably, employing missingness indicators as the only predictors achieved reasonable mortality prediction on all days and for all mortality types (for instance, in 30-day mortality prediction with LR, we achieved area under the curve of the receiver operating characteristic [AUROC] of 0.6836±0.012). Including indicators with observed measurements in the prediction models also improved the AUROC; the maximum improvement was 0.0426. Indicators also improved the AUROC for Simplified Acute Physiology Score II model—a well-known ICU severity of illness score—confirming the additive information of the indicators (AUROC of 0.8045±0.0109 for 30-day mortality prediction for LR).

Conclusions

Our study demonstrated that the presence or absence of LT measurements is informative and can be considered a potential predictor of in-hospital and 30-day mortality. The comparative analysis of prediction models also showed statistically significant prediction improvement when indicators were included. Moreover, missing data might reflect the opinions of examining clinicians. Therefore, the absence of measurements can be informative in ICUs and has predictive power beyond the measured data themselves. This initial case study shows promise for more in-depth analysis of missing data and its informativeness in ICUs. Future studies are needed to generalize these results.

Exploring completeness in clinical data research networks with DQe-c

Objective

To provide an open source, interoperable, and scalable data quality assessment tool for evaluation and visualization of completeness and conformance in electronic health record (EHR) data repositories.

Materials and Methods

This article describes the tool’s design and architecture and gives an overview of its outputs using a sample dataset of 200 000 randomly selected patient records with an encounter since January 1, 2010, extracted from the Research Patient Data Registry (RPDR) at Partners HealthCare. All the code and instructions to run the tool and interpret its results are provided in the Supplementary Appendix.

Results

DQ^e-c produces a web-based report that summarizes data completeness and conformance in a given EHR data repository through descriptive graphics and tables. Results from running the tool on the sample RPDR data are organized into 4 sections: load and test details, completeness test, data model conformance test, and test of missingness in key clinical indicators.

Discussion

Open science, interoperability across major clinical informatics platforms, and scalability to large databases are key design considerations for DQ^e-c. Iterative implementation of the tool across different institutions directed us to improve the scalability and interoperability of the tool and find ways to facilitate local setup.

Conclusion

EHR data quality assessment has been hampered by implementation of ad hoc processes. The architecture and implementation of DQ^e-c offer valuable insights for developing reproducible and scalable data science tools to assess, manage, and process data in clinical data repositories.

Accurate Measurement In California's Safety-Net Health Systems Has Gaps And Barriers

Patient safety in ambulatory care has not been routinely measured. California implemented a pay-for-performance program in safety-net hospitals that incentivized measurement and improvement in key areas of ambulatory safety: referral completion, medication safety, and test follow-up. We present two years of program data (collected during July 2015-June 2017) and show both suboptimal performance in aspects of ambulatory safety and questionable reliability in data reporting. Performance was better in areas that required limited coordination or patient engagement-for example, annual medication monitoring versus follow-up after high-risk mammograms. Health care systems that lack seamlessly integrated electronic health records and patient registries encountered barriers to reporting reliable ambulatory safety data, particularly for measures that integrated multiple data elements. These data challenges precluded accurate performance measurement in many areas. Policy makers and safety advocates need to support the development of information systems and measures that facilitate the accurate ascertainment of the health systems, patients, and clinical tasks at greatest risk for ambulatory safety failures.

The use of Electronic Health Records to Support Population Health: A Systematic Review of the Literature

Electronic health records (EHRs) have emerged among health information technology as "meaningful use" to improve the quality and efficiency of healthcare, and health disparities in population health. In other instances, they have also shown lack of interoperability, functionality and many medical errors. With proper implementation and training, are electronic health records a viable source in managing population health? The primary objective of this systematic review is to assess the relationship of electronic health records' use on population health through the identification and analysis of facilitators and barriers to its adoption for this purpose. Authors searched Cumulative Index of Nursing and Allied Health Literature (CINAHL) and MEDLINE (PubMed), 10/02/2012-10/02/2017, core clinical/academic journals, MEDLINE full text, English only, human species and evaluated the articles that were germane to our research objective. Each article was analyzed by multiple reviewers. Group members recognized common facilitators and barriers associated with EHRs effect on population health. A final list of articles was selected by the group after three consensus meetings (n = 55). Among a total of 26 factors identified, 63% (147/232) of those were facilitators and 37% (85/232) barriers. About 70% of the facilitators consisted of productivity/efficiency in EHRs occurring 33 times, increased quality and data management each occurring 19 times, surveillance occurring 17 times, and preventative care occurring 15 times. About 70% of the barriers consisted of missing data occurring 24 times, no standards (interoperability) occurring 13 times, productivity loss occurring 12 times, and technology too complex occurring 10 times. The analysis identified more facilitators than barriers to the use of the EHR to support public health. Wider adoption of the EHR and more comprehensive standards for interoperability will only enhance the ability for the EHR to support this important area of surveillance and disease prevention. This review identifies more facilitators than barriers to using the EHR to support public health, which implies a certain level of usability and acceptance to use the EHR in this manner. The public-health industry should combine their efforts with the interoperability projects to make the EHR both fully adopted and fully interoperable. This will greatly increase the availability, accuracy, and comprehensiveness of data across the country, which will enhance benchmarking and disease surveillance/prevention capabilities.

Automated Pulmonary Embolism Risk Classification and Guideline Adherence for Computed Tomography Pulmonary Angiography Ordering

BACKGROUND

The assessment of clinical guideline adherence for the evaluation of pulmonary embolism (PE) via computed tomography pulmonary angiography (CTPA) currently requires either labor-intensive, retrospective chart review or prospective collection of PE risk scores at the time of CTPA order. The recording of clinical data in a structured manner in the electronic health record (EHR) may make it possible to automate the calculation of a patient's PE risk classification and determine whether the CTPA order was guideline concordant.

OBJECTIVES

The objective of this study was to measure the performance of automated, structured data-only versions of the Wells and revised Geneva risk scores in emergency department (ED) encounters during which a CTPA was ordered. The hypothesis was that such an automated method would classify a patient's PE risk with high accuracy compared to manual chart review.

METHODS

We developed automated, structured data-only versions of the Wells and revised Geneva risk scores to classify 212 ED encounters during which a CTPA was performed as "PE likely" or "PE unlikely." We then combined these classifications with D-dimer ordering data to assess each encounter as guideline concordant or discordant. The accuracy of these automated classifications and assessments of guideline concordance were determined by comparing them to classifications and concordance based on the complete Wells and revised Geneva scores derived via abstractor manual chart review.

RESULTS

The automatically derived Wells and revised Geneva risk classifications were 91.5 and 92% accurate compared to the manually determined classifications, respectively. There was no statistically significant difference between guideline adherence calculated by the automated scores compared to manual chart review (Wells, 70.8% vs. 75%, p = 0.33; revised Geneva, 65.6% vs. 66%, p = 0.92).

CONCLUSION

The Wells and revised Geneva score risk classifications can be approximated with high accuracy using automated extraction of structured EHR data elements in patients who received a CTPA. Combining these automated scores with D-dimer ordering data allows for the automated assessment of clinical guideline adherence for CTPA ordering in the ED, without the burden of manual chart review.

Data collection and research with MargheritaTre

OBJECTIVE

MargheritaTre is an electronic health record developed by the Italian Group for the Evaluation of Interventions in Intensive Care Medicine designed to support clinical practice in intensive care units (ICUs) and ensure high-quality data for research purposes.

APPROACH

MargheritaTre was developed in collaboration with clinical experts, researchers, and IT specialists. It is currently installed in 40 ICUs and its database contains complete records of more than 65,000 patients. To facilitate data analysis, information is mostly stored in structured or partially structured form.

MAIN RESULTS

Data collected with MargheritaTre allow one to conduct research studies on complex clinical problems from manifold perspectives and with different levels of detail, such as epidemiological studies, analyses of the process of care and physiopathological investigations, at both single-organ and organism level. In this paper we describe some of the first projects based on this electronic health record to illustrate its potential for research.

SIGNIFICANCE

The MargheritaTre database is a huge and rapidly growing mine of data that will be exploited by our laboratory and shared with other groups to address complex and innovative research and clinical questions. The ultimate aim of these projects is the improvement of the quality of care and patient outcomes, through the development of expert systems integrated in the electronic health record to support clinical practice.

How We Should Assess the Delivery of End-Of-Life Care in General Practice? A Systematic Review

BACKGROUND

The majority of end-of-life (EOL) care occurs in general practice. However, we still have little knowledge about how this care is delivered or how it can be assessed and supported.

AIM

(i) To review the existing evaluation tools used for assessment of the delivery of EOL care from the perspective of general practice; (ii) To describe how EOL care is provided in general practice; (iii) To identify major areas of concern in providing EOL care in this context.

DESIGN

A systematic review.

DATA SOURCES

Systematic searches of major electronic databases (Medline, EMBASE, PsycINFO, and CINAHL) from inception to 2017 were used to identify evaluation tools focusing on organizational structures/systems and process of end-of-life care from a general practice perspective.

RESULTS

A total of 43 studies representing nine evaluation tools were included. A relatively restricted focus and lack of validation were common limitations. Key general practitioner (GP) activities assessed by the evaluation tools were summarized and the main issues in current GP EOL care practice were identified.

CONCLUSIONS

The review of evaluation tools revealed that GPs are highly involved in management of patients at the EOL, but there are a range of issues relating to the delivery of care. An EOL care registration system integrated with electronic health records could provide an optimal approach to address the concerns about recall bias and time demands in retrospective analyses. Such a system should ideally capture the core GP activities and any major issues in care provision on a case-by-case basis.

Recognising the importance of informal communication events in improving collaborative care

Background

While the concept of collaboration is highly touted in the literature, most descriptions of effective collaboration highlight formal collaborative events; largely ignored are the informal collaborative events and none focusing on the frequent, ‘seemingly’ by chance communication events that arise and their role in supporting patient safety and quality care.

Objective

To identify the types of informal communication events that exist in the inpatient setting and better understand the barriers contributing to their necessity.

Methods

We undertook a constructivist grounded theory study in an inpatient internal medicine teaching unit in Ontario, Canada. Interview and observational data were collected across two phases; in total, 56 participants were consented for the study. Data collection and analysis occurred iteratively; themes were identified using constant comparison methods.

Results

Several types of informal communication events were identified and appeared valuable in three ways: (1) providing a better sense of a patient’s baseline function in comparison to their current function; (2) gaining a more holistic understanding of the patient’s needs; and (3) generating better insight into a patient’s wishes and goals of care. Participants identified a number of organisational and communication challenges leading to the need for informal communication events. These included: scheduling, competing demands and the spatial and temporal organisation of the ward. As a result, nursing staff, allied health professionals and caregivers had to develop strategies for interacting with the physician team.

Conclusion

We highlight the importance of informal communication in supporting patient care and the gaps in the system contributing to their necessity. Changes at the system level are needed to ensure we are not leaving important collaborative opportunities to chance alone.

Technology-enabled performance monitoring in intensive care: An integrative literature review

BACKGROUND

Implementation of evidence-based bundles in intensive care units is integral to improving quality of care and patient outcomes. However, it increases the burden of data collection and analysis required for performance monitoring and feedback of an inter-disciplinary care team. Health information technology including electronic health records and data analytic tools could automate this process and provide real-time feedback to the team.

AIM

This integrative literature review aimed to examine the extent to which technology-enabled performance monitoring and feedback contributed to improving quality of care and patient outcomes when implementing evidence-based bundles.

METHODS

A literature search of scientific databases was conducted using PubMed, Embase, Scopus, CINHAL and Ovid Medline.

RESULTS

Of nine studies included in this review, all reported improved compliance of the team with evidence-based bundles, ranging from 3% to 60% post implementation of technology-enabled performance monitoring and feedback. Significant reductions (p < .05) in hospital acquired infections were also reported in five studies.

CONCLUSIONS

Overall, the addition of documentation fields to electronic health records was essential in providing real-time feedback to teams and improving their compliance with evidence-based bundles. Further research is needed to assess the effectiveness of technology-enabled performance monitoring and feedback in improving patient outcomes on a larger scale, especially in resource-limited settings such as community hospitals.

Use of an electronic health record data sharing system for identifying current contraceptive use within the WWAMI region Practice and Research Network

OBJECTIVE

To evaluate the ability of electronic health record (EHR) data extracted into a data-sharing system to accurately identify contraceptive use.

STUDY DESIGN

We compared rates of contraceptive use from electronic extraction of EHR data via a data-sharing system and manual abstraction of the EHR among 142 female patients ages 15-49 years from a family medicine clinic within a primary care practice-based research network (PBRN). Cohen's kappa coefficient measured agreement between electronic extraction and manual abstraction.

RESULTS

Manual abstraction identified 62% of women as contraceptive users, whereas electronic extraction identified only 27%. Long acting reversible (LARC) methods had 96% agreement (Cohen's kappa 0.78; confidence interval, 0.57-0.99) between electronic extraction and manual abstraction. EHR data extracted via a data-sharing system was unable to identify barrier or over-the-counter contraceptives.

CONCLUSIONS

Electronic extraction found substantially lower overall rates of contraceptive method use, but produced more comparable LARC method use rates when compared to manual abstraction among women in this study's primary care clinic.

IMPLICATIONS

Quality metrics related to contraceptive use that rely on EHR data in this study's data-sharing system likely under-estimated true contraceptive use.

Using Clinical Data Standards to Measure Quality: A New Approach

Background  Value-based payment for care requires the consistent, objective calculation of care quality. Previous initiatives to calculate ambulatory quality measures have relied on billing data or individual electronic health records (EHRs) to calculate and report performance. New methods for quality measure calculation promoted by federal regulations allow qualified clinical data registries to report quality outcomes based on data aggregated across facilities and EHRs using interoperability standards.

Objective  This research evaluates the use of clinical document interchange standards as the basis for quality measurement.

Methods  Using data on 1,100 patients from 11 ambulatory care facilities and 5 different EHRs, challenges to quality measurement are identified and addressed for 17 certified quality measures.

Results  Iterative solutions were identified for 14 measures that improved patient inclusion and measure calculation accuracy. Findings validate this approach to improving measure accuracy while maintaining measure certification.

Conclusion  Organizations that report care quality should be aware of how identified issues affect quality measure selection and calculation. Quality measure authors should consider increasing real-world validation and the consistency of measure logic in respect to issues identified in this research.

Possible Sources of Bias in Primary Care Electronic Health Record Data Use and Reuse

BACKGROUND

Enormous amounts of data are recorded routinely in health care as part of the care process, primarily for managing individual patient care. There are significant opportunities to use these data for other purposes, many of which would contribute to establishing a learning health system. This is particularly true for data recorded in primary care settings, as in many countries, these are the first place patients turn to for most health problems.

OBJECTIVE

In this paper, we discuss whether data that are recorded routinely as part of the health care process in primary care are actually fit to use for other purposes such as research and quality of health care indicators, how the original purpose may affect the extent to which the data are fit for another purpose, and the mechanisms behind these effects. In doing so, we want to identify possible sources of bias that are relevant for the use and reuse of these type of data.

METHODS

This paper is based on the authors' experience as users of electronic health records data, as general practitioners, health informatics experts, and health services researchers. It is a product of the discussions they had during the Translational Research and Patient Safety in Europe (TRANSFoRm) project, which was funded by the European Commission and sought to develop, pilot, and evaluate a core information architecture for the learning health system in Europe, based on primary care electronic health records.

RESULTS

We first describe the different stages in the processing of electronic health record data, as well as the different purposes for which these data are used. Given the different data processing steps and purposes, we then discuss the possible mechanisms for each individual data processing step that can generate biased outcomes. We identified 13 possible sources of bias. Four of them are related to the organization of a health care system, whereas some are of a more technical nature.

CONCLUSIONS

There are a substantial number of possible sources of bias; very little is known about the size and direction of their impact. However, anyone that uses or reuses data that were recorded as part of the health care process (such as researchers and clinicians) should be aware of the associated data collection process and environmental influences that can affect the quality of the data. Our stepwise, actor- and purpose-oriented approach may help to identify these possible sources of bias. Unless data quality issues are better understood and unless adequate controls are embedded throughout the data lifecycle, data-driven health care will not live up to its expectations. We need a data quality research agenda to devise the appropriate instruments needed to assess the magnitude of each of the possible sources of bias, and then start measuring their impact. The possible sources of bias described in this paper serve as a starting point for this research agenda.

Reliability of a Canadian database for primary care nursing services' clinical and administrative data

BACKGROUND

The use of electronic clinical and administrative data can be an advantageous source of information for assessing nursing performance in primary care. In Québec (Canada), the I-CLSC electronic database could be used to measure performance indicators. However, little is known about the reliability of the data contained in this database. The objective of this study was to assess the reliability of the clinical and administrative data contained in the I-CLSC electronic database based on the data entered in medical records.

METHODS

We used a longitudinal design for this study. A sample of 100 patients who had experienced 107 episodes of wound care were randomly selected from all patients who had two or more consultations during the year 2015. The paper records were used as reference. We collected data regarding eight nursing sensitive indicators from both sources. We assessed the concordance between the electronic data and the paper records by measuring inter-rater agreement.

RESULTS

Six of the eight indicators showed a percentage agreement ≥ 85%, and kappa scores between 0.7 and 1.00 (p < 0.001), indicating high to perfect levels of agreement between the two data sources. Two indicators presented fair kappa scores.

CONCLUSION

This database provides reliable data relating to the organization of care but shows lower reliability for specific acts performed by nurses in primary care. This existing database can be used to assess, manage and improve certain dimensions of nursing performance in primary care.

Evaluation of Multimedia Medication Reconciliation Software: A Randomized Controlled, Single-Blind Trial to Measure Diagnostic Accuracy for Discrepancy Detection

BACKGROUND

The Veterans Affairs Portland Healthcare System developed a medication history collection software that displays prescription names and medication images.

OBJECTIVE

This article measures the frequency of medication discrepancy reporting using the medication history collection software and compares with the frequency of reporting using a paper-based process. This article also determines the accuracy of each method by comparing both strategies to a best possible medication history.

STUDY DESIGN

Randomized, controlled, single-blind trial.

SETTING

Three community-based primary care clinics associated with the Veterans Affairs Portland Healthcare System: a 300-bed teaching facility and ambulatory care network serving Veteran soldiers in the Pacific Northwest United States.

PARTICIPANTS

Of 212 patients with primary care appointments, 209 patients fulfilled the study requirements.

INTERVENTION

Patients randomized to a software-directed medication history or a paper-based medication history. Randomization and allocation to treatment groups were performed using a computer-based random number generator. Assignments were placed in a sealed envelope and opened after participant consent. The research coordinator did not know or have access to the treatment assignment until the time of presentation.

MAIN OUTCOME MEASURES

The primary analysis compared the discrepancy detection rates between groups with respect to the health record and a best possible medication history.

RESULTS

Of 3,500 medications reviewed, we detected 1,435 discrepancies. Forty-six percent of those discrepancies were potentially high risk for causing an adverse drug event. There was no difference in detection rates between treatment arms. Software sensitivity was 83% and specificity was 91%; paper sensitivity was 81% and specificity was 94%. No participants were lost to follow-up.

CONCLUSION

The medication history collection software is an efficient and scalable method for gathering a medication history and detecting high-risk discrepancies. Although it included medication images, the technology did not improve accuracy over a paper list when compared with a best possible medication history.

TRIAL REGISTRATION

ClinicalTrials.gov Identifier: NCT02135731.

Police-Recorded Crime and Disparities in Obesity and Blood Pressure Status in Chicago

BACKGROUND

The purpose of this study was to examine associations between several types of police-recorded crime (violent, nonviolent, and homicide) and cardiometabolic health (obesity and elevated blood pressure [BP]), and to determine if associations were modified by age and sex.

METHODS AND RESULTS

We analyzed cross-sectional data (N=14 799 patients) from 3 primary care clinics at an academic medical center in Chicago, IL. Patient-level health data were obtained from the electronic health record (June 1, 2014-May 31, 2015), including body mass index and BP, and linked to the City of Chicago Police Data Portal. Geocoded crime counts were aggregated to census tract and calculated as the annual crime rate per 1000 population. Generalized linear mixed models were used to assess obesity and BP status as a function of crime rate quartile, controlling for patient, clinic, and neighborhood characteristics. Median violent crime rates in each quartile ranged from 15 to 84 per 1000 population. Median age was 56 years (interquartile range, 38-72 years); 42% of patients were obese and 33% had elevated BP. Compared with patients living in the lowest quartile, patients living in the highest quartile for violent crime had 53% higher adjusted odds of obesity (95% confidence interval, 1.15-2.03) and 25% higher adjusted odds of elevated BP (95% confidence interval, 1.01-1.56). In subanalyses examining homicide, a relatively rare event, exposure was not associated with obesity and was inconsistently associated with elevated BP.

CONCLUSIONS

In a densely populated, high-poverty region in Chicago, recurrent exposure to high rates of violent crime was consistently associated with obesity and elevated BP, but rare exposure to homicide was not.

Knowledge discovery for Deep Phenotyping serious mental illness from Electronic Mental Health records

Background: Deep Phenotyping is the precise and comprehensive analysis of phenotypic features in which the individual components of the phenotype are observed and described. In UK mental health clinical practice, most clinically relevant information is recorded as free text in the Electronic Health Record, and offers a granularity of information beyond what is expressed in most medical knowledge bases. The SNOMED CT nomenclature potentially offers the means to model such information at scale, yet given a sufficiently large body of clinical text collected over many years, it is difficult to identify the language that clinicians favour to express concepts.

Methods: By utilising a large corpus of healthcare data, we sought to make use of semantic modelling and clustering techniques to represent the relationship between the clinical vocabulary of internationally recognised SMI symptoms and the preferred language used by clinicians within a care setting. We explore how such models can be used for discovering novel vocabulary relevant to the task of phenotyping Serious Mental Illness (SMI) with only a small amount of prior knowledge. 

Results: 20 403 terms were derived and curated via a two stage methodology. The list was reduced to 557 putative concepts based on eliminating redundant information content. These were then organised into 9 distinct categories pertaining to different aspects of psychiatric assessment. 235 concepts were found to be expressions of putative clinical significance. Of these, 53 were identified having novel synonymy with existing SNOMED CT concepts. 106 had no mapping to SNOMED CT.

Conclusions: We demonstrate a scalable approach to discovering new concepts of SMI symptomatology based on real-world clinical observation. Such approaches may offer the opportunity to consider broader manifestations of SMI symptomatology than is typically assessed via current diagnostic frameworks, and create the potential for enhancing nomenclatures such as SNOMED CT based on real-world expressions.

Developing and Trialling an Independent, Scalable and Repeatable IT-benchmarking Procedure for Healthcare Organisations

Background: Continuous improvements of IT-performance in healthcare organisations require actionable performance indicators, regularly conducted, independent measurements and meaningful and scalable reference groups. Existing IT-benchmarking initiatives have focussed on the development of reliable and valid indicators, but less on the questions about how to implement an environment for conducting easily repeatable and scalable IT-benchmarks.

Objectives: This study aims at developing and trialling a procedure that meets the afore-mentioned requirements.

Methods: We chose a well established, regularly conducted (inter-) national IT-survey of healthcare organisations (IT-Report Health-care) as the environment and offered the participants of the 2011 survey (CIOs of hospitals) to enter a benchmark. The 61 structural and functional performance indicators cov -ered among others the implementation status and integration of IT-systems and functions, global user satisfaction and the resources of the IT-department. Healthcare organisations were grouped by size and ownership. The benchmark results were made available electronically and feedback on the use of these results was requested after several months.

Results: Fifty-nine hospitals participated in the benchmarking. Reference groups consisted of up to 141 members depending on the number of beds (size) and the ownership (public vs. private). A total of 122 charts showing single indicator frequency views were sent to each participant. The evaluation showed that 94.1% of the CIOs who participated in the evaluation considered this benchmarking beneficial and reported that they would enter again. Based on the feedback of the participants we developed two additional views that provide a more consolidated picture.

Conclusion: The results demonstrate that establishing an independent, easily repeatable and scalable IT-benchmarking procedure is possible and was deemed desirable. Based on these encouraging results a new benchmarking round which includes process indicators is currently conducted.

Cross sectional study to assess the accuracy of electronic health record data to identify patients in need of lung cancer screening

Objective

Lung cancer is the leading cause of cancer death in the United States [Siegel et al. in CA Cancer J Clin 66:7–30, 1]. However, evidence from clinical trials indicates that annual low-dose computed tomography screening reduces lung cancer mortality [Humphrey et al. in Ann Intern Med 159:411–420, 2]. The objective of this study is to report results of a study designed to assess the sensitivity, specificity, and positive and negative predictive value of an electronic health record (EHR) query in comparison to patient self-report, to identify patients who may benefit from lung cancer screening. Cross sectional study comparing patient self report to EHR derived assessment of tobacco status and need for lung cancer screening. We invited 200 current or former smokers, ages 55–80 to complete a brief paper survey. 26 responded and 24 were included in the analysis.

Results

For 30% of respondents, there was not adequate EHR data to make a lung cancer screening determination. Compared to patient self-report, EHR derived data has a 67% sensitivity and 82% specificity for identifying patients that meet criteria for lung cancer screening. While the degree of accuracy may be insufficient to make a final lung cancer screening determination, EHR data may be useful in prompting clinicians to initiate conversations with patients in regards to lung cancer screening.

Data Cleaning in the Evaluation of a Multi-Site Intervention Project

Context:

The High Value Healthcare Collaborative (HVHC) sepsis project was a two-year multi-site project where Member health care delivery systems worked on improving sepsis care using a dissemination & implementation framework designed by HVHC. As part of the project evaluation, participating Members provided 5 data submissions over the project period. Members created data files using a uniform specification, but the data sources and methods used to create the data sets differed. Extensive data cleaning was necessary to get a data set usable for the evaluation analysis.

Case Description:

HVHC was the coordinating center for the project and received and cleaned all data submissions. Submissions received 3 sequentially more detailed levels of checking by HVHC. The most detailed level evaluated validity by comparing values within-Member over time and between Member. For a subset of episodes Member-submitted data were compared to matched Medicare claims data.

Findings:

Inconsistencies in data submissions, particularly for length-of-stay variables were common in early submissions and decreased with subsequent submissions. Multiple resubmissions were sometimes required to get clean data. Data checking also uncovered a systematic difference in the way Medicare and some members defined intensive care unit stay.

Conclusions:

Data checking is a critical for ensuring valid analytic results for projects using electronic health record data. It is important to budget sufficient resources for data checking. Interim data submissions and checks help find anomalies early. Data resubmissions should be checked as fixes can introduce new errors. Communicating with those responsible for creating the data set provides critical information.

Quantifying the Effect of Data Quality on the Validity of an eMeasure

Objective The objective of this study was to demonstrate the utility of a healthcare data quality framework by using it to measure the impact of synthetic data quality issues on the validity of an eMeasure (CMS178—urinary catheter removal after surgery). Methods Data quality issues were artificially created by systematically degrading the underlying quality of EHR data using two methods: independent and correlated degradation. A linear model that describes the change in the events included in the eMeasure quantifies the impact of each data quality issue. Results Catheter duration had the most impact on the CMS178 eMeasure with every 1% reduction in data quality causing a 1.21% increase in the number of missing events. For birth date and admission type, every 1% reduction in data quality resulted in a 1% increase in missing events. Conclusion This research demonstrated that the impact of data quality issues can be quantified using a generalized process and that the CMS178 eMeasure, as currently defined, may not measure how well an organization is meeting the intended best practice goal. Secondary use of EHR data is warranted only if the data are of sufficient quality. The assessment approach described in this study demonstrates how the impact of data quality issues on an eMeasure can be quantified and the approach can be generalized for other data analysis tasks. Healthcare organizations can prioritize data quality improvement efforts to focus on the areas that will have the most impact on validity and assess whether the values that are reported should be trusted.

[Effectiveness of an assessment of computerised medical records in a university hospital]

OBJECTIVE

The objective of this study was to evaluate a sample of electronic medical records (EMR) that was sufficiently representative of the different areas of care in a university hospital, as well as to verify the effectiveness of an initial intervention through a second evaluation.

METHODS

Medical records audits were performed in 2012 and 2013 by a blind peer review of random samples of care episodes, proportional to the activity of each clinical department, and with the same evaluation method being applied to all of them.

RESULTS

More than 1,000 episodes of care were reviewed in the 2audits. A significant improvement was found in hospital admissions (P=.000) in all the sections of the EMR analysed (P=.002), and was especially significant for the reason for consultation, for which its completion increased by 8.5% (p<.05), and also in the sections of the current process record (7.1%), physical examination (4.7%), allergies (3.9%), and clinical course (3.6%). The assessment of the discharge report, as a whole, showed an improvement (P=.001). In outpatient follow-up visits, a significant positive improvement was observed in the 4sections evaluated (P<.05), and also overall (P=.000).

CONCLUSIONS

According to study conditions, the dissemination of the results was effective in improving the quality of the EMR. The results have made it possible to implement actions to review the work processes in certain departments, and also the partial redesign of the interface on being a reproducible methodology accepted by the organisation.

A Data Quality Assessment Guideline for Electronic Health Record Data Reuse

Introduction:

We describe the formulation, development, and initial expert review of 3x3 Data Quality Assessment (DQA), a dynamic, evidence-based guideline to enable electronic health record (EHR) data quality assessment and reporting for clinical research.

Methods:

3x3 DQA was developed through the triangulation results from three studies: a review of the literature on EHR data quality assessment, a quantitative study of EHR data completeness, and a set of interviews with clinical researchers. Following initial development, the guideline was reviewed by a panel of EHR data quality experts.

Results:

The guideline embraces the task-dependent nature of data quality and data quality assessment. The core framework includes three constructs of data quality: complete, correct, and current data. These constructs are operationalized according to the three primary dimensions of EHR data: patients, variables, and time. Each of the nine operationalized constructs maps to a methodological recommendation for EHR data quality assessment. The initial expert response to the framework was positive, but improvements are required.

Discussion:

The initial version of 3x3 DQA promises to enable explicit guideline-based best practices for EHR data quality assessment and reporting. Future work will focus on increasing clarity on how and when 3x3 DQA should be used during the research process, improving the feasibility and ease-of-use of recommendation execution, and clarifying the process for users to determine which operationalized constructs and recommendations are relevant for a given dataset and study.

Reporting Data Quality Assessment Results: Identifying Individual and Organizational Barriers and Solutions

Introduction:

Electronic health record (EHR) data are known to have significant data quality issues, yet the practice and frequency of assessing EHR data is unknown. We sought to understand current practices and attitudes towards reporting data quality assessment (DQA) results by data professionals.

Methods:

The project was conducted in four Phases: (1) examined current DQA practices among informatics/CER stakeholders via engagement meeting (07/2014); (2) characterized organizations conducting DQA by interviewing key personnel and data management professionals (07-08/2014); (3) developed and administered an anonymous survey to data professionals (03-06/2015); and (4) validated survey results during a follow-up informatics/CER stakeholder engagement meeting (06/2016).

Results:

The first engagement meeting identified the theme of unintended consequences as a primary barrier to DQA. Interviewees were predominantly medical groups serving distributed networks with formalized DQAs. Consistent with the interviews, most survey (N=111) respondents utilized DQA processes/programs. A lack of resources and clear definitions of how to judge the quality of a dataset were the most commonly cited individual barriers. Vague quality action plans/expectations and data owners not trained in problem identification and problem-solving skills were the most commonly cited organizational barriers. Solutions included allocating resources for DQA, establishing standards and guidelines, and changing organizational culture.

Discussion:

Several barriers affecting DQA and reporting were identified. Community alignment towards systematic DQA and reporting is needed to overcome these barriers.

Conclusion:

Understanding barriers and solutions to DQA reporting is vital for establishing trust in the secondary use of EHR data for quality improvement and the pursuit of personalized medicine.

A validation of clinical data captured from a novel Cancer Care Quality Program directly integrated with administrative claims data

Background

Data from a Cancer Care Quality Program are directly integrated with administrative claims data to provide a level of clinical detail not available in claims-based studies, and referred to as the HealthCore Integrated Research Environment (HIRE)-Oncology data. This study evaluated the validity of the HIRE-Oncology data compared with medical records of breast, lung, and colorectal cancer patients.

Methods

Data elements included cancer type, stage, histology (lung only), and biomarkers. A sample of 300 breast, 200 lung, and 200 colorectal cancer patients within the HIRE-Oncology data were identified for medical record review. Statistical measures of validity (agreement, positive predictive value [PPV], negative predictive value [NPV], sensitivity, specificity) were used to compare clinical information between data sources, with medical record data considered the gold standard.

Results

All 300 breast cancer records reviewed were confirmed breast cancer, while 197 lung and 197 colorectal records were confirmed (PPV =0.99 for each). The agreement of disease stage was 85% for breast, 90% for lung, and 94% for colorectal cancer. The agreement of lung cancer histology (small cell vs non-small cell) was 97%. Agreement of progesterone receptor, estrogen receptor, and human epidermal growth factor receptor 2 status biomarkers in breast cancer was 92%, 97%, and 92%, respectively; epidermal growth factor receptor and anaplastic lymphoma kinase agreement in lung was 97% and 92%, respectively; and agreement of KRAS status in colorectal cancer was 95%. Measures of PPV, NPV, sensitivity, and specificity showed similarly strong evidence of validity.

Conclusion

Good agreement between the HIRE-Oncology data and medical records supports the validity of these data for research.

Comparing Population-based Risk-stratification Model Performance Using Demographic, Diagnosis and Medication Data Extracted From Outpatient Electronic Health Records Versus Administrative Claims

BACKGROUND

There is an increasing demand for electronic health record (EHR)-based risk stratification and predictive modeling tools at the population level. This trend is partly due to increased value-based payment policies and the increasing availability of EHRs at the provider level. Risk stratification models, however, have been traditionally derived from claims or encounter systems. This study evaluates the challenges and opportunities of using EHR data instead of or in addition to administrative claims for risk stratification.

METHODS

This study used the structured EHR records and administrative claims of 85,581 patients receiving outpatient care at a large integrated provider system. Common data elements for risk stratification (ie, age, sex, diagnosis, and medication) were extracted from outpatient EHR records and administrative claims. The performance of a validated risk-stratification model was assessed using data extracted from claims alone, EHR alone, and claims and EHR combined.

RESULTS

EHR-derived metrics overlapped considerably with administrative claims (eg, number of chronic conditions). The accuracy of the model, when using EHR data alone, was acceptable with an area under the curve of ∼0.81 for hospitalization and ∼0.85 for identifying top 1% utilizers using the concurrent model. However, when using EHR data alone, the predictive model explained a lower amount of variation in utilization-based outcomes compared with administrative claims.

DISCUSSION

The results show a promising performance of models predicting cost and hospitalization using outpatient EHR's diagnosis and medication data. More research is needed to evaluate the benefits of other EHR data types (eg, lab values and vital signs) for risk stratification.

Bayesian analysis of genetic association across tree-structured routine healthcare data in the UK Biobank

Genetic discovery from the multitude of phenotypes extractable from routine healthcare data can transform our understanding of the human phenome and accelerate progress towards precision medicine. However, a critical question when analysing high-dimensional and heterogeneous data is how to best interrogate increasingly specific subphenotypes whilst retaining statistical power to detect genetic associations. Here we develop and employ a novel Bayesian analysis framework that exploits the hierarchical structure of diagnosis classifications to analyse genetic variants against UK Biobank disease phenotypes derived from self-reporting and hospital episode statistics. Our method displays a more than 20% increase in power to detect genetic effects over other approaches and identifies novel associations between classical human leukocyte antigen (HLA) alleles and common immune-mediated diseases (IMDs). By applying the approach to genetic risk scores (GRSs) we reveal the extent of genetic sharing between IMDs and expose differences in disease perception or diagnosis with potential clinical implications.

Validation of multisource electronic health record data: an application to blood transfusion data

Background

Although data from electronic health records (EHR) are often used for research purposes, systematic validation of these data prior to their use is not standard practice. Existing validation frameworks discuss validity concepts without translating these into practical implementation steps or addressing the potential influence of linking multiple sources. Therefore we developed a practical approach for validating routinely collected data from multiple sources and to apply it to a blood transfusion data warehouse to evaluate the usability in practice.

Methods

The approach consists of identifying existing validation frameworks for EHR data or linked data, selecting validity concepts from these frameworks and establishing quantifiable validity outcomes for each concept. The approach distinguishes external validation concepts (e.g. concordance with external reports, previous literature and expert feedback) and internal consistency concepts which use expected associations within the dataset itself (e.g. completeness, uniformity and plausibility). In an example case, the selected concepts were applied to a transfusion dataset and specified in more detail.

Results

Application of the approach to a transfusion dataset resulted in a structured overview of data validity aspects. This allowed improvement of these aspects through further processing of the data and in some cases adjustment of the data extraction. For example, the proportion of transfused products that could not be linked to the corresponding issued products initially was 2.2% but could be improved by adjusting data extraction criteria to 0.17%.

Conclusions

This stepwise approach for validating linked multisource data provides a basis for evaluating data quality and enhancing interpretation. When the process of data validation is adopted more broadly, this contributes to increased transparency and greater reliability of research based on routinely collected electronic health records.

Electronic supplementary material

The online version of this article (doi:10.1186/s12911-017-0504-7) contains supplementary material, which is available to authorized users.

A Comparison of Data Quality Assessment Checks in Six Data Sharing Networks

Objective:

To compare rule-based data quality (DQ) assessment approaches across multiple national clinical data sharing organizations.

Methods:

Six organizations with established data quality assessment (DQA) programs provided documentation or source code describing current DQ checks. DQ checks were mapped to the categories within the data verification context of the harmonized DQA terminology. To ensure all DQ checks were consistently mapped, conventions were developed and four iterations of mapping performed. Difficult-to-map DQ checks were discussed with research team members until consensus was achieved.

Results:

Participating organizations provided 11,026 DQ checks, of which 99.97 percent were successfully mapped to a DQA category. Of the mapped DQ checks (N=11,023), 214 (1.94 percent) mapped to multiple DQA categories. The majority of DQ checks mapped to Atemporal Plausibility (49.60 percent), Value Conformance (17.84 percent), and Atemporal Completeness (12.98 percent) categories.

Discussion:

Using the common DQA terminology, near-complete (99.97 percent) coverage across a wide range of DQA programs and specifications was reached. Comparing the distributions of mapped DQ checks revealed important differences between participating organizations. This variation may be related to the organization’s stakeholder requirements, primary analytical focus, or maturity of their DQA program. Not within scope, mapping checks within the data validation context of the terminology may provide additional insights into DQA practice differences.

Conclusion:

A common DQA terminology provides a means to help organizations and researchers understand the coverage of their current DQA efforts as well as highlight potential areas for additional DQA development. Sharing DQ checks between organizations could help expand the scope of DQA across clinical data networks.

DQe-v: A Database-Agnostic Framework for Exploring Variability in Electronic Health Record Data Across Time and Site Location

Data variability is a commonly observed phenomenon in Electronic Health Records (EHR) data networks. A common question asked in scientific investigations of EHR data is whether the cross-site and -time variability reflects an underlying data quality error at one or more contributing sites versus actual differences driven by various idiosyncrasies in the healthcare settings. Although research analysts and data scientists have commonly used various statistical methods to detect and account for variability in analytic datasets, self service tools to facilitate exploring cross-organizational variability in EHR data warehouses are lacking and could benefit from meaningful data visualizations. DQ^e-v, an interactive, database-agnostic tool for visually exploring variability in EHR data provides such a solution. DQ^e-v is built on an open source platform, R statistical software, with annotated scripts and a readme document that makes it fully reproducible. To illustrate and describe functionality of DQ^e-v, we describe the DQ^e-v’s readme document which includes a complete guide to installation, running the program, and interpretation of the outputs. We also provide annotated R scripts and an example dataset as supplemental materials. DQ^e-v offers a self service tool to visually explore data variability within EHR datasets irrespective of the data model. GitHub and CIELO offer hosting and distribution of the tool and can facilitate collaboration across any interested community of users as we target improving usability, efficiency, and interoperability.

Measuring Preventable Outcomes: Global Cardiovascular Risk (GCVR)

The National Committee for Quality Assurance (NCQA) piloted a new approach to quality measurement meant to reduce avoidable cardiac events and improve overall population health. In this pilot, we investigated whether a standardized technical specification could sufficiently define a process to reliably generate predicted outcome scores from heterogeneous electronic clinical data systems (ECDS) [1]. Patient data were electronically extracted from four health care organizations and processed by the Archimedes, Inc. Global Outcomes calculator, generating scores indicating future cardiovascular event probability for each provider's patient population. These Global Cardiovascular Risk (GCVR) scores represent the gap between current level of care achieved and optimal care for each clinician's patients, with a greater score indicating better performance. As GCVR requires more patient data than do traditional quality measures, we addressed feasibility and data completeness questions in order to understand the prospects of a wholly new quality concept. This pilot successfully produced GCVR scores for 2,251 clinicians, representing approximately 60 percent of the total patient population under study. To our knowledge, this is the first time predictive models have been proposed for quality measurement, and our pilot successfully demonstrated that a predicted outcome measure is feasible using electronic patient data. However, new specification standards are required before this approach is fully scalable to a national quality reporting program.

Strategies for handling missing clinical data for automated surgical site infection detection from the electronic health record

Proper handling of missing data is important for many secondary uses of electronic health record (EHR) data. Data imputation methods can be used to handle missing data, but their use for analyzing EHR data is limited and specific efficacy for postoperative complication detection is unclear. Several data imputation methods were used to develop data models for automated detection of three types (i.e., superficial, deep, and organ space) of surgical site infection (SSI) and overall SSI using American College of Surgeons National Surgical Quality Improvement Project (NSQIP) Registry 30-day SSI occurrence data as a reference standard. Overall, models with missing data imputation almost always outperformed reference models without imputation that included only cases with complete data for detection of SSI overall achieving very good average area under the curve values. Missing data imputation appears to be an effective means for improving postoperative SSI detection using EHR clinical data.

Data Quality in Electronic Health Records Research: Quality Domains and Assessment Methods

The proliferation of the electronic health record (EHR) has led to increasing interest and opportunities for nurse scientists to use EHR data in a variety of research designs. However, methodological problems pertaining to data quality may arise when EHR data are used for nonclinical purposes. Therefore, this article describes common domains of data quality and approaches for quality appraisal in EHR research. Common data quality domains include data accuracy, completeness, consistency, credibility, and timeliness. Approaches for quality appraisal include data validation with data rules, evaluation and verification of data abstraction methods with statistical measures, data comparisons with manual chart review, management of missing data using statistical methods, and data triangulation between multiple EHR databases. Quality data enhance the validity and reliability of research findings, form the basis for conclusions derived from the data, and are, thus, an integral component in EHR-based study design and implementation.

How to Screen for Non-Adherence to Antihypertensive Therapy

The quality of assessment of non-adherence to treatment in hypertensive is poor. Within this review, we discuss the different methods used to assess adherence to blood-pressure-lowering medications in hypertension patients. Subjective reports such as physicians' perceptions are inaccurate, and questionnaires completed by patients tend to overreport adherence and show a low diagnostic specificity. Indirect objective methods such as pharmacy database records can be useful, but they are limited by the robustness of the recorded data. Electronic medication monitoring devices are accurate but usually track adherence to only a single medication and can be expensive. Overall, the fundamental issue with indirect objective measures is that they do not fully confirm ingestion of antihypertensive medications. Detection of antihypertensive medications in body fluids using liquid chromatography-tandem mass spectrometry is currently, in our view, the most robust and clinically useful method to assess non-adherence to blood-pressure-lowering treatment. It is particularly helpful in patients presenting with resistant, refractory or uncontrolled hypertension despite the optimal therapy. We recommend using this diagnostic strategy to detect non-adherence alongside a no-blame approach tailoring support to address the perceptions (e.g. beliefs about the illness and treatment) and practicalities (e.g. capability and resources) influencing motivation and ability to adhere.

Increasing the Detection of Familial Hypercholesterolaemia Using General Practice Electronic Databases

BACKGROUND

Familial hypercholesterolaemia (FH) is a common autosomal co-dominant condition that causes premature cardiovascular disease. Awareness of FH is poor and only 10-15% of the affected population is identified. Electronic health records provide an opportunity to increase detection and awareness in general practice OBJECTIVE: To determine whether a simple electronic extraction tool can increase detection of FH in general practice.

METHOD

An extraction tool applied to general practice electronic health records (EHR) to screen for FH, total cholesterol and low density lipoprotein cholesterol (LDL-c) levels in association with entered diagnostic criteria and demographic data in five general practices.

RESULTS

Of 157,290 active patients examined, 0.7% (n=1081) had an LDL-c>5.0 mmol/L representing 1 in 146 of active patients. An additional 0.8% (n=1276) patients were at possible risk of FH. Of those with an LDL-c>5.0 mmol/L 43.7% of patients had no record of being prescribed statins. Twenty patients (0.013%) had a clinical diagnosis of FH entered in the EHR.

CONCLUSIONS

Patients at high risk of FH can be identified by a simple electronic screening method in general practice. Clinical data entry is variable in general practice. Targeted screening enables clinical assessment of patients at risk of cardiovascular disease and using the DLCNS will enable primary care to increase identification of FH. Approximately one in five patients extracted using this method, are likely to have phenotypically probable FH, making it a useful screening tool.

Clinical Research Informatics for Big Data and Precision Medicine

OBJECTIVES

To reflect on the notable events and significant developments in Clinical Research Informatics (CRI) in the year of 2015 and discuss near-term trends impacting CRI.

METHODS

We selected key publications that highlight not only important recent advances in CRI but also notable events likely to have significant impact on CRI activities over the next few years or longer, and consulted the discussions in relevant scientific communities and an online living textbook for modern clinical trials. We also related the new concepts with old problems to improve the continuity of CRI research.

RESULTS

The highlights in CRI in 2015 include the growing adoption of electronic health records (EHR), the rapid development of regional, national, and global clinical data research networks for using EHR data to integrate scalable clinical research with clinical care and generate robust medical evidence. Data quality, integration, and fusion, data access by researchers, study transparency, results reproducibility, and infrastructure sustainability are persistent challenges.

CONCLUSION

The advances in Big Data Analytics and Internet technologies together with the engagement of citizens in sciences are shaping the global clinical research enterprise, which is getting more open and increasingly stakeholder-centered, where stakeholders include patients, clinicians, researchers, and sponsors.

Data extraction from electronic health records (EHRs) for quality measurement of the physical therapy process: comparison between EHR data and survey data

BACKGROUND

With the emergence of the electronic health records (EHRs) as a pervasive healthcare information technology, new opportunities and challenges for use of clinical data for quality measurements arise with respect to data quality, data availability and comparability. The objective of this study is to test whether data extracted from electronic health records (EHRs) was of comparable quality as survey data for the calculation of quality indicators.

METHODS

Data from surveys describing patient cases and filled out by physiotherapists in 2009-2010 were used to calculate scores on eight quality indicators (QIs) to measure the quality of physiotherapy care. In 2011, data was extracted directly from EHRs. The data collection methods were evaluated for comparability. EHR data was compared to survey data on completeness and correctness.

RESULTS

Five of the eight QIs could be extracted from the EHRs. Three were omitted from the indicator set, as they proved too difficult to be extracted from the EHRs. Another QI proved incomparable due to errors in the extraction software of some of the EHRs. Three out of four comparable QIs performed better (p < 0.001) in EHR data on completeness. EHR data also proved to be correct; the relative change in indicator scores between EHR and survey data were small (<5 %) in three out of four QIs.

CONCLUSION

Data quality of EHRs was sufficient to be used for the calculation of QIs, although comparability to survey data was problematic. Standardization is needed, not only to be able to compare different data collection methods properly, but also to compare between practices with different EHRs. EHRs have the option to administrate narrative data, but natural language processing tools are needed to quantify these text boxes. Such development, can narrow the comparability gap between scoring QIs based on EHR data and based on survey data. EHRs have the potential to provide real time feedback to professionals and quality measurements for research, but more effort is needed to create unambiguous and uniform information and to unlock written text in a standardized manner.

Measuring Preventive Care Delivery: Comparing Rates Across Three Data Sources

INTRODUCTION

Preventive care delivery is an important quality outcome, and electronic data reports are being used increasingly to track these services. It is highly informative when electronic data sources are compared to information manually extracted from medical charts to assess validity and completeness.

METHODS

This cross-sectional study used a random sample of Medicaid-insured patients seen at 43 community health centers in 2011 to calculate standard measures of correspondence between manual chart review and two automated sources (electronic health records [EHRs] and Medicaid claims), comparing documentation of orders for and receipt of ten preventive services (n=150 patients/service). Data were analyzed in 2015.

RESULTS

Using manual chart review as the gold standard, automated EHR extraction showed near-perfect to perfect agreement (κ=0.96-1.0) for services received within the primary care setting (e.g., BMI, blood pressure). Receipt of breast and colorectal cancer screenings, services commonly referred out, showed moderate (κ=0.42) to substantial (κ=0.62) agreement, respectively. Automated EHR extraction showed near-perfect agreement (κ=0.83-0.97) for documentation of ordered services. Medicaid claims showed near-perfect agreement (κ=0.87) for hyperlipidemia and diabetes screening, and substantial agreement (κ=0.67-0.80) for receipt of breast, cervical, and colorectal cancer screenings, and influenza vaccination. Claims showed moderate agreement (κ=0.59) for chlamydia screening receipt. Medicaid claims did not capture ordered or unbilled services.

CONCLUSIONS

Findings suggest that automated EHR and claims data provide valid sources for measuring receipt of most preventive services; however, ordered and unbilled services were primarily captured via EHR data and completed referrals were more often documented in claims data.

Big Data, Big Problems: Incorporating Mission, Values, and Culture in Provider Affiliations

This article explores how integration of data from clinical registries and electronic health records produces a quality impact within orthopedic practices. Data are differentiated from information, and several types of data that are collected and used in orthopedic outcome measurement are defined. Furthermore, the concept of comparative effectiveness and its impact on orthopedic clinical research are assessed. This article places emphasis on how the concept of big data produces health care challenges balanced with benefits that may be faced by patients and orthopedic surgeons. Finally, essential characteristics of an electronic health record that interlinks musculoskeletal care and big data initiatives are reviewed.

A Harmonized Data Quality Assessment Terminology and Framework for the Secondary Use of Electronic Health Record Data

Objective:

Harmonized data quality (DQ) assessment terms, methods, and reporting practices can establish a common understanding of the strengths and limitations of electronic health record (EHR) data for operational analytics, quality improvement, and research. Existing published DQ terms were harmonized to a comprehensive unified terminology with definitions and examples and organized into a conceptual framework to support a common approach to defining whether EHR data is ‘fit’ for specific uses.

Materials and Methods:

DQ publications, informatics and analytics experts, managers of established DQ programs, and operational manuals from several mature EHR-based research networks were reviewed to identify potential DQ terms and categories. Two face-to-face stakeholder meetings were used to vet an initial set of DQ terms and definitions that were grouped into an overall conceptual framework. Feedback received from data producers and users was used to construct a draft set of harmonized DQ terms and categories. Multiple rounds of iterative refinement resulted in a set of terms and organizing framework consisting of DQ categories, subcategories, terms, definitions, and examples. The harmonized terminology and logical framework’s inclusiveness was evaluated against ten published DQ terminologies.

Results:

Existing DQ terms were harmonized and organized into a framework by defining three DQ categories: (1) Conformance (2) Completeness and (3) Plausibility and two DQ assessment contexts: (1) Verification and (2) Validation. Conformance and Plausibility categories were further divided into subcategories. Each category and subcategory was defined with respect to whether the data may be verified with organizational data, or validated against an accepted gold standard, depending on proposed context and uses. The coverage of the harmonized DQ terminology was validated by successfully aligning to multiple published DQ terminologies.

Discussion:

Existing DQ concepts, community input, and expert review informed the development of a distinct set of terms, organized into categories and subcategories. The resulting DQ terms successfully encompassed a wide range of disparate DQ terminologies. Operational definitions were developed to provide guidance for implementing DQ assessment procedures. The resulting structure is an inclusive DQ framework for standardizing DQ assessment and reporting. While our analysis focused on the DQ issues often found in EHR data, the new terminology may be applicable to a wide range of electronic health data such as administrative, research, and patient-reported data.

Conclusion:

A consistent, common DQ terminology, organized into a logical framework, is an initial step in enabling data owners and users, patients, and policy makers to evaluate and communicate data quality findings in a well-defined manner with a shared vocabulary. Future work will leverage the framework and terminology to develop reusable data quality assessment and reporting methods.

Learning About Missing Data Mechanisms in Electronic Health Records-based Research: A Survey-based Approach

BACKGROUND

Bias due to missing data is a major concern in electronic health record (EHR)-based research. As part of an ongoing EHR-based study of weight change among patients treated for depression, we conducted a survey to investigate determinants of missingness in the available weight information and to evaluate the missing-at-random assumption.

METHODS

We identified 8,345 individuals enrolled in a large EHR-based health care system who had monotherapy treatment for depression from April 2008 to March 2010. A stratified sample of 1,153 individuals completed a detailed survey. Logistic regression was used to investigate determinants of whether a patient (1) had an opportunity to be weighed at treatment initiation (baseline), and (2) had a weight measurement recorded. Parallel analyses were conducted to investigate missingness during follow-up. Throughout, inverse-probability weighting was used to adjust for the design and survey nonresponse. Analyses were also conducted to investigate potential recall bias.

RESULTS

Missingness at baseline and during follow-up was associated with numerous factors not routinely collected in the EHR including whether or not the patient had ever chosen not to be weighed, external weight control activities, and self-reported baseline weight. Patient attitudes about their weight and perceptions regarding the potential impact of their depression treatment on weight were not related to missingness.

CONCLUSION

Adopting a comprehensive strategy to investigate missingness early in the research process gives researchers information necessary to evaluate key assumptions. While the survey presented focuses on outcome data, the overarching strategy can be applied to any and all data elements subject to missingness.

Using primary care electronic health record data for comparative effectiveness research: experience of data quality assessment and preprocessing in The Netherlands

AIM

Details of data quality and how quality issues were solved have not been reported in published comparative effectiveness studies using electronic health record data.

METHODS

We developed a conceptual framework of data quality assessment and preprocessing and apply it to a study comparing angiotensin-converting enzyme inhibitors with angiotensin receptor blockerss on renal function decline in diabetes patients.

RESULTS

The framework establishes a line of thought to identify and act on data issues. The core concept is to evaluate whether data are fit-for-use for research tasks. Possible quality problems are listed through specific signal detections, and verified whether they are true problems. Optimal solutions are selected for the identified problems.

CONCLUSION

This framework can be used in observational studies to improve validity of results.