Using computer-extracted data from electronic health records to measure the quality of adolescent well-care

Process analysis of the patient pathway for automated data collection: an exemplar using pituitary surgery

Introduction

Automation of routine clinical data shows promise in relieving health systems of the burden associated with manual data collection. Identifying consistent points of documentation in the electronic health record (EHR) provides salient targets to improve data entry quality. Using our pituitary surgery service as an exemplar, we aimed to demonstrate how process mapping can be used to identify reliable areas of documentation in the patient pathway to target structured data entry interventions.

Materials and methods

This mixed methods study was conducted in the largest pituitary centre in the UK. Purposive snowball sampling identified frontline stakeholders for process mapping to produce a patient pathway. The final patient pathway was subsequently validated against a real-world dataset of 50 patients who underwent surgery for pituitary adenoma. Events were categorized by frequency and mapped to the patient pathway to determine critical data points.

Results

Eighteen stakeholders encompassing all members of the multidisciplinary team (MDT) were consulted for process mapping. The commonest events recorded were neurosurgical ward round entries (N = 212, 14.7%), pituitary clinical nurse specialist (CNS) ward round entries (N = 88, 6.12%) and pituitary MDT treatment decisions (N = 88, 6.12%) representing critical data points. Operation notes and neurosurgical ward round entries were present for every patient. 43/44 (97.7%) had a pre-operative pituitary MDT entry, pre-operative clinic letter, a post-operative clinic letter, an admission clerking entry, a discharge summary, and a post-operative histopathology pituitary multidisciplinary (MDT) team entries.

Conclusion

This is the first study to produce a validated patient pathway of patients undergoing pituitary surgery, serving as a comparison to optimise this patient pathway. We have identified salient targets for structured data entry interventions, including mandatory datapoints seen in every admission and have also identified areas to improve documentation adherence, both of which support movement towards automation.

Development and validation of automated electronic health record data reuse for a multidisciplinary quality dashboard

Objective

To describe the development and validation of automated electronic health record data reuse for a multidisciplinary quality dashboard.

Materials and methods

Comparative study analyzing a manually extracted and an automatically extracted dataset with 262 patients treated for HNC cancer in a tertiary oncology center in the Netherlands in 2020. The primary outcome measures were the percentage of agreement on data elements required for calculating quality indicators and the difference between indicators results calculated using manually collected and indicators that used automatically extracted data.

Results

The results of this study demonstrate high agreement between manual and automatically collected variables, reaching up to 99.0% agreement. However, some variables demonstrate lower levels of agreement, with one variable showing only a 20.0% agreement rate. The indicator results obtained through manual collection and automatic extraction show high agreement in most cases, with discrepancy rates ranging from 0.3% to 3.5%. One indicator is identified as a negative outlier, with a discrepancy rate of nearly 25%.

Conclusions

This study shows that it is possible to use routinely collected structured data to reliably measure the quality of care in real-time, which could render manual data collection for quality measurement obsolete. To achieve reliable data reuse, it is important that relevant data is recorded as structured data during the care process. Furthermore, the results also imply that data validation is conditional to development of a reliable dashboard.

Comparison of electronic versus manual abstraction for 2 standardized perinatal care measures

OBJECTIVE

Given that electronic clinical quality measures (eCQMs) are playing a central role in quality improvement applications nationwide, a stronger evidence base demonstrating their reliability is critically needed. To assess the reliability of electronic health record-extracted data elements and measure results for the Elective Delivery and Exclusive Breast Milk Feeding measures (vs manual abstraction) among a national sample of US acute care hospitals, as well as common sources of discrepancies and change over time.

MATERIALS AND METHODS

eCQM and chart-abstracted data for the same patients were matched and compared at the data element and measure level for hospitals submitting both sources of data to The Joint Commission between 2017 and 2019. Sensitivity, specificity, and kappa statistics were used to assess reliability.

RESULTS

Although eCQM denominator reliability had moderate to substantial agreement for both measures and both improved over time (Elective Delivery: kappa = 0.59 [95% confidence interval (CI), 0.58-0.61] in 2017 and 0.84 [95% CI, 083-0.85] in 2019; Exclusive Breast Milk Feeding: kappa = 0.58 [95% CI, 0.54-0.62] in 2017 and 0.70 [95% CI, 0.67-0.73] in 2019), the numerator status reliability was poor for Elective Delivery (kappa = 0.08 [95% CI, 0.03-0.12] in 2017 and 0.10 [95% CI, 0.05-0.15] in 2019) but near perfect for Exclusive Breast Milk Feeding (kappa = 0.85 [0.83, 0.87] in 2017 and 0.84 [0.83, 0.85] in 2019). The failure of the eCQM to accurately capture estimated gestational age, conditions possibly justifying elective delivery, active labor, and medical induction were the main reasons for the discrepancies.

CONCLUSIONS

Although eCQM denominator reliability for the Elective Delivery and Exclusive Breast Milk Feeding measures had moderate agreement when compared to medical record review, the numerator status reliability was poor for Elective Delivery, but near perfect for Exclusive Breast Milk Feeding. Improvements in eCQM data capture of some key data elements would greatly improve the reliability.

Comparison of Performance on ADHD Quality of Care Indicators: Practitioner Self-Report Versus Chart Review

Objective: This study compared practitioner self-report of ADHD quality of care measures with actual performance, as documented by chart review. Method: In total, 188 practitioners from 50 pediatric practices completed questionnaires in which they self-reported estimates of ADHD quality of care indicators. A total of 1,599 charts were reviewed. Results: The percentage of patients for whom practitioners self-reported that they used evidence-based care was higher in every performance category when compared with chart review, including higher use of parent and teacher rating scales during assessment and treatment compared with chart review. Self-reported use of Diagnostic and Statistical Manual of Mental Disorders (4th ed.; DSM-IV) criteria during assessment was also higher than by chart review. The actual number of days until the first contact after starting medication was nearly three times longer than self-report estimates. Conclusion: Practitioners overreport performance on quality of care indicators. These differences were large and consistent across ADHD diagnostic and treatment monitoring practices. Practitioner self-report of ADHD guideline adherence should not be considered a valid measure of performance.

The Association Between Care Coordination and Preventive Care Among Children With Special Health Care Needs

Care coordination (CC) has shown positive outcomes among children with special health care needs (CSHCN); however, the association between CC and well-child care (WCC) visits is unknown. We hypothesize that CSHCN who receive CC are more likely to attend the recommended WCC visits. A retrospective cohort analysis was conducted of patients aged 15 months attending the Arizona Children's Center clinic. Logistic regression models explored the association between children receiving CC and attending the recommended minimum WCC visits before 15 months of age. CC was associated with higher odds of proper WCC attendance (any CC service, adjusted odds ratio = 2.14, 95% confidence interval = 1.75-2.62; high level of CC, adjusted odds ratio = 2.61, 95% confidence interval = 1.73-3.94). Pediatric CC is associated with greater up-to-date status of the WCC schedule among CSHCN 15 months of age, and higher odds among children who receive higher levels of CC. Further research is needed to validate findings.

Secondary EMR data for quality improvement and research: A comparison of manual and electronic data collection from an integrated critical care electronic medical record system

PURPOSE

This study measured the quality of data extracted from a clinical information system widely used for critical care quality improvement and research.

MATERIALS AND METHODS

We abstracted data from 30 fields in a random sample of 207 patients admitted to nine adult, medical-surgical intensive care units. We assessed concordance between data collected: (1) manually from the bedside system (eCritical MetaVision) by trained auditors, and (2) electronically from the system data warehouse (eCritical TRACER). Agreement was assessed using Cohen's Kappa for categorical variables and intraclass correlation coefficient (ICC) for continuous variables.

RESULTS

Concordance between data sets was excellent. There was perfect agreement for 11/30 variables (35%). The median Kappa score for the 16 categorical variables was 0.99 (IQR 0.92-1.00). APACHE II had an ICC of 0.936 (0.898-0.960). The lowest concordance was observed for SOFA renal and respiratory components (ICC 0.804 and 0.846, respectively). Score translation errors by the manual auditor were the most common source of data discrepancies.

CONCLUSIONS

Manual validation processes of electronic data are complex in comparison to validation of traditional clinical documentation. This study represents a straightforward approach to validate the use of data repositories to support reliable and efficient use of high quality secondary use data.

Using Clinical Data Standards to Measure Quality: A New Approach

Background  Value-based payment for care requires the consistent, objective calculation of care quality. Previous initiatives to calculate ambulatory quality measures have relied on billing data or individual electronic health records (EHRs) to calculate and report performance. New methods for quality measure calculation promoted by federal regulations allow qualified clinical data registries to report quality outcomes based on data aggregated across facilities and EHRs using interoperability standards.

Objective  This research evaluates the use of clinical document interchange standards as the basis for quality measurement.

Methods  Using data on 1,100 patients from 11 ambulatory care facilities and 5 different EHRs, challenges to quality measurement are identified and addressed for 17 certified quality measures.

Results  Iterative solutions were identified for 14 measures that improved patient inclusion and measure calculation accuracy. Findings validate this approach to improving measure accuracy while maintaining measure certification.

Conclusion  Organizations that report care quality should be aware of how identified issues affect quality measure selection and calculation. Quality measure authors should consider increasing real-world validation and the consistency of measure logic in respect to issues identified in this research.

Is It Feasible to Use Electronic Health Records for Quality Measurement of Adolescent Care?

OBJECTIVE

To determine the extent to which it is feasible to implement quality measures on electronic health records (EHRs) as currently implemented in pediatric health centers.

METHODS

A survey of information technology professionals at 10 institutions that provide primary care services to adolescents. The survey asked whether data about care was being captured electronically across the nine domains relevant to adolescent well care: Screening, Health Risks, Sexual Health, Diagnosis and History, Laboratory Results, Prescriptions, Referrals, Forms Management, and Patient Demographics. For each domain, we developed a scale of the extent to which the EHR makes quality measurement feasible.

RESULTS

Overall feasibility scores varied across centers from 34% to 85% and from 53% to 80% across care domains. One centre reported 100% feasibility for 8 of 10 care domains.

CONCLUSIONS

Electronic health records can facilitate quality improvement, but the feasibility of such use depends on the presence, validity, and accessibility of the quality data in the EHR. Even among the largest and most sophisticated pediatric EHR systems, quality of care measurement is not possible yet for all aspects of adolescent well care without manual effort to review and code data. Nevertheless, almost all quality measures were reported to be feasible in some systems.

"Salt in the Wound": Safety Net Clinician Perspectives on Performance Feedback Derived From EHR Data

Electronic health record (EHR) data can be extracted for calculating performance feedback, but users' perceptions of such feedback impact its effectiveness. Through qualitative analyses, we identified perspectives on barriers and facilitators to the perceived legitimacy of EHR-based performance feedback, in 11 community health centers (CHCs). Providers said such measures rarely accounted for CHC patients' complex lives or for providers' decisions as informed by this complexity, which diminished the measures' perceived validity. Suggestions for improving the perceived validity of performance feedback in CHCs are presented. Our findings add to the literature on EHR-based performance feedback by exploring provider perceptions in CHCs.

A new era of quality measurement in rheumatology: electronic clinical quality measures and national registries

PURPOSE OF REVIEW

This article reviews the evolution of quality measurement in rheumatology, highlighting new health-information technology infrastructure and standards that are enabling unprecedented innovation in this field.

RECENT FINDINGS

Spurred by landmark legislation that ties physician payment to value, the widespread use of electronic health records, and standards such as the Quality Data Model, quality measurement in rheumatology is rapidly evolving. Rather than relying on retrospective assessments of care gathered through administrative claims or manual chart abstraction, new electronic clinical quality measures (eCQMs) allow automated data capture from electronic health records. At the same time, qualified clinical data registries, like the American College of Rheumatology's Rheumatology Informatics System for Effectiveness registry, are enabling large-scale implementation of eCQMs across national electronic health record networks with real-time performance feedback to clinicians. Although successful examples of eCQM development and implementation in rheumatology and other fields exist, there also remain challenges, such as lack of health system data interoperability and problems with measure accuracy.

SUMMARY

Quality measurement and improvement is increasingly an essential component of rheumatology practice. Advances in health information technology are likely to continue to make implementation of eCQMs easier and measurement more clinically meaningful and accurate in coming years.

Calculations of Financial Incentives for Providers in a Pay-for-Performance Program: Manual Review Versus Data From Structured Fields in Electronic Health Records

BACKGROUND

Hospital report cards and financial incentives linked to performance require clinical data that are reliable, appropriate, timely, and cost-effective to process. Pay-for-performance plans are transitioning to automated electronic health record (EHR) data as an efficient method to generate data needed for these programs.

OBJECTIVE

To determine how well data from automated processing of structured fields in the electronic health record (AP-EHR) reflect data from manual chart review and the impact of these data on performance rewards.

RESEARCH DESIGN

Cross-sectional analysis of performance measures used in a cluster randomized trial assessing the impact of financial incentives on guideline-recommended care for hypertension.

SUBJECTS

A total of 2840 patients with hypertension assigned to participating physicians at 12 Veterans Affairs hospital-based outpatient clinics. Fifty-two physicians and 33 primary care personnel received incentive payments.

MEASURES

Overall, positive and negative agreement indices and Cohen's kappa were calculated for assessments of guideline-recommended antihypertensive medication use, blood pressure (BP) control, and appropriate response to uncontrolled BP. Pearson's correlation coefficient was used to assess how similar participants' calculated earnings were between the data sources.

RESULTS

By manual chart review data, 72.3% of patients were considered to have received guideline-recommended antihypertensive medications compared with 65.0% by AP-EHR review (κ=0.51). Manual review indicated 69.5% of patients had controlled BP compared with 66.8% by AP-EHR review (κ=0.87). Compared with 52.2% of patients per the manual review, 39.8% received an appropriate response by AP-EHR review (κ=0.28). Participants' incentive payments calculated using the 2 methods were highly correlated (r≥0.98). Using the AP-EHR data to calculate earnings, participants' payment changes ranged from a decrease of $91.00 (-30.3%) to an increase of $18.20 (+7.4%) for medication use (interquartile range, -14.4% to 0%) and a decrease of $100.10 (-31.4%) to an increase of $36.40 (+15.4%) for BP control or appropriate response to uncontrolled BP (interquartile range, -11.9% to -6.1%).

CONCLUSIONS

Pay-for-performance plans that use only EHR data should carefully consider the measures and the structure of the EHR before data collection and financial incentive disbursement. For this study, we feel that a 10% difference in the total amount of incentive earnings disbursed based on AP-EHR data compared with manual review is acceptable given the time and resources required to abstract data from medical records.

Statewide Hospital Discharge Data: Collection, Use, Limitations, and Improvements

OBJECTIVES

To provide an overview of statewide hospital discharge databases (HDD), including their uses in health services research and limitations, and to describe Agency for Healthcare Research and Quality (AHRQ) Enhanced State Data grants to address clinical and race-ethnicity data limitations.

PRINCIPAL FINDINGS

Almost all states have statewide HDD collected by public or private data organizations. Statewide HDD, based on the hospital claim with state variations, contain useful core variables and require minimal collection burden. AHRQ's Healthcare Cost and Utilization Project builds uniform state and national research files using statewide HDD. States, hospitals, and researchers use statewide HDD for many purposes. Illustrating researchers' use, during 2012-2014, HSR published 26 HDD-based articles on health policy, access, quality, clinical aspects of care, race-ethnicity and insurance impacts, economics, financing, and research methods. HDD have limitations affecting their use. Five AHRQ grants focused on enhancing clinical data and three grants aimed at improving race-ethnicity data.

CONCLUSION

ICD-10 implementation will significantly affect the HDD. The AHRQ grants, information technology advances, payment policy changes, and the need for outpatient information may stimulate other statewide HDD changes. To remain a mainstay of health services research, statewide HDD need to keep pace with changing user needs while minimizing collection burdens.