The challenge of measuring quality of care from the electronic health record

Developing a questionnaire to evaluate an automated audit & feedback intervention: a Rand-modified Delphi method

BACKGROUND

Audit and feedback (A&F) is a widely used implementation strategy to evaluate and improve medical practice. The optimal design of an A&F system is uncertain and structured process evaluations are currently lacking. This study aimed to develop and validate a questionnaire to evaluate the use of automated A&F systems.

METHODS

Based on the Clinical Performance Feedback Intervention Theory (CP-FIT) and the REFLECT-52 (REassessing audit & Feedback interventions: a tooL for Evaluating Compliance with suggested besT practices) evaluation tool a questionnaire was designed for the purpose of evaluating automated A&F systems. A Rand-modified Delphi method was used to develop the process evaluation and obtain validation. Fourteen experts from different domains in primary care consented to participate and individually scored the questions on a 9-point Likert scale. Afterwards, the questions were discussed in a consensus meeting. After approval, the final questionnaire was compiled.

RESULTS

A 34-question questionnaire composed of 57 items was developed and presented to the expert panel. The consensus meeting resulted in a selection of 31 questions, subdivided into 43 items. A final list of 30 questions consisting of 42 items was obtained.

CONCLUSION

A questionnaire consisting of 30 questions was drawn up for the assessment and improvement of automated A&F systems, based on CP-FIT and REFLECT-52 theory and approved by experts. Next steps will be piloting and implementation of the questionnaire.

Implementing American Academy of Neurology Quality Measures in Antigua Using Quality Improvement Methodology

Background and Objectives

The American Academy of Neurology has developed quality measures related to various neurologic disorders. A gap exists in the implementation of these measures in the different health care systems. To date, there has been no electronic health care record nor implementation of quality measures in Antigua. Therefore, we aimed to increase the percent of patients who have epilepsy quality measures documented using standardized common data elements in the outpatient neurology clinic at Sir Lester Bird Medical Center from 0% to 80% per week by June 1, 2022 and sustain for 6 months.

Methods

We used the Institute for Health care Improvement Model for Improvement methodology. A data use agreement was implemented. Data were displayed using statistical process control charts and the American Society for Quality criteria to determine statistical significance and centerline shifts.

Results

Current and future state process maps were developed to determine areas of opportunity for interventions. Interventions were developed following a "Plan-Do-Study-Act cycle." One intervention was the creation of a RedCap survey and database to be used by health care providers during clinical patient encounters. Because of multiple interventions, we achieved a 100% utilization of the survey for clinical care.

Discussion

Quality improvement (QI) methodology can be used for implementation of quality measures in various settings to improve patient care outcomes without use of significant resources. Implementation of quality measures can increase efficiency in clinical delivery. Similar QI methodology could be implemented in other resource-limited countries of the Caribbean and globally.

Improving Seizure Frequency Documentation and Classification

Background and Objectives

Accurate and reliable seizure data are essential for evaluating treatment strategies and tracking the quality of care in epilepsy clinics. This quality improvement project aimed to increase seizure documentation (i.e., documentation of seizure frequency from 80% to 100%, date of last seizure from 35% to 50%, and International League Against Epilepsy (ILAE) seizure classification from 35% to at least 50%) over 6 months.

Methods

We surveyed 7 epileptologists to determine their perceived seizure frequency, ILAE classification, and date of last seizure documentation habits. Baseline data were collected weekly from September to December 2021. Subsequently, we implemented a newly created flowsheet in our Electronic Health Record (EHR) based on the Epilepsy Learning Healthcare System (ELHS) Case Report Forms to increase seizure documentation in a standardized way. Two epileptologists tested this flowsheet tool in their epilepsy clinics between February 2022 and July 2022. Data were collected weekly and compared with documentation from other epileptologists within the same group.

Results

Epileptologists at our center believed they documented seizure frequency for 84%-87% of clinic visits, which aligned with baseline data collection, showing they recorded seizure frequency for 83% of clinic visits. Epileptologists believed they documented ILAE classification for 47%-52% of clinic visits, and baseline data showed this was documented in 33% of clinic visits. They also reported documenting the date of the last seizure for 52%-63% of clinic visits, but this occurred in only 35% of clinic visits. After implementing the new flowsheet, documentation increased to nearly 100% for all fields being completed by the providers who tested the flowsheet.

Discussion

We demonstrated that by implementing an easy-to-use standardized EHR documentation tool, our documentation of critical metrics, as defined by the ELHS, improved dramatically. This shows that simple and practical interventions can substantially improve clinically meaningful documentation.

Incidence and Outcomes of Non-Ventilator-Associated Hospital-Acquired Pneumonia in 284 US Hospitals Using Electronic Surveillance Criteria

Importance

Non-ventilator-associated hospital-acquired pneumonia (NV-HAP) is a common and deadly hospital-acquired infection. However, inconsistent surveillance methods and unclear estimates of attributable mortality challenge prevention.

Objective

To estimate the incidence, variability, outcomes, and population attributable mortality of NV-HAP.

Design, Setting, and Participants

This cohort study retrospectively applied clinical surveillance criteria for NV-HAP to electronic health record data from 284 US hospitals. Adult patients admitted to the Veterans Health Administration hospital from 2015 to 2020 and HCA Healthcare hospitals from 2018 to 2020 were included. The medical records of 250 patients who met the surveillance criteria were reviewed for accuracy.

Exposures

NV-HAP, defined as sustained deterioration in oxygenation for 2 or more days in a patient who was not ventilated concurrent with abnormal temperature or white blood cell count, performance of chest imaging, and 3 or more days of new antibiotics.

Main Outcomes and Measures

NV-HAP incidence, length-of-stay, and crude inpatient mortality. Attributable inpatient mortality by 60 days follow-up was estimated using inverse probability weighting, accounting for both baseline and time-varying confounding.

Results

Among 6 022 185 hospitalizations (median [IQR] age, 66 [54-75] years; 1 829 475 [26.1%] female), there were 32 797 NV-HAP events (0.55 per 100 admissions [95% CI, 0.54-0.55] per 100 admissions and 0.96 per 1000 patient-days [95% CI, 0.95-0.97] per 1000 patient-days). Patients with NV-HAP had multiple comorbidities (median [IQR], 6 [4-7]), including congestive heart failure (9680 [29.5%]), neurologic conditions (8255 [25.2%]), chronic lung disease (6439 [19.6%]), and cancer (5,467 [16.7%]); 24 568 cases (74.9%) occurred outside intensive care units. Crude inpatient mortality was 22.4% (7361 of 32 797) for NV-HAP vs 1.9% (115 530 of 6 022 185) for all hospitalizations; 12 449 (8.0%) were discharged to hospice. Median [IQR] length-of-stay was 16 (11-26) days vs 4 (3-6) days. On medical record review, pneumonia was confirmed by reviewers or bedside clinicians in 202 of 250 patients (81%). It was estimated that NV-HAP accounted for 7.3% (95% CI, 7.1%-7.5%) of all hospital deaths (total hospital population inpatient death risk of 1.87% with NV-HAP events included vs 1.73% with NV-HAP events excluded; risk ratio, 0.927; 95% CI, 0.925-0.929).

Conclusions and Relevance

In this cohort study, NV-HAP, which was defined using electronic surveillance criteria, was present in approximately 1 in 200 hospitalizations, of whom 1 in 5 died in the hospital. NV-HAP may account for up to 7% of all hospital deaths. These findings underscore the need to systematically monitor NV-HAP, define best practices for prevention, and track their impact.

Identifying nursing sensitive indicators from electronic health records in acute cardiac care-Towards intelligent automated assessment of care quality

AIM

The aim of this study is to explore the potential of using electronic health records for assessment of nursing care quality through nursing-sensitive indicators in acute cardiac care.

BACKGROUND

Nursing care quality is a multifaceted phenomenon, making a holistic assessment of it difficult. Quality assessment systems in acute cardiac care units could benefit from big data-based solutions that automatically extract and help interpret data from electronic health records.

METHODS

This is a deductive descriptive study that followed the theory of value-added analysis. A random sample from electronic health records of 230 patients was analysed for selected indicators. The data included documentation in structured and free-text format.

RESULTS

One thousand six hundred seventy-six expressions were extracted and divided into (1) established and (2) unestablished expressions, providing positive, neutral and negative descriptions related to care quality.

CONCLUSIONS

Electronic health records provide a potential source of information for information systems to support assessment of care quality. More research is warranted to develop, test and evaluate the effectiveness of such tools in practice.

IMPLICATIONS FOR NURSING MANAGEMENT

Knowledge-based health care management would benefit from the development and implementation of advanced information systems, which use continuously generated already available real-time big data for improved data access and interpretation to better support nursing management in quality assessment.

Behavioral Health Professionals' Perceptions on Patient-Controlled Granular Information Sharing (Part 2): Focus Group Study

BACKGROUND

Patient-directed selection and sharing of health information "granules" is known as granular information sharing. In a previous study, patients with behavioral health conditions categorized their own health information into sensitive categories (eg, mental health) and chose the health professionals (eg, pharmacists) who should have access to those records. Little is known about behavioral health professionals' perspectives of patient-controlled granular information sharing (PC-GIS).

OBJECTIVE

This study aimed to assess behavioral health professionals' (1) understanding of and opinions about PC-GIS; (2) accuracy in assessing redacted medical information; (3) reactions to patient rationale for health data categorization, assignment of sensitivity, and sharing choices; and (4) recommendations to improve PC-GIS.

METHODS

Four 2-hour focus groups and pre- and postsurveys were conducted at 2 facilities. During the focus groups, outcomes from a previous study on patients' choices for medical record sharing were discussed. Thematic analysis was applied to focus group transcripts to address study objectives.

RESULTS

A total of 28 health professionals were recruited. Over half (14/25, 56%) were unaware or provided incorrect definitions of granular information sharing. After PC-GIS was explained, all professionals demonstrated understanding of the terminology and process. Most (26/32 codes, 81%) recognized that key medical data had been redacted from the study case. A majority (41/62 codes, 66%) found the patient rationale for categorization and data sharing choices to be unclear. Finally, education and other approaches to inform and engage patients in granular information sharing were recommended.

CONCLUSIONS

This study provides detailed insights from behavioral health professionals on granular information sharing. Outcomes will inform the development, deployment, and evaluation of an electronic consent tool for granular health data sharing.

Behavioral Health Professionals' Perceptions on Patient-Controlled Granular Information Sharing (Part 1): Focus Group Study

BACKGROUND

Patient-controlled granular information sharing (PC-GIS) allows a patient to select specific health information "granules," such as diagnoses and medications; choose with whom the information is shared; and decide how the information can be used. Previous studies suggest that health professionals have mixed or concerned opinions about the process and impact of PC-GIS for care and research. Further understanding of behavioral health professionals' views on PC-GIS are needed for successful implementation and use of this technology.

OBJECTIVE

The aim of this study was to evaluate changes in health professionals' opinions on PC-GIS before and after a demonstrative case study.

METHODS

Four focus groups were conducted at two integrated health care facilities: one serious mental illness facility and one general behavioral health facility. A total of 28 participants were given access to outcomes of a previous study where patients had control over medical record sharing. Participants were surveyed before and after focus groups on their views about PC-GIS. Thematic analysis of focus group output was paired with descriptive statistics and exploratory factor analysis of surveys.

RESULTS

Behavioral health professionals showed a significant opinion shift toward concern after the focus group intervention, specifically on the topics of patient understanding (P=.001), authorized electronic health record access (P=.03), patient-professional relationship (P=.006), patient control acceptance (P<.001), and patient rights (P=.02). Qualitative methodology supported these results. The themes of professional considerations (2234/4025, 55.5% of codes) and necessity of health information (260/766, 33.9%) identified key aspects of PC-GIS concerns.

CONCLUSIONS

Behavioral health professionals agreed that a trusting patient-professional relationship is integral to the optimal implementation of PC-GIS, but were concerned about the potential negative impacts of PC-GIS on patient safety and quality of care.

Comparison of electronic versus manual abstraction for 2 standardized perinatal care measures

OBJECTIVE

Given that electronic clinical quality measures (eCQMs) are playing a central role in quality improvement applications nationwide, a stronger evidence base demonstrating their reliability is critically needed. To assess the reliability of electronic health record-extracted data elements and measure results for the Elective Delivery and Exclusive Breast Milk Feeding measures (vs manual abstraction) among a national sample of US acute care hospitals, as well as common sources of discrepancies and change over time.

MATERIALS AND METHODS

eCQM and chart-abstracted data for the same patients were matched and compared at the data element and measure level for hospitals submitting both sources of data to The Joint Commission between 2017 and 2019. Sensitivity, specificity, and kappa statistics were used to assess reliability.

RESULTS

Although eCQM denominator reliability had moderate to substantial agreement for both measures and both improved over time (Elective Delivery: kappa = 0.59 [95% confidence interval (CI), 0.58-0.61] in 2017 and 0.84 [95% CI, 083-0.85] in 2019; Exclusive Breast Milk Feeding: kappa = 0.58 [95% CI, 0.54-0.62] in 2017 and 0.70 [95% CI, 0.67-0.73] in 2019), the numerator status reliability was poor for Elective Delivery (kappa = 0.08 [95% CI, 0.03-0.12] in 2017 and 0.10 [95% CI, 0.05-0.15] in 2019) but near perfect for Exclusive Breast Milk Feeding (kappa = 0.85 [0.83, 0.87] in 2017 and 0.84 [0.83, 0.85] in 2019). The failure of the eCQM to accurately capture estimated gestational age, conditions possibly justifying elective delivery, active labor, and medical induction were the main reasons for the discrepancies.

CONCLUSIONS

Although eCQM denominator reliability for the Elective Delivery and Exclusive Breast Milk Feeding measures had moderate agreement when compared to medical record review, the numerator status reliability was poor for Elective Delivery, but near perfect for Exclusive Breast Milk Feeding. Improvements in eCQM data capture of some key data elements would greatly improve the reliability.

An updated, computable MEDication-Indication resource for biomedical research

The MEDication-Indication (MEDI) knowledgebase has been utilized in research with electronic health records (EHRs) since its publication in 2013. To account for new drugs and terminology updates, we rebuilt MEDI to overhaul the knowledgebase for modern EHRs. Indications for prescribable medications were extracted using natural language processing and ontology relationships from six publicly available resources: RxNorm, Side Effect Resource 4.1, Mayo Clinic, WebMD, MedlinePlus, and Wikipedia. We compared the estimated precision and recall between the previous MEDI (MEDI-1) and the updated version (MEDI-2) with manual review. MEDI-2 contains 3031 medications and 186,064 indications. The MEDI-2 high precision subset (HPS) includes indications found within RxNorm or at least three other resources. MEDI-2 and MEDI-2 HPS contain 13% more medications and over triple the indications compared to MEDI-1 and MEDI-1 HPS, respectively. Manual review showed MEDI-2 achieves the same precision (0.60) with better recall (0.89 vs. 0.79) compared to MEDI-1. Likewise, MEDI-2 HPS had the same precision (0.92) and improved recall (0.65 vs. 0.55) than MEDI-1 HPS. The combination of MEDI-1 and MEDI-2 achieved a recall of 0.95. In updating MEDI, we present a more comprehensive medication-indication knowledgebase that can continue to facilitate applications and research with EHRs.

The effect of electronic audits and feedback in primary care and factors that contribute to their effectiveness: a systematic review

PURPOSE

The aim of this systematic review was (i) to assess whether electronic audit and feedback (A&F) is effective in primary care and (ii) to evaluate important features concerning content and delivery of the feedback in primary care, including the use of benchmarks, the frequency of feedback, the cognitive load of feedback and the evidence-based aspects of the feedback.

DATA SOURCES

The MEDLINE, Embase, CINAHL and CENTRAL databases were searched for articles published since 2010 by replicating the search strategy used in the last Cochrane review on A&F.

STUDY SELECTION

Two independent reviewers assessed the records for their eligibility, performed the data extraction and evaluated the risk of bias. Our search resulted in 8744 records, including the 140 randomized controlled trials (RCTs) from the last Cochrane Review. The full texts of 431 articles were assessed to determine their eligibility. Finally, 29 articles were included.

DATA EXTRACTION

Two independent reviewers extracted standard data, data on the effectiveness and outcomes of the interventions, data on the kind of electronic feedback (static versus interactive) and data on the aforementioned feedback features.

RESULTS OF DATA SYNTHESIS

Twenty-two studies (76%) showed that electronic A&F was effective. All interventions targeting medication safety, preventive medicine, cholesterol management and depression showed an effect. Approximately 70% of the included studies used benchmarks and high-quality evidence in the content of the feedback. In almost half of the studies, the cognitive load of feedback was not reported. Due to high heterogeneity in the results, no meta-analysis was performed.

CONCLUSION

This systematic review included 29 articles examining electronic A&F interventions in primary care, and 76% of the interventions were effective. Our findings suggest electronic A&F is effective in primary care for different conditions such as medication safety and preventive medicine. Some of the benefits of electronic A&F include its scalability and the potential to be cost effective. The use of benchmarks as comparators and feedback based on high-quality evidence are widely used and important features of electronic feedback in primary care. However, other important features such as the cognitive load of feedback and the frequency of feedback provision are poorly described in the design of many electronic A&F intervention, indicating that a better description or implementation of these features is needed. Developing a framework or methodology for automated A&F interventions in primary care could be useful for future research.

Quality of Neurologic Care in the United States: Initial Report From the Axon Registry

OBJECTIVE

To provide the initial description of the quality of outpatient US neurologic care as collected and reported in the Axon Registry.

METHODS

We describe characteristics of registry participants and the performance of neurology providers on 20 of the 2019 Axon Registry quality measures. From the distribution of providers' scores on a quality measure, we calculate the median performance for each quality measure. We test for associations between quality measure performance, provider characteristics, and intrinsic measure parameters.

RESULTS

There were 948 neurology providers who contributed a total of 6,480 provider-metric observations. Overall, the average quality measure performance score at the provider level was 66 (median 77). At the measure level (n = 20), the average quality measure performance score was 53 (median 55) with a range of 2 to 100 (interquartile range 20-91). Measures with a lower-complexity category (e.g., discrete orders, singular concepts) or developed through the specialty's qualified clinical data registry pathway had higher performance distributions. There was no difference in performance between Merit-Based Incentive Payment System (MIPS) and non-MIPS providers. There was no association between quality measure performance and practice size, measure clinical topic/neurologic condition, or measure year of entry.

CONCLUSIONS

This cross-sectional assessment of quality measure performance in 2019 Axon Registry data demonstrates modest performance scores and considerable variability across measures and providers. More complex measures were associated with lower performance. These findings serve as a baseline assessment of quality of ambulatory neurologic care in the United States and provide insights into future measure design.

The effect of automated audit and feedback on data completeness in the electronic health record of the general physician: protocol for a cluster randomized controlled trial

Background

The electronic health record (EHR) of the general physician (GP) is an important tool that can be used to assess and improve the quality of healthcare. However, there are some problems when (re) using the data gathered in the EHR for quality assessments. One problem is the lack of data completeness in the EHR. Audit and feedback (A&F) is a well-known quality intervention that can improve the quality of healthcare. We hypothesize that an automated A&F intervention can be adapted to improve the data completeness of the EHR of the GP, more specifically, the number of correctly registered diagnoses of type 2 diabetes and chronic kidney disease.

Methods

This study is a pragmatic cluster randomized controlled trial with an intervention at the level of GP practice. The intervention consists of an audit and extended electronically delivered feedback with multiple components that will be delivered 4 times electronically to general practices over 12 months. The data will be analyzed on an aggregated level (per GP practice). The primary outcome is the percentage of correctly registered diagnoses of type 2 diabetes. The key secondary outcome is the registration of chronic kidney disease. Exploratory secondary outcomes are the registration of heart failure, biometric data and lifestyle habits, and the evolution of 4 different EHR-extractable quality indicators.

Discussion

This cluster randomized controlled trial intends to primarily improve the registration of type 2 diabetes in the EHR of the GP and to secondarily improve the registration of chronic kidney disease. In addition, the registration of heart failure, lifestyle parameters, and biometric data in the EHR of the GP are explored together with 4 EHR-extractable quality indicators. By doing so, this study aims to improve the data completeness of the EHR, paving the way for future quality assessments.

Trial registration

ClinicalTrials.gov NCT04388228. Registered on May 14, 2020.

Supplementary Information

The online version contains supplementary material available at 10.1186/s13063-021-05259-9.

The Value of New Fields in the Medical Record for Quality Improvement

Introduction

Quality in healthcare delivery is important for the safety and experience of patients with cancer. Effective documentation is an integral component of quality improvment, and accurate documentation can be affected by prompts in the medical record, potentially improving quality of services.

Methods

The Contemporary Oncology Team (COT) is a Greek private oncology practice that participated in the American Society of Clinical Oncology's (ASCO's) Quality in Oncology Practice Initiative (QOPI). Between 2014 and 2019, COT implemented changes in its paper patient medical record, in order to improve quality of care and documentation. Fields regarding pain, emotional well-being and psychosocial assessment, discussions with the patient and consent about treatment and disease, medication details and cumulative dose, treatment goals, side-effect grading, pregnancy screening, treatment adherence and anticipated duration were added. In this report, we present the association of these improvements with COT performance in QOPI.

Results

Pain and emotional well-being assessment and documentation were significantly improved by the development of a structured patient follow-up form. In contrast, the assessment of fertility issues, tobacco use, and the documentation of treatment plan and intent did not present a drastic change, because COT performance was already above QOPI average.

Conclusion

A thorough reform of COT paper medical record according to QOPI standards improved QOPI scores, but more importantly effected a shift in the team's culture to safer and more standardized quality based care.

A Scoping Review of Process Indicators for Measuring Quality of Care in Glaucoma

PRCIS

There are no standardized process quality indicators (QIs) in glaucoma care. Although they can be inferred from guidelines and trials, they should be designed and standardized to allow better assessment of the quality of care.

PURPOSE

QIs are crucial for assessing the performance of any health care system. To allow efficiency, effectiveness, and patient-centeredness, there is a need for prompt acquisition of up-to-date information. Among the available QIs, process indicators have the highest sensitivity to frequent changes and could better reflect the implementation outcomes of novel ideas and technology. This study aimed to map the available information regarding process QIs in glaucoma care, identify the current development stage of these indicators, and systematically synthesize them.

MATERIALS AND METHODS

We performed a scoping review of 4 electronic bibliographic databases for studies reporting on process QIs in glaucoma. We retrieved 7502 references and created a domain list reflecting the core idea underlying each indicator.

RESULTS

We summarized information from 18 documents and listed 20 domains. The most mentioned domains were follow-up, optic nerve head assessment, visual field test, and intraocular pressure. Indicators regarding the quality of life assessment, patient assistance, or presence of written protocols were less frequently mentioned.

CONCLUSIONS

There are notable variations among process QIs in glaucoma and significant heterogeneity in their descriptions in published studies. Although novel indicators can be inferred from guidelines and trials, they should be designed and standardized for better assessment of performance in health systems to improve their quality.

Dhavileswarapu AVL, Naik N, Karimi H, Hegde P, Rai BP, Somani BK. Big Data Analytics in urology: the story so far and the road ahead

Artificial intelligence (AI) has a proven record of application in the field of medicine and is used in various urological conditions such as oncology, urolithiasis, paediatric urology, urogynaecology, infertility and reconstruction. Data is the driving force of AI and the past decades have undoubtedly witnessed an upsurge in healthcare data. Urology is a specialty that has always been at the forefront of innovation and research and has rapidly embraced technologies to improve patient outcomes and experience. Advancements made in Big Data Analytics raised the expectations about the future of urology. This review aims to investigate the role of big data and its blend with AI for trends and use in urology. We explore the different sources of big data in urology and explicate their current and future applications. A positive trend has been exhibited by the advent and implementation of AI in urology with data available from several databases. The extensive use of big data for the diagnosis and treatment of urological disorders is still in its early stage and under validation. In future however, big data will no doubt play a major role in the management of urological conditions.

Implementing standardized provider documentation in a tertiary epilepsy clinic

OBJECTIVE

To incorporate standardized documentation into an epilepsy clinic and to use these standardized data to compare patients' perception of epilepsy diagnosis to provider documentation.

METHODS

Using quality improvement methodology, we implemented interventions to increase documentation of epilepsy diagnosis, seizure frequency, and type from 49.8% to 70% of adult nonemployee patients seen by 6 providers over 5 months of routine clinical care. The main intervention consisted of an interactive SmartPhrase that mirrored a documentation template developed by the Epilepsy Learning Healthcare System. We assessed the weekly proportion of complete SmartPhrases among eligible patient encounters with a statistical process control chart. We used a subset of patients with established epilepsy care linked to existing patient-reported survey data to examine the proportion of patient-to-provider agreement on epilepsy diagnosis (yes vs no/unsure). We also examined sociodemographic and clinical characteristics of patients who disagreed vs agreed with provider's documentation of epilepsy diagnosis.

RESULTS

The median SmartPhrase weekly completion rate was 78%. Established patients disagreed with providers with respect to epilepsy diagnosis in 18.5% of encounters (κ = 0.13), indicating that they did not have or were unsure if they had epilepsy despite having a provider-documented epilepsy diagnosis. Patients who disagreed with providers were similar to those who agreed with respect to age, sex, ethnicity, marital status, seizure frequency, type, and other quality-of-life measures.

CONCLUSION

This project supports the feasibility of implementing standardized documentation of data relevant to epilepsy care in a tertiary epilepsy clinic and highlights an opportunity for improvement in patient-provider communication.

Feasibility of reusing routinely recorded data to monitor the safe preparation and administration of injectable medication: A multicenter cross-sectional study

BACKGROUND

Reusing routinely recorded data from electronic hospital records (EHR) may offer a less-time consuming, and more real time alternative for monitoring compliance by nurses with a protocol for the safe preparation and administration of injectable medication. However, at present it is unknown if the data necessary to calculate the quality indicators (QIs) are recorded in EHRs, or if these data are suitable for automated QI calculation. Therefore, the aim of this study was to determine the feasibility of monitoring compliance by nurses with a protocol for the safe injectable medication preparation and administration by reusing routinely recorded EHR data for the automated calculation of QIs.

METHODS

A cross-sectional study in 12 Dutch hospitals (October 2015-May 2016). The checks included in the currently prevailing national protocol for the safe preparation and administration of injectable medication were translated into 16 data elements required to calculate the QIs. At each hospital, one interview was conducted using a structured questionnaire to decide whether the data elements were available in EHRs. To present these results, descriptive statistics were used.

RESULTS

In total, 20 health-care professionals were interviewed and four different EHR systems were evaluated. The availability of data elements was comparable between the four evaluated EHR systems. Nine of the 16 required data elements were recorded in EHRs, eight in a structured format. The seven missing data elements were mainly related to checks such as 'gather all materials needed' or 'conduct hand hygiene'. Furthermore, changes were identified in the process for the preparation and administration of injectable medication. These changes are mostly related to the increased use of electronic medication administration registration and barcode medication administration systems.

CONCLUSIONS

Reusing EHR data to monitor compliance by nurses with the currently prevailing protocol for the safe preparation and administration of injectable medication is not entirely feasible. A decision should be made on which checks should be recorded in the EHRs and which checks should be audited in order to minimize the registration burden for nurses. Moreover, the currently prevailing protocol should be revised to bring it in line with work-as-done. Our results can be used as guidance for such a revision and also for designing new QIs that can be calculated by reusing routinely recorded EHR data.

Trading patients' choice in providers for quality of maternity care? A discrete choice experiment amongst pregnant women

BACKGROUND

The introduction of bundled payment for maternity care, aimed at improving the quality of maternity care, may affect pregnant women's choice in providers of maternity care. This paper describes a Dutch study which examined pregnant women's preferences when choosing a maternity care provider. The study focused on factors that enhance the quality of maternity care versus (restricted) provider choice.

METHODS

A discrete choice experiment was conducted amongst 611 pregnant women living in the Netherlands using an online questionnaire. The data were analysed with Latent Class Analyses. The outcome measure consisted of stated preferences in the discrete choice experiment. Included factors were: information exchange by care providers through electronic medical records, information provided by midwife, information provided by friends, freedom to choose maternity care provider and travel distance.

RESULTS

Four different preference structures were found. In two of those structures, respondents found aspects of the maternity care related to quality of care more important than being able to choose a provider (provider choice). In the two other preference structures, respondents found provider choice more important than aspects related to quality of maternity care.

CONCLUSIONS

In a country with presumed high-quality maternity care like the Netherlands, about half of pregnant women prefer being able to choose their maternity care provider over organisational factors that might imply better quality of care. A comparable amount of women find quality-related aspects most important when choosing a maternity care provider and are willing to accept limitations in their choice of provider. These insights are relevant for policy makers in order to be able to design a bundled payment model which justify the preferences of all pregnant women.

Development of quality indicators for type 2 diabetes, extractable from the electronic health record of the general physician. A rand-modified Delphi method

AIMS

Type 2 diabetes mellitus is a worldwide cause of significant morbidity and mortality. The general physician (GP) plays an important role in managing this disease and the use of the electronic health record (EHR) can improve quality for type 2 diabetes care. We aimed to develop a set of evidence-based and EHR extractable quality indicators for type 2 diabetes, enabling an automated quality assessment.

METHODS

We used a Rand-modified Delphi method. Recommendations were selected from (inter)national guidelines using the 'SMART' principle and scored by a multidisciplinary expert panel. After analysis of the median score, prioritization and consensus, recommendations were discussed during a consensus meeting. A final validation round resulted in a core set of recommendations, which were transformed into quality indicators.

RESULTS

A total of 101 recommendations originating from 10 (inter)national guidelines were presented to the expert panel, which resulted in a core set of 50 recommendations that were merged and modified into 36 recommendations after the consensus meeting. The panel added 3 recommendations. This resulted in a final set of 39 quality indicators.

CONCLUSIONS

Our study presents a set of 39 quality indicators for type 2 diabetes in primary care that are EHR extractable, enabling automated quality assessment.

Development and Pilot Testing of Quality Indicators for Primary Care in Japan

Introduction:

To the best of our knowledge, no quality indicators (QIs) for primary care provided by local clinics have yet been developed in Japan. We aimed to develop valid and applicable QIs to evaluate primary care in Japan.

Methods:

Two focus group interviews were held to identify conceptual categories. Existing indicators for these categories were identified, and initial sets of potential QIs were developed. Using a modified Delphi appropriateness method, a multidisciplinary expert panel then developed and selected the QIs. Feasibility and applicability of these QIs were then confirmed in pilot testing at six local clinics in Hokkaido, Japan. To determine patient acceptance of these quality improvement activities, the survey asked two questions, “Do you think it is preferable that the patients of this clinic be periodically surveyed?” and “Do you think it is preferable that this clinic periodically undergo an external quality review by an independent body?”

Results:

Seven categories emerged from the focus group discussions as key components of primary care in Japan. Thirty-nine QIs under five categories (Comprehensive care/Standardized care, Access, Communication, Co-ordination, and Understanding of patient background) were finally selected and named the QIs for Primary Care Practice in Japan. In pilot testing at six primary care clinics in 2015, 65.4% of patients answered favorably to the idea that clinics should conduct regular patient surveys, and 71.8% answered favorably to the idea that clinics should undergo periodic external quality review by an independent body.

Conclusions:

We developed QIs to assess primary care services provided by clinics in Japan, for the first time. Although further refinement is required, establishment of these QIs is the first step in quality improvement for primary care practices in Japan.

Primary Care Practices' Ability to Report Electronic Clinical Quality Measures in the EvidenceNOW Southwest Initiative to Improve Heart Health

IMPORTANCE

The capability and capacity of primary care practices to report electronic clinical quality measures (eCQMs) are questionable.

OBJECTIVE

To determine how quickly primary care practices can report eCQMs and the practice characteristics associated with faster reporting.

DESIGN, SETTING, AND PARTICIPANTS

This quality improvement study examined an initiative (EvidenceNOW Southwest) to enhance primary care practices' ability to adopt evidence-based cardiovascular care approaches: aspirin prescribing, blood pressure control, cholesterol management, and smoking cessation (ABCS). A total of 211 primary care practices in Colorado and New Mexico participating in EvidenceNOW Southwest between February 2015 and December 2017 were included.

INTERVENTIONS

Practices were instructed on eCQM specifications that could be produced by an electronic health record, a registry, or a third-party platform. Practices received 9 months of support from a practice facilitator, a clinical health information technology advisor, and the research team. Practices were instructed to report their baseline ABCS eCQMs as soon as possible.

MAIN OUTCOMES AND MEASURES

The main outcome was time to report the ABCS eCQMs. Cox proportional hazards models were used to examine practice characteristics associated with time to reporting.

RESULTS

Practices were predominantly clinician owned (48%) and in urban or suburban areas (71%). Practices required a median (interquartile range) of 8.2 (4.6-11.9) months to report any ABCS eCQM. Time to report differed by eCQM: practices reported blood pressure management the fastest (median [interquartile range], 7.8 [3.5-10.4] months) and cholesterol management the slowest (median [interquartile range], 10.5 [6.6 to >12] months) (log-rank P < .001). In multivariable models, the blood pressure eCQM was reported more quickly by practices that participated in accountable care organizations (hazard ratio [HR], 1.88; 95% CI, 1.40-2.53; P < .001) or participated in a quality demonstration program (HR, 1.58; 95% CI, 1.14-2.18; P = .006). The cholesterol eCQM was reported more quickly by practices that used clinical guidelines for cardiovascular disease management (HR, 1.35; 95% CI, 1.18-1.53; P < .001). Compared with Federally Qualified Health Centers, hospital-owned practices had greater ability to report blood pressure eCQMs (HR, 2.66; 95% CI, 95% CI, 1.73-4.09; P < .001), and clinician-owned practices had less ability to report cholesterol eCQMs (HR, 0.52; 95% CI, 0.35-0.76; P < .001).

CONCLUSIONS AND RELEVANCE

In this study, time to report eCQMs varied by measure and practice type, with very few practices reporting quickly. Practices took longer to report a new cholesterol measure than other measures. Programs that require eCQM reporting should consider the time and effort practices must exert to produce reports. Practices may benefit from additional support to succeed in new programs that require eCQM reporting.

Development of Quality Indicators for the Care of Women with Abnormal Uterine Bleeding by Primary Care Providers in the Veterans Health Administration

BACKGROUND

Abnormal uterine bleeding (AUB) is a common women's health complaint. However, the quality of primary care (PC) management of AUB is unknown. Our objective was to develop quality indicators for Veterans Health Administration (VA) PC assessment and management of AUB.

METHODS

We drafted candidate indicators based on comprehensive review of the scientific literature, including published consensus guidelines. Then, we convened a national panel of nine experts including PC providers, obstetrician-gynecologists, VA policy stakeholders, and quality measurement experts, and used a modified Delphi panel process. First, panelists individually rated 19 candidate indicators, using 9-point scales, on three metrics: consistency with established guidelines, importance to women's health, and reliability of measurement from VA electronic health records. Panelists then discussed the indicators. Finally, panelists re-rated revised indicators using the same metrics. Indicators were selected if final median ratings were ≥7 on each 9-point scale, without dispersion in ratings.

RESULTS

Eighteen indicators were selected. Three focused on assessing need for emergency care (e.g., profuse bleeding or pregnancy). Three addressed ascertaining key aspects of the medical history (e.g., endometrial cancer risk). Two addressed performing a physical examination (e.g., pelvic examination). Six addressed indications for diagnostic studies and specialty care referrals, (e.g., transvaginal ultrasound examination). Four addressed initiation of treatment and counseling (e.g., hormone therapy).

CONCLUSIONS

We developed quality indicators for PC assessment and management of AUB that span reproductive and postmenopausal life phases. Applying these indicators in VA and other health systems with integrated electronic health records can assess need for, and effects of, AUB quality improvement programs.

In Data We Trust? Comparison of Electronic Versus Manual Abstraction of Antimicrobial Prescribing Quality Metrics for Hospitalized Veterans With Pneumonia

BACKGROUND

Electronic health records provide the opportunity to assess system-wide quality measures. Veterans Affairs Pharmacy Benefits Management Center for Medication Safety uses medication use evaluation (MUE) through manual review of the electronic health records.

OBJECTIVE

To compare an electronic MUE approach versus human/manual review for extraction of antibiotic use (choice and duration) and severity metrics.

RESEARCH DESIGN

Retrospective.

SUBJECTS

Hospitalizations for uncomplicated pneumonia occurring during 2013 at 30 Veterans Affairs facilities.

MEASURES

We compared summary statistics, individual hospitalization-level agreement, facility-level consistency, and patterns of variation between electronic and manual MUE for initial severity, antibiotic choice, daily clinical stability, and antibiotic duration.

RESULTS

Among 2004 hospitalizations, electronic and manual abstraction methods showed high individual hospitalization-level agreement for initial severity measures (agreement=86%-98%, κ=0.5-0.82), antibiotic choice (agreement=89%-100%, κ=0.70-0.94), and facility-level consistency for empiric antibiotic choice (anti-MRSA r=0.97, P<0.001; antipseudomonal r=0.95, P<0.001) and therapy duration (r=0.77, P<0.001) but lower facility-level consistency for days to clinical stability (r=0.52, P=0.006) or excessive duration of therapy (r=0.55, P=0.005). Both methods identified widespread facility-level variation in antibiotic choice, but we found additional variation in manual estimation of excessive antibiotic duration and initial illness severity.

CONCLUSIONS

Electronic and manual MUE agreed well for illness severity, antibiotic choice, and duration of therapy in pneumonia at both the individual and facility levels. Manual MUE showed additional reviewer-level variation in estimation of initial illness severity and excessive antibiotic use. Electronic MUE allows for reliable, scalable tracking of national patterns of antimicrobial use, enabling the examination of system-wide interventions to improve quality.

Primary Care Practices' Abilities And Challenges In Using Electronic Health Record Data For Quality Improvement

Federal value-based payment programs require primary care practices to conduct quality improvement activities, informed by the electronic reports on clinical quality measures that their electronic health records (EHRs) generate. To determine whether EHRs produce reports adequate to the task, we examined survey responses from 1,492 practices across twelve states, supplemented with qualitative data. Meaningful-use participation, which requires the use of a federally certified EHR, was associated with the ability to generate reports-but the reports did not necessarily support quality improvement initiatives. Practices reported numerous challenges in generating adequate reports, such as difficulty manipulating and aligning measurement time frames with quality improvement needs, lack of functionality for generating reports on electronic clinical quality measures at different levels, discordance between clinical guidelines and measures available in reports, questionable data quality, and vendors that were unreceptive to changing EHR configuration beyond federal requirements. The current state of EHR measurement functionality may be insufficient to support federal initiatives that tie payment to clinical quality measures.

Exploring completeness in clinical data research networks with DQe-c

Objective

To provide an open source, interoperable, and scalable data quality assessment tool for evaluation and visualization of completeness and conformance in electronic health record (EHR) data repositories.

Materials and Methods

This article describes the tool’s design and architecture and gives an overview of its outputs using a sample dataset of 200 000 randomly selected patient records with an encounter since January 1, 2010, extracted from the Research Patient Data Registry (RPDR) at Partners HealthCare. All the code and instructions to run the tool and interpret its results are provided in the Supplementary Appendix.

Results

DQ^e-c produces a web-based report that summarizes data completeness and conformance in a given EHR data repository through descriptive graphics and tables. Results from running the tool on the sample RPDR data are organized into 4 sections: load and test details, completeness test, data model conformance test, and test of missingness in key clinical indicators.

Discussion

Open science, interoperability across major clinical informatics platforms, and scalability to large databases are key design considerations for DQ^e-c. Iterative implementation of the tool across different institutions directed us to improve the scalability and interoperability of the tool and find ways to facilitate local setup.

Conclusion

EHR data quality assessment has been hampered by implementation of ad hoc processes. The architecture and implementation of DQ^e-c offer valuable insights for developing reproducible and scalable data science tools to assess, manage, and process data in clinical data repositories.

Coordination of Care Around Surgery for Colon Cancer: Insights From National Patterns of Physician Encounters With Medicare Beneficiaries

PURPOSE

To improve care coordination for complex cancers, it is critical to establish a more nuanced understanding of the types of providers involved. As the number of provider types increases, strategies to support cancer care coordination must adapt to a greater variety of information needs, communication styles, and treatment strategies.

METHODS

We categorized providers into 11 types, using National Provider Identifier specialties. Using Medicare claims, we counted the number of unique combinations of provider types billed during preoperative, operative, and postdischarge care for colon cancer surgery and assessed how this count varies across hospitals. The study included 70,567 beneficiaries in fee-for-service Medicare A and B for 6 months before and 60 days after an admission for colectomy for colon cancer between 2008 and 2011.

RESULTS

We observed 1,554 preoperative provider-type combinations, 975 operative combinations, and 1,571 postdischarge combinations. The three most common combinations in the preoperative phase were general medicine only, other medical specialists only, and general medicine and other medical specialists. In the operative phase, the three most common combinations were primary surgery, anesthesiology, and pathology; general medicine, other medical specialists, radiology, primary surgery, anesthesiology, and pathology; and other medical specialists, radiology, primary surgery, anesthesiology, and pathology. In the postdischarge phase, the three most common combinations were general medicine, general medicine and other medical specialists, and general medicine and oncology. On average, each hospital had 15 preoperative, 11 operative, and 15 postoperative combinations. High-volume, larger, teaching, urban, and noncritical access hospitals had more combinations in all phases.

CONCLUSION

Many provider-type combinations are involved in colon cancer surgery care. Substantial variation exists across hospitals types, suggesting that certain hospitals need additional resources and more flexible infrastructure to coordinate care.

Accurate Measurement In California's Safety-Net Health Systems Has Gaps And Barriers

Patient safety in ambulatory care has not been routinely measured. California implemented a pay-for-performance program in safety-net hospitals that incentivized measurement and improvement in key areas of ambulatory safety: referral completion, medication safety, and test follow-up. We present two years of program data (collected during July 2015-June 2017) and show both suboptimal performance in aspects of ambulatory safety and questionable reliability in data reporting. Performance was better in areas that required limited coordination or patient engagement-for example, annual medication monitoring versus follow-up after high-risk mammograms. Health care systems that lack seamlessly integrated electronic health records and patient registries encountered barriers to reporting reliable ambulatory safety data, particularly for measures that integrated multiple data elements. These data challenges precluded accurate performance measurement in many areas. Policy makers and safety advocates need to support the development of information systems and measures that facilitate the accurate ascertainment of the health systems, patients, and clinical tasks at greatest risk for ambulatory safety failures.

Role of Nursing Informatics in the Automation of Pneumonia Quality Measure Data Elements

Core measures are standard metrics to reflect the processes of care provided by hospitals. Hospitals in the United States are expected to extract data from electronic health records, automated computation of core measures, and electronic submission of the quality measures data. Traditional manual calculation processes are time intensive and susceptible to error. Automated calculation has the potential to provide timely, accurate information, which could guide quality-of-care decisions, but this vision has yet to be achieved. In this study, nursing informaticists and data analysts implemented a method to automatically extract data elements from electronic health records to calculate a core measure. We analyzed the sensitivity, specificity, and accuracy of core measure data elements extracted via SQL query and compared the results to manually extracted data elements. This method achieved excellent performance for the structured data elements but was less efficient for semistructured and unstructured elements. We analyzed challenges in automating the calculation of quality measures and proposed a rule-based (hybrid) approach for semistructured and unstructured data elements.

Errors in Electronic Health Record-Based Data Query of Statin Prescriptions in Patients With Coronary Artery Disease in a Large, Academic, Multispecialty Clinic Practice

BACKGROUND

With the recent implementation of the Medicare Quality Payment Program, providers face increasing accountability for delivering high-quality care. Such pay-for-performance programs aim to leverage systematic data captured by electronic health record (EHR) systems to measure performance; however, the fidelity of EHR query for assessing performance has not been validated compared with manual chart review. We sought to determine whether our institution's methodology of EHR query could accurately identify cases in which providers failed to prescribe statins for eligible patients with coronary artery disease.

METHODS AND RESULTS

A total of 9459 patients with coronary artery disease were seen at least twice at the Emory Clinic between July 2014 and June 2015, of whom 1338 (14.1%, 95% confidence interval 13.5-14.9%) had no statin prescription or exemption per EHR query. A total of 120 patient cases were randomly selected and reviewed by 2 physicians for further adjudication. Of the 120 cases initially classified as statin prescription failures, only 21 (17.5%; 95% confidence interval, 11.7-25.3%) represented true failure following physician review.

CONCLUSIONS

Sole reliance on EHR data query to measure quality metrics may lead to significant errors in assessing provider performance. Institutions should be cognizant of these potential sources of error, provide support to medical providers, and form collaborative data management teams to promote and improve meaningful use of EHRs. We propose actionable steps to improve the accuracy of EHR data query that require hypothesis testing and prospective validation in future studies.

Is It Feasible to Use Electronic Health Records for Quality Measurement of Adolescent Care?

OBJECTIVE

To determine the extent to which it is feasible to implement quality measures on electronic health records (EHRs) as currently implemented in pediatric health centers.

METHODS

A survey of information technology professionals at 10 institutions that provide primary care services to adolescents. The survey asked whether data about care was being captured electronically across the nine domains relevant to adolescent well care: Screening, Health Risks, Sexual Health, Diagnosis and History, Laboratory Results, Prescriptions, Referrals, Forms Management, and Patient Demographics. For each domain, we developed a scale of the extent to which the EHR makes quality measurement feasible.

RESULTS

Overall feasibility scores varied across centers from 34% to 85% and from 53% to 80% across care domains. One centre reported 100% feasibility for 8 of 10 care domains.

CONCLUSIONS

Electronic health records can facilitate quality improvement, but the feasibility of such use depends on the presence, validity, and accessibility of the quality data in the EHR. Even among the largest and most sophisticated pediatric EHR systems, quality of care measurement is not possible yet for all aspects of adolescent well care without manual effort to review and code data. Nevertheless, almost all quality measures were reported to be feasible in some systems.

Challenges in the use of the mental health information system in a resource-limited setting: lessons from Ghana

BACKGROUND

One of the most successful modes of record-keeping and data collection is the use of health management information systems, where patient information and management plans are uniformly entered into a database to streamline the information and for ease of further patient management. For mental healthcare, a Mental Health Information System (MHIS) has been found most successful since a properly established and operational MHIS is helpful for developing equitable and appropriate mental health care systems. Until 2010, the system of keeping patient records and information in the Accra Psychiatric Hospital of Ghana was old and outdated. In light of this and other factors, a complete reforming of the mental health information systems in three psychiatric hospitals in Ghana was undertaken in 2010. Four years after its implementation, we explored user experiences with the new system, and report here the challenges that were identified with use of the new MHIS.

METHODS

Individual semi-structured interviews were conducted with nine clinical and administrative staff of the Accra Psychiatric Hospital to examine their experiences with the new MHIS. Participants in the study were in three categories: clinical staff, administrator, and records clerk. Participants' knowledge of the system and its use, as well as the challenges they had experienced in its use were explored using an interpretative phenomenological approach.

RESULTS

The data suggest that optimal use of the current MHIS had faced significant implementation challenges in a number of areas. Central challenges reported by users included increased workload, poor staff involvement and training, and absence of logistic support to keep the system running.

CONCLUSIONS

Setting up a new system does not guarantee its success. As important as it is to have a mental health information system, its usefulness is largely dependent on proper implementation and maintenance. Further, the system can facilitate policy transformation only when the place of mental health in district, regional and national health discourse improves.

The Use of Clinical Registries in the United States: A Landscape Survey

Introduction

The use of information from clinical registries for improvement and value-based payment is increasing, yet information about registry use is not widely available. We conducted a landscape survey to understand registry uses, focus areas and challenges. The survey addressed the structure and organization of registry programs, as well as their purpose and scope.

Setting

The survey was conducted by the National Quality Registry Network (NQRN), a community of organizations interested in registries. NQRN is a program of the PCPI, a national convener of medical specialty and professional societies and associations, which constitute a majority of registry stewards in the United States.

Methods

We surveyed 152 societies and associations, asking about registry programs, governance, number of registries, purpose and data uses, data collection, expenses, funding and interoperability.

Results

The response rate was 52 percent. Many registries were self-funded, with 39 percent spending less than $1 million per year, and 32 percent spending $1-9.9 million. The typical registry had three full-time equivalent staff. Registries were frequently used for quality improvement, benchmarking and clinical decision support. 85 percent captured outpatient data. Most registries collected demographics, treatments, practitioner information and comorbidities; 53 percent captured patient-reported outcomes. 88 percent used manual data entry and 18 percent linked to external secondary data sources. Cost, interoperability and vendor management were barriers to continued registry development.

Conclusions

Registries captured data across a broad scope, audited data quality using multiple techniques, and used a mix of automated and manual data capture methods. Registry interoperability was still a challenge, even among registries using nationally accepted data standards.

DQe-v: A Database-Agnostic Framework for Exploring Variability in Electronic Health Record Data Across Time and Site Location

Data variability is a commonly observed phenomenon in Electronic Health Records (EHR) data networks. A common question asked in scientific investigations of EHR data is whether the cross-site and -time variability reflects an underlying data quality error at one or more contributing sites versus actual differences driven by various idiosyncrasies in the healthcare settings. Although research analysts and data scientists have commonly used various statistical methods to detect and account for variability in analytic datasets, self service tools to facilitate exploring cross-organizational variability in EHR data warehouses are lacking and could benefit from meaningful data visualizations. DQ^e-v, an interactive, database-agnostic tool for visually exploring variability in EHR data provides such a solution. DQ^e-v is built on an open source platform, R statistical software, with annotated scripts and a readme document that makes it fully reproducible. To illustrate and describe functionality of DQ^e-v, we describe the DQ^e-v’s readme document which includes a complete guide to installation, running the program, and interpretation of the outputs. We also provide annotated R scripts and an example dataset as supplemental materials. DQ^e-v offers a self service tool to visually explore data variability within EHR datasets irrespective of the data model. GitHub and CIELO offer hosting and distribution of the tool and can facilitate collaboration across any interested community of users as we target improving usability, efficiency, and interoperability.

A novel use of the discrete templated notes within an electronic health record software to monitor resident supervision

Objective

Monitoring the supervision of residents can be a challenging task. We describe our experience with the implementation of a templated note system for documenting procedures with the aim of enabling automated, discrete, and standardized capture of documentation of supervision of residents performing floor-based procedures, with minimal extra effort from the residents.

Materials and methods

Procedural note templates were designed using the standard existing template within a commercial electronic health record software. Templates for common procedures were created such that residents could document every procedure performed outside of the formal procedural areas. Automated reports were generated and letters were sent to noncompliers.

Results

A total of 27 045 inpatient non-formal procedural area procedures were recorded from August 2012 to June 2014. Compliance with NoteWriter template usage averaged 86% in the first year and increased to 94.6% in the second year ( P  = .0055).

Initially, only 12.5% of residents documented supervision of any form. By the end of the first year, this was above 80%, with the gains maintained into the second year and beyond. Direct supervision was documented to have occurred where required in 62.8% in the first year and increased to 99.8% in the second year ( P  = .0001) after the addition of hard stops. Notification of attendings prior to procedures was documented 100% of the time by September 2013. Letters sent to errant residents decreased from 3.6 to 0.83 per 100 residents per week.

Conclusion

The templated procedure note system with hard stops and integrated reporting can successfully be used to improve monitoring of resident supervision. This has potential impact on resident education and patient safety.

The effect of referral templates on out-patient quality of care in a hospital setting: a cluster randomized controlled trial

BACKGROUND

The assessment of quality of care is an integral part of modern medicine. The referral represents the handing over of care from the general practitioner to the specialist. This study aimed to assess whether an improved referral could lead to improved quality of care.

METHODS

A cluster randomized trial with the general practitioner surgery as the clustering unit was performed. Fourteen surgeries in the area surrounding the University Hospital of North Norway Harstad were randomized stratified by town versus countryside location. The intervention consisted of implementing referral templates for new referrals in four clinical areas: dyspepsia; suspected colorectal cancer; chest pain; and confirmed or suspected chronic obstructive pulmonary disease. The control group followed standard referral practice. Quality of treatment pathway as assessed by newly developed quality indicators was used as main outcome. Secondary outcomes included subjective quality assessment, positive predictive value of referral and adequacy of prioritization. Assessment of outcomes was done at the individual level. The patients, hospital doctors and outcome assessors were blinded to the intervention status.

RESULTS

A total of 500 patients were included, with 281 in the intervention and 219 in the control arm. From the multilevel regression model the effect of the intervention on the quality indicator score was insignificant at 1.80% (95% CI, -1.46 to 5.06, p = 0.280). No significant differences between the intervention and the control groups were seen in the secondary outcomes. Active use of the referral intervention was low, estimated at approximately 50%. There was also wide variation in outcome scoring between the different assessors.

CONCLUSIONS

In this study no measurable effect on quality of care or prioritization was revealed after implementation of referral templates at the general practitioner/hospital interface. The results were hindered by a limited uptake of the intervention at GP surgeries and inconsistencies in outcome assessment.

TRIAL REGISTRATION

The study was registered under registration number NCT01470963 on September 5th, 2011.

Leveraging Electronic Health Care Record Information to Measure Pressure Ulcer Risk in Veterans With Spinal Cord Injury: A Longitudinal Study Protocol

Background

Pressure ulcers (PrUs) are a frequent, serious, and costly complication for veterans with spinal cord injury (SCI). The health care team should periodically identify PrU risk, although there is no tool in the literature that has been found to be reliable, valid, and sensitive enough to assess risk in this vulnerable population.

Objective

The immediate goal is to develop a risk assessment model that validly estimates the probability of developing a PrU. The long-term goal is to assist veterans with SCI and their providers in preventing PrUs through an automated system of risk assessment integrated into the veteran’s electronic health record (EHR).

Methods

This 5-year longitudinal, retrospective, cohort study targets 12,344 veterans with SCI who were cared for in the Veterans Health Administration (VHA) in fiscal year (FY) 2009 and had no record of a PrU in the prior 12 months. Potential risk factors identified in the literature were reviewed by an expert panel that prioritized factors and determined if these were found in structured data or unstructured form in narrative clinical notes for FY 2009-2013. These data are from the VHA enterprise Corporate Data Warehouse that is derived from the EHR structured (ie, coded in database/table) or narrative (ie, text in clinical notes) data for FY 2009-2013.

Results

This study is ongoing and final results are expected in 2017. Thus far, the expert panel reviewed the initial list of risk factors extracted from the literature; the panel recommended additions and omissions and provided insights about the format in which the documentation of the risk factors might exist in the EHR. This list was then iteratively refined through review and discussed with individual experts in the field. The cohort for the study was then identified, and all structured, unstructured, and semistructured data were extracted. Annotation schemas were developed, samples of documents were extracted, and annotations are ongoing. Operational definitions of structured data elements have been created and steps to create an analytic dataset are underway.

Conclusions

To our knowledge, this is the largest cohort employed to identify PrU risk factors in the United States. It also represents the first time natural language processing and statistical text mining will be used to expand the number of variables available for analysis. A major strength of this quantitative study is that all VHA SCI centers were included in the analysis, reducing potential for selection bias and providing increased power for complex statistical analyses. This longitudinal study will eventually result in a risk prediction tool to assess PrU risk that is reliable and valid, and that is sensitive to this vulnerable population.

The Use of Evidence-Based, Problem-Oriented Templates as a Clinical Decision Support in an Inpatient Electronic Health Record System

BACKGROUND

The integration of clinical decision support (CDS) in documentation practices remains limited due to obstacles in provider workflows and design restrictions in electronic health records (EHRs). The use of electronic problem-oriented templates (POTs) as a CDS has been previously discussed but not widely studied.

OBJECTIVE

We evaluated the voluntary use of evidence-based POTs as a CDS on documentation practices.

METHODS

This was a randomized cohort (before and after) study of Hospitalist Attendings in an Academic Medical Center using EPIC EHRs. Primary Outcome measurement was note quality, assessed by the 9-item Physician Documentation Quality Instrument (PDQI-9). Secondary Outcome measurement was physician efficiency, assessed by the total charting time per note.

RESULTS

Use of POTs increased the quality of note documentation [score 37.5 vs. 39.0, P = 0.0020]. The benefits of POTs scaled with use; the greatest improvement in note quality was found in notes using three or more POTs [score 40.2, P = 0.0262]. There was no significant difference in total charting time [30 minutes vs. 27 minutes, P = 0.42].

CONCLUSION

Use of evidence-based and problem-oriented templates is associated with improved note quality without significant change in total charting time. It can be used as an effective CDS during note documentation.

Variation in Screening Abnormality Rates and Follow-Up of Breast, Cervical and Colorectal Cancer Screening within the PROSPR Consortium

BACKGROUND

Primary care providers and health systems have prominent roles in guiding effective cancer screening.

OBJECTIVE

To characterize variation in screening abnormality rates and timely initial follow-up for common cancer screening tests.

DESIGN

Population-based cohort undergoing screening in 2011, 2012, or 2013 at seven research centers comprising the National Cancer Institute-sponsored Population-based Research Optimizing Screening through Personalized Regimens (PROSPR) consortium.

PARTICIPANTS

Adults undergoing mammography with or without digital breast tomosynthesis (n = 97,683 ages 40-75 years), fecal occult blood or fecal immunochemical tests (n = 759,553 ages 50-75 years), or Papanicolaou with or without human papillomavirus tests (n = 167,330 ages 21-65 years).

INTERVENTION

Breast, colorectal, or cervical cancer screening.

MAIN MEASURES

Abnormality rates per 1000 screens; percentage with timely initial follow-up (within 90 days, except 9-month window for BI-RADS 3). Primary care clinic-level variation in percentage with screening abnormality and percentage with timely initial follow-up.

KEY RESULTS

There were 10,248/97,683 (104.9 per 1000) abnormal breast cancer screens, 35,847/759,553 (47.2 per 1000) FOBT/FIT-positive colorectal cancer screens, and 13,266/167,330 (79.3 per 1000) abnormal cervical cancer screens. The percentage with timely follow-up was 93.2 to 96.7 % for breast centers, 46.8 to 68.7  % for colorectal centers, and 46.6 % for the cervical cancer screening center (low-grade squamous intraepithelial lesions or higher). The primary care clinic variation (25th to 75th percentile) was smaller for the percentage with an abnormal screen (breast, 8.5-10.3 %; colorectal, 3.0-4.8 %; cervical, 6.3-9.9 %) than for the percentage with follow-up within 90 days (breast, 90.2-95.8 %; colorectal, 43.4-52.0 %; cervical, 29.6-61.4 %).

CONCLUSIONS

Variation in both the rate of screening abnormalities and their initial follow-up was evident across organ sites and primary care clinics. This highlights an opportunity for improving the delivery of cancer screening through focused study of patient, provider, clinic, and health system characteristics associated with timely follow-up of screening abnormalities.

Guiding Principles for Data Architecture to Support the Pathways Community HUB Model

INTRODUCTION

The Pathways Community HUB Model provides a unique strategy to effectively supplement health care services with social services needed to overcome barriers for those most at risk of poor health outcomes. Pathways are standardized measurement tools used to define and track health and social issues from identification through to a measurable completion point. The HUB use Pathways to coordinate agencies and service providers in the community to eliminate the inefficiencies and duplication that exist among them.

PATHWAYS COMMUNITY HUB MODEL AND FORMALIZATION

Experience with the Model has brought out the need for better information technology solutions to support implementation of the Pathways themselves through decision-support tools for care coordinators and other users to track activities and outcomes, and to facilitate reporting. Here we provide a basis for discussing recommendations for such a data infrastructure by developing a conceptual model that formalizes the Pathway concept underlying current implementations.

REQUIREMENTS FOR DATA ARCHITECTURE TO SUPPORT THE PATHWAYS COMMUNITY HUB MODEL

The main contribution is a set of core recommendations as a framework for developing and implementing a data architecture to support implementation of the Pathways Community HUB Model. The objective is to present a tool for communities interested in adopting the Model to learn from and to adapt in their own development and implementation efforts.

PROBLEMS WITH QUALITY OF DATA EXTRACTED FROM THE CHAP DATABASE

Experience with the Community Health Access Project (CHAP) data base system (the core implementation of the Model) has identified several issues and remedies that have been developed to address these issues. Based on analysis of issues and remedies, we present several key features for a data architecture meeting the just mentioned recommendations.

IMPLEMENTATION OF FEATURES

Presentation of features is followed by a practical guide to their implementation allowing an organization to consider either tailoring off-the-shelf generic systems to meet the requirements or offerings that are specialized for community-based care coordination.

DISCUSSION

Looking to future extensions, we discuss the utility and prospects for an ontology to include care coordination in the Unified Medical Language System (UMLS) of the National Library of Medicine and other existing medical and nursing taxonomies.

CONCLUSIONS AND RECOMMENDATIONS

Pathways structures are an important principle, not only for organizing the care coordination activities, but also for structuring the data stored in electronic form in the conduct of such care. We showed how the proposed architecture encourages design of effective decision support systems for coordinated care and suggested how interested organizations can set about acquiring such systems. Although the presentation focuses on the Pathways Community HUB Model, the principles for data architecture are stated in generic form and are applicable to any health information system for improving care coordination services and population health.

The impact of electronic health records on healthcare quality: a systematic review and meta-analysis

OBJECTIVE

To assess the impact of electronic health record (EHR) on healthcare quality, we hence carried out a systematic review and meta-analysis of published studies on this topic.

METHODS

PubMed, Web of Knowledge, Scopus and Cochrane Library databases were searched to identify studies that investigated the association between the EHR implementation and process or outcome indicators. Two reviewers screened identified citations and extracted data according to the PRISMA guidelines. Meta-analysis was performed using the random effects model for each indicator. Heterogeneity was quantified using the Cochran Q test and I2 statistics, and publication bias was assessed using the Egger's test.

RESULTS

Of the 23 398 citations identified, 47 articles were included in the analysis. Meta-analysis showed an association between EHR use and a reduced documentation time with a difference in mean of -22.4% [95% confidence interval (CI) = -38.8 to -6.0%; P < 0.007]. EHR resulted also associated with a higher guideline adherence with a risk ratio (RR) of 1.33 (95% CI = 1.01 to 1.76; P = 0.049) and a lower number of medication errors with an overall RR of 0.46 (95% CI = 0.38 to 0.55; P < 0.001), and adverse drug effects (ADEs) with an overall RR of 0.66 (95% CI = 0.44 to 0.99; P = 0.045). No association with mortality was evident (P = 0.936). High heterogeneity among the studies was evident. Publication bias was not evident.

CONCLUSIONS

EHR system, when properly implemented, can improve the quality of healthcare, increasing time efficiency and guideline adherence and reducing medication errors and ADEs. Strategies for EHR implementation should be therefore recommended and promoted.

The Challenges of Data Quality Evaluation in a Joint Data Warehouse

Introduction:

The use of clinically derived data from electronic health records (EHRs) and other electronic clinical systems can greatly facilitate clinical research as well as operational and quality initiatives. One approach for making these data available is to incorporate data from different sources into a joint data warehouse. When using such a data warehouse, it is important to understand the quality of the data. The primary objective of this study was to determine the completeness and concordance of common types of clinical data available in the Knowledge Program (KP) joint data warehouse, which contains feeds from several electronic systems including the EHR.

Methods:

A manual review was performed of specific data elements for 250 patients from an EHR, and these were compared with corresponding elements in the KP data warehouse. Completeness and concordance were calculated for five categories of data including demographics, vital signs, laboratory results, diagnoses, and medications.

Results:

In general, data elements for demographics, vital signs, diagnoses, and laboratory results were present in more cases in the source EHR compared to the KP. When data elements were available in both sources, there was a high concordance. In contrast, the KP data warehouse documented a higher prevalence of deaths and medications compared to the EHR.

Discussion:

Several factors contributed to the discrepancies between data in the KP and the EHR—including the start date and frequency of data feeds updates into the KP, inability to transfer data located in nonstructured formats (e.g., free text or scanned documents), as well as incomplete and missing data variables in the source EHR.

Conclusion:

When evaluating the quality of a data warehouse with multiple data sources, assessing completeness and concordance between data set and source data may be better than designating one to be a gold standard. This will allow the user to optimize the method and timing of data transfer in order to capture data with better accuracy.

Visualizing Anomalies in Electronic Health Record Data: The Variability Explorer Tool

As Electronic Health Record (EHR) systems are becoming more prevalent in the U.S. health care domain, the utility of EHR data in translational research and clinical decision-making gains prominence. Leveraging primay· care-based. multi-clinic EHR data, this paper introduces a web-based visualization tool, the Variability Explorer Tool (VET), to assist researchers with profiling variability among diagnosis codes. VET applies a simple statistical method to approximate probability distribution functions for the prevalence of any given diagnosis codes to visualize between-clinic and across-year variability. In a depression diagnoses use case, VET outputs demonstrated substantial variability in code use. Even though data quality research often characterizes variability as an indicator for data quality, variability can also reflect real characteristics of data, such as practice-level, and patient-level issues. Researchers benefit from recognizing variability in early stages of research to improve their research design and ensure validity and generalizability of research findings.

Future Networked Healthcare Systems

Future healthcare systems and organizations demand huge computational resources, and the ability for the applications to interact and communicate with each other, within and across organizational boundaries. This chapter aims to explore state-of-the-art of the healthcare landscape and presents an analysis of networked healthcare systems with a focus on networking traffic and architectures. To this end, the relevant technologies including networked healthcare architectures and performance studies, Health Level 7 (HL7), big data, and cloud computing, are reviewed. Subsequently, a study of healthcare systems, applications and traffic over local, metro, and wide area networks is presented using multi-hospital cross-continent scenarios. The network architectures for these systems are described. A detailed study to explore quality of service (QoS) performance for these healthcare systems with a range of applications, system sizes, and network sizes is presented. Conclusions are drawn regarding future healthcare systems and internet designs along with directions for future research.

A national clinical quality program for Veterans Affairs catheterization laboratories (from the Veterans Affairs clinical assessment, reporting, and tracking program)

A "learning health care system", as outlined in a recent Institute of Medicine report, harnesses real-time clinical data to continuously measure and improve clinical care. However, most current efforts to understand and improve the quality of care rely on retrospective chart abstractions complied long after the provision of clinical care. To align more closely with the goals of a learning health care system, we present the novel design and initial results of the Veterans Affairs (VA) Clinical Assessment, Reporting, and Tracking (CART) program-a national clinical quality program for VA cardiac catheterization laboratories that harnesses real-time clinical data to support clinical care and quality-monitoring efforts. Integrated within the VA electronic health record, the CART program uses a specialized software platform to collect real-time patient and procedural data for all VA patients undergoing coronary procedures in VA catheterization laboratories. The program began in 2005 and currently contains data on 434,967 catheterization laboratory procedures, including 272,097 coronary angiograms and 86,481 percutaneous coronary interventions, performed by 801 clinicians on 246,967 patients. We present the initial data from the CART program and describe 3 quality-monitoring programs that use its unique characteristics-procedural and complications feedback to individual labs, coronary device surveillance, and major adverse event peer review. The VA CART program is a novel approach to electronic health record design that supports clinical care, quality, and safety in VA catheterization laboratories. Its approach holds promise in achieving the goals of a learning health care system.

Assessing the role of a medication-indication resource in the treatment relation extraction from clinical text

OBJECTIVE

To evaluate the contribution of the MEDication Indication (MEDI) resource and SemRep for identifying treatment relations in clinical text.

MATERIALS AND METHODS

We first processed clinical documents with SemRep to extract the Unified Medical Language System (UMLS) concepts and the treatment relations between them. Then, we incorporated MEDI into a simple algorithm that identifies treatment relations between two concepts if they match a medication-indication pair in this resource. For a better coverage, we expanded MEDI using ontology relationships from RxNorm and UMLS Metathesaurus. We also developed two ensemble methods, which combined the predictions of SemRep and the MEDI algorithm. We evaluated our selected methods on two datasets, a Vanderbilt corpus of 6864 discharge summaries and the 2010 Informatics for Integrating Biology and the Bedside (i2b2)/Veteran's Affairs (VA) challenge dataset.

RESULTS

The Vanderbilt dataset included 958 manually annotated treatment relations. A double annotation was performed on 25% of relations with high agreement (Cohen's κ = 0.86). The evaluation consisted of comparing the manual annotated relations with the relations identified by SemRep, the MEDI algorithm, and the two ensemble methods. On the first dataset, the best F1-measure results achieved by the MEDI algorithm and the union of the two resources (78.7 and 80, respectively) were significantly higher than the SemRep results (72.3). On the second dataset, the MEDI algorithm achieved better precision and significantly lower recall values than the best system in the i2b2 challenge. The two systems obtained comparable F1-measure values on the subset of i2b2 relations with both arguments in MEDI.

CONCLUSIONS

Both SemRep and MEDI can be used to extract treatment relations from clinical text. Knowledge-based extraction with MEDI outperformed use of SemRep alone, but superior performance was achieved by integrating both systems. The integration of knowledge-based resources such as MEDI into information extraction systems such as SemRep and the i2b2 relation extractors may improve treatment relation extraction from clinical text.