Enhanced Quality Measurement Event Detection: An Application to Physician Reporting

Development and Evaluation of an Automated Protocol Recommendation System for Chest CT Using Natural Language Processing With CLEVER Terminology Word Replacement

Purpose: To evaluate the clinical performance of a Protocol Recommendation System (PRS) automatic protocolling of chest CT imaging requests. Materials and Methods: 322 387 consecutive historical imaging requests for chest CT between 2017 and 2022 were extracted from a radiology information system (RIS) database containing 16 associated patient information values. Records with missing fields and protocols with <100 occurrences were removed, leaving 18 protocols for training. After freetext pre-processing and applying CLEVER terminology word replacements, the features of a bag-of-words model were used to train a multinomial logistic regression classifier. Four readers protocolled 300 clinically executed protocols (CEP) based on all clinically available information. After their selection was made, the PRS and CEP were unblinded, and the readers were asked to score their agreement (1 = severe error, 2 = moderate error, 3 = disagreement but acceptable, 4 = agreement). The ground truth was established by the readers' majority selection, a judge helped break ties. For the PRS and CEP, the accuracy and clinical acceptability (scores 3 and 4) were calculated. The readers' protocolling reliability was measured using Fleiss' Kappa. Results: Four readers agreed on 203/300 protocols, 3 on 82/300 cases, and in 15 cases, a judge was needed. PRS errors were found by the 4 readers in 1%, 2.7%, 1%, and 0.7% of the cases, respectively. The accuracy/clinical acceptability of the PRS and CEP were 84.3%/98.6% and 83.0%/99.3%, respectively. The Fleiss' Kappa for all readers and all protocols was 0.805. Conclusion: The PRS achieved similar accuracy to human performance and may help radiologists master the ever-increasing workload.

Suicide theory-guided natural language processing of clinical progress notes to improve prediction of veteran suicide risk: protocol for a mixed-method study

INTRODUCTION

The state-of-the-art 3-step Theory of Suicide (3ST) describes why people consider suicide and who will act on their suicidal thoughts and attempt suicide. The central concepts of 3ST-psychological pain, hopelessness, connectedness, and capacity for suicide-are among the most important drivers of suicidal behaviour but they are missing from clinical suicide risk prediction models in use at the US Veterans Health Administration (VHA). These four concepts are not systematically recorded in structured fields of VHA's electronic healthcare records. Therefore, this study will develop a domain-specific ontology that will enable automated extraction of these concepts from clinical progress notes using natural language processing (NLP), and test whether NLP-based predictors for these concepts improve accuracy of existing VHA suicide risk prediction models.

METHODS AND ANALYSIS

Our mixed-method study has an exploratory sequential design where a qualitative component (aim 1) will inform quantitative analyses (aims 2 and 3). For aim 1, subject matter experts will manually annotate progress notes of clinical encounters with veterans who attempted or died by suicide to develop a domain-specific ontology for the 3ST concepts. During aim 2, we will use NLP to machine-annotate clinical progress notes and derive longitudinal representations for each patient with respect to the presence and intensity of hopelessness, psychological pain, connectedness and capacity for suicide in temporal proximity of suicide attempts and deaths by suicide. These longitudinal representations will be evaluated during aim 3 for their ability to improve existing VHA prediction models of suicide and suicide attempts, STORM (Stratification Tool for Opioid Risk Mitigation) and REACHVET (Recovery Engagement and Coordination for Health - Veterans Enhanced Treatment).

ETHICS AND DISSEMINATION

Ethics approval for this study was granted by the Stanford University Institutional Review Board and the Research and Development Committee of the VA Palo Alto Health Care System. Results of the study will be disseminated through several outlets, including peer-reviewed publications and presentations at national conferences.

Type 1 Diabetes Management With Technology: Patterns of Utilization and Effects on Glucose Control Using Real-World Evidence

This retrospective cohort study evaluated diabetes device utilization and the effectiveness of these devices for newly diagnosed type 1 diabetes. Investigators examined the use of continuous glucose monitoring (CGM) systems, self-monitoring of blood glucose (SMBG), continuous subcutaneous insulin infusion (CSII), and multiple daily injection (MDI) insulin regimens and their effects on A1C. The researchers identified 6,250 patients with type 1 diabetes, of whom 32% used CGM and 37.1% used CSII. A higher adoption rate of either CGM or CSII in newly diagnosed type 1 diabetes was noted among White patients and those with private health insurance. CGM users had lower A1C levels than nonusers (P = 0.039), whereas no difference was noted between CSII users and nonusers (P = 0.057). Furthermore, CGM use combined with CSII yielded lower A1C than MDI regimens plus SMBG (P <0.001).

Assessment of Extractability and Accuracy of Electronic Health Record Data for Joint Implant Registries

IMPORTANCE

Implant registries provide valuable information on the performance of implants in a real-world setting, yet they have traditionally been expensive to establish and maintain. Electronic health records (EHRs) are widely used and may include the information needed to generate clinically meaningful reports similar to a formal implant registry.

OBJECTIVES

To quantify the extractability and accuracy of registry-relevant data from the EHR and to assess the ability of these data to track trends in implant use and the durability of implants (hereafter referred to as implant survivorship), using data stored since 2000 in the EHR of the largest integrated health care system in the United States.

DESIGN, SETTING, AND PARTICIPANTS

Retrospective cohort study of a large EHR of veterans who had 45 351 total hip arthroplasty procedures in Veterans Health Administration hospitals from 2000 to 2017. Data analysis was performed from January 1, 2000, to December 31, 2017.

EXPOSURES

Total hip arthroplasty.

MAIN OUTCOMES AND MEASURES

Number of total hip arthroplasty procedures extracted from the EHR, trends in implant use, and relative survivorship of implants.

RESULTS

A total of 45 351 total hip arthroplasty procedures were identified from 2000 to 2017 with 192 805 implant parts. Data completeness improved over the time. After 2014, 85% of prosthetic heads, 91% of shells, 81% of stems, and 85% of liners used in the Veterans Health Administration health care system were identified by part number. Revision burden and trends in metal vs ceramic prosthetic femoral head use were found to reflect data from the American Joint Replacement Registry. Recalled implants were obvious negative outliers in implant survivorship using Kaplan-Meier curves.

CONCLUSIONS AND RELEVANCE

Although loss to follow-up remains a challenge that requires additional attention to improve the quantitative nature of calculated implant survivorship, we conclude that data collected during routine clinical care and stored in the EHR of a large health system over 18 years were sufficient to provide clinically meaningful data on trends in implant use and to identify poor implants that were subsequently recalled. This automated approach was low cost and had no reporting burden. This low-cost, low-overhead method to assess implant use and performance within a large health care setting may be useful to internal quality assurance programs and, on a larger scale, to postmarket surveillance of implant performance.

Leveraging Digital Data to Inform and Improve Quality Cancer Care

BACKGROUND

Efficient capture of routine clinical care and patient outcomes is needed at a population-level, as is evidence on important treatment-related side effects and their effect on well-being and clinical outcomes. The increasing availability of electronic health records (EHR) offers new opportunities to generate population-level patient-centered evidence on oncologic care that can better guide treatment decisions and patient-valued care.

METHODS

This study includes patients seeking care at an academic medical center, 2008 to 2018. Digital data sources are combined to address missingness, inaccuracy, and noise common to EHR data. Clinical concepts were identified and extracted from EHR unstructured data using natural language processing (NLP) and machine/deep learning techniques. All models are trained, tested, and validated on independent data samples using standard metrics.

RESULTS

We provide use cases for using EHR data to assess guideline adherence and quality measurements among patients with cancer. Pretreatment assessment was evaluated by guideline adherence and quality metrics for cancer staging metrics. Our studies in perioperative quality focused on medications administered and guideline adherence. Patient outcomes included treatment-related side effects and patient-reported outcomes.

CONCLUSIONS

Advanced technologies applied to EHRs present opportunities to advance population-level quality assessment, to learn from routinely collected clinical data for personalized treatment guidelines, and to augment epidemiologic and population health studies. The effective use of digital data can inform patient-valued care, quality initiatives, and policy guidelines.

IMPACT

A comprehensive set of health data analyzed with advanced technologies results in a unique resource that facilitates wide-ranging, innovative, and impactful research on prostate cancer. This work demonstrates new ways to use the EHRs and technology to advance epidemiologic studies and benefit oncologic care.See all articles in this CEBP Focus section, "Modernizing Population Science."

Quality Measures and Quality Improvement Initiatives in Osteoporosis-an Update

PURPOSE OF REVIEW

The aims of this review are to summarize current performance for osteoporosis quality measures used by Centers for Medicare and Medicaid (CMS) for pay-for-performance programs and to describe recent quality improvement strategies around these measures.

RECENT FINDINGS

Healthcare Effectiveness Data and Information (HEDIS) quality measures for the managed care population indicate gradual improvement in osteoporosis screening, osteoporosis identification and treatment following fragility fracture, and documentation of fall risk assessment and plan of care between 2006 and 2016. However, population-based studies suggest achievement for these process measures is lower where reporting is not mandated. Performance gaps remain, particularly for post-fracture care. Elderly patients with increased comorbidity are especially vulnerable to fractures, yet underperformance is documented in this population. Gender and racial disparities also exist. As has been shown for other areas of health care, education alone has a limited role as a quality improvement intervention. Multifactorial and systems-based interventions seem to be most successful in leading to measurable change for osteoporosis care and fall prevention. Despite increasing recognition of evidence-based quality measures for osteoporosis and incentives to improve upon performance for these measures, persistent gaps in care exist that will require further investigation into sustainable and value-adding quality improvement interventions.

Using natural language processing to construct a metastatic breast cancer cohort from linked cancer registry and electronic medical records data

Objectives

Most population-based cancer databases lack information on metastatic recurrence. Electronic medical records (EMR) and cancer registries contain complementary information on cancer diagnosis, treatment and outcome, yet are rarely used synergistically. To construct a cohort of metastatic breast cancer (MBC) patients, we applied natural language processing techniques within a semisupervised machine learning framework to linked EMR-California Cancer Registry (CCR) data.

Materials and Methods

We studied all female patients treated at Stanford Health Care with an incident breast cancer diagnosis from 2000 to 2014. Our database consisted of structured fields and unstructured free-text clinical notes from EMR, linked to CCR, a component of the Surveillance, Epidemiology and End Results Program (SEER). We identified de novo MBC patients from CCR and extracted information on distant recurrences from patient notes in EMR. Furthermore, we trained a regularized logistic regression model for recurrent MBC classification and evaluated its performance on a gold standard set of 146 patients.

Results

There were 11 459 breast cancer patients in total and the median follow-up time was 96.3 months. We identified 1886 MBC patients, 512 (27.1%) of whom were de novo MBC patients and 1374 (72.9%) were recurrent MBC patients. Our final MBC classifier achieved an area under the receiver operating characteristic curve (AUC) of 0.917, with sensitivity 0.861, specificity 0.878, and accuracy 0.870.

Discussion and Conclusion

To enable population-based research on MBC, we developed a framework for retrospective case detection combining EMR and CCR data. Our classifier achieved good AUC, sensitivity, and specificity without expert-labeled examples.

Extracting Patient-Centered Outcomes from Clinical Notes in Electronic Health Records: Assessment of Urinary Incontinence After Radical Prostatectomy

Objective:

To assess documentation of urinary incontinence (UI) in prostatectomy patients using unstructured clinical notes from Electronic Health Records (EHRs).

Methods:

We developed a weakly-supervised natural language processing tool to extract assessments, as recorded in unstructured text notes, of UI before and after radical prostatectomy in a single academic practice across multiple clinicians. Validation was carried out using a subset of patients who completed EPIC-26 surveys before and after surgery. The prevalence of UI as assessed by EHR and EPIC-26 was compared using repeated-measures ANOVA. The agreement of reported UI between EHR and EPIC-26 was evaluated using Cohen’s Kappa coefficient.

Results:

A total of 4870 patients and 716 surveys were included. Preoperative prevalence of UI was 12.7 percent. Postoperative prevalence was 71.8 percent at 3 months, 50.2 percent at 6 months and 34.4 and 41.8 at 12 and 24 months, respectively. Similar rates were recorded by physicians in the EHR, particularly for early follow-up. For all time points, the agreement between EPIC-26 and the EHR was moderate (all p < 0.001) and ranged from 86.7 percent agreement at baseline (Kappa = 0.48) to 76.4 percent agreement at 24 months postoperative (Kappa = 0.047).

Conclusions:

We have developed a tool to assess documentation of UI after prostatectomy using EHR clinical notes. Our results suggest such a tool can facilitate unbiased measurement of important PCOs using real-word data, which are routinely recorded in EHR unstructured clinician notes. Integrating PCO information into clinical decision support can help guide shared treatment decisions and promote patient-valued care.

Weakly supervised natural language processing for assessing patient-centered outcome following prostate cancer treatment

Background

The population-based assessment of patient-centered outcomes (PCOs) has been limited by the efficient and accurate collection of these data. Natural language processing (NLP) pipelines can determine whether a clinical note within an electronic medical record contains evidence on these data. We present and demonstrate the accuracy of an NLP pipeline that targets to assess the presence, absence, or risk discussion of two important PCOs following prostate cancer treatment: urinary incontinence (UI) and bowel dysfunction (BD).

Methods

We propose a weakly supervised NLP approach which annotates electronic medical record clinical notes without requiring manual chart review. A weighted function of neural word embedding was used to create a sentence-level vector representation of relevant expressions extracted from the clinical notes. Sentence vectors were used as input for a multinomial logistic model, with output being either presence, absence or risk discussion of UI/BD. The classifier was trained based on automated sentence annotation depending only on domain-specific dictionaries (weak supervision).

Results

The model achieved an average F1 score of 0.86 for the sentence-level, three-tier classification task (presence/absence/risk) in both UI and BD. The model also outperformed a pre-existing rule-based model for note-level annotation of UI with significant margin.

Conclusions

We demonstrate a machine learning method to categorize clinical notes based on important PCOs that trains a classifier on sentence vector representations labeled with a domain-specific dictionary, which eliminates the need for manual engineering of linguistic rules or manual chart review for extracting the PCOs. The weakly supervised NLP pipeline showed promising sensitivity and specificity for identifying important PCOs in unstructured clinical text notes compared to rule-based algorithms.

Quality improvement initiatives in rheumatology: an integrative review of the last 5 years

PURPOSE OF REVIEW

We reviewed recent quality improvement initiatives in the field of rheumatology to identify common strategies and themes leading to measurable change.

RECENT FINDINGS

Efforts to improve quality of care in rheumatology have accelerated in the last 5 years. Most studies in this area have focused on interventions to improve process measures such as increasing the collection of patient-reported outcomes and vaccination rates, but some studies have examined interventions to improve health outcomes. Increasingly, researchers are studying electronic health record (EHR)-based interventions, such as standardized templates, flowsheets, best practice alerts and order sets. EHR-based interventions were most successful when reinforced with provider education, reminders and performance feedback. Most studies also redesigned workflows, distributing tasks among clinical staff. Given the common challenges and solutions facing rheumatology clinics under new value-based payment models, there are important opportunities to accelerate quality improvement by building on the successful efforts to date. Structured quality improvement models such as the learning collaborative may help to disseminate successful initiatives across practices.

SUMMARY

Review of recent quality improvement initiatives in rheumatology demonstrated common solutions, particularly involving leveraging health IT and workflow redesign.

Automatic inference of BI-RADS final assessment categories from narrative mammography report findings

We propose an efficient natural language processing approach for inferring the BI-RADS final assessment categories by analyzing only the mammogram findings reported by the mammographer in narrative form. The proposed hybrid method integrates semantic term embedding with distributional semantics, producing a context-aware vector representation of unstructured mammography reports. A large corpus of unannotated mammography reports (300,000) was used to learn the context of the key-terms using a distributional semantics approach, and the trained model was applied to generate context-aware vector representations of the reports annotated with BI-RADS category (22,091). The vectorized reports were utilized to train a supervised classifier to derive the BI-RADS assessment class. Even though the majority of the proposed embedding pipeline is unsupervised, the classifier was able to recognize substantial semantic information for deriving the BI-RADS categorization not only on a holdout internal testset and also on an external validation set (1900 reports). Our proposed method outperforms a recently published domain-specific rule-based system and could be relevant for evaluating concordance between radiologists. With minimal requirement for task specific customization, the proposed method can be easily transferable to a different domain to support large scale text mining or derivation of patient phenotype.