Automated Extraction of VTE Events From Narrative Radiology Reports in Electronic Health Records: A Validation Study

Derivation and external validation of a portable method to identify patients with pulmonary embolism from radiology reports: The READ-PE algorithm

BACKGROUND

Identification of pulmonary embolism (PE) across a cohort currently requires burdensome manual review. Previous approaches to automate capture of PE diagnosis have either been too complex for widespread use or have lacked external validation. We sought to develop and validate the Regular Expression Aided Determination of PE (READ-PE) algorithm, which uses a portable text-matching approach to identify PE in reports from computed tomography with angiography (CTA).

METHODS

We identified derivation and validation cohorts of final radiology reports for CTAs obtained on adults (≥ 18 years) at two independent, quaternary academic emergency departments (EDs) in the United States. All reports were in the English language. We manually reviewed CTA reports for PE as a reference standard. In the derivation cohort, we developed the READ-PE algorithm by iteratively combining regular expressions to identify PE. We validated the READ-PE algorithm in an independent cohort, and compared performance against three prior algorithms with sensitivity, specificity, positive-predictive-value (PPV), negative-predictive-value (NPV), and the F1 score.

RESULTS

Among 2948 CTAs in the derivation cohort 10.8 % had PE and the READ-PE algorithm reached 93 % sensitivity, 99 % specificity, 94 % PPV, 99 % NPV, and 0.93 F1 score, compared to F1 scores ranging from 0.50 to 0.85 for three prior algorithms. Among 1206 CTAs in the validation cohort 9.2 % had PE and the algorithm had 98 % sensitivity, 98 % specificity, 85 % PPV, 100 % NPV, and 0.91 F1 score.

CONCLUSIONS

The externally validated READ-PE algorithm identifies PE in English-language reports from CTAs obtained in the ED with high accuracy. This algorithm may be used in the electronic health record to accurately identify PE for research or surveillance. If implemented at other EDs, it should first undergo local validation and may require maintenance over time.

Validation of a natural language processing algorithm using national reporting data to improve identification of anesthesia-related ADVerse evENTs: The "ADVENTURE" study

BACKGROUND

Reporting and analysis of adverse events (AE) is associated with improved health system learning, quality outcomes, and patient safety. Manual text analysis is time-consuming, costly, and prone to human errors. We aimed to demonstrate the feasibility of novel machine learning and natural language processing (NLP) approaches for early predictions of adverse events and provide input to direct quality improvement and patient safety initiatives.

METHODS

We used machine learning to analyze 9559 continuously reported AE by clinicians and healthcare systems to the French National Health accreditor (HAS) between January 1, 2009, and December 31, 2020 . We validated the labeling of 135,000 unique de-identified AE reports and determined the associations between different system's root causes and patient consequences. The model was validated by independent expert anesthesiologists.

RESULTS

The machine learning (ML) and Artificial Intelligence (AI) model trained on 9559 AE datasets accurately categorized 8800 (88%) of reported AE. The three most frequent AE types were "difficult orotracheal intubation" (16.9% of AE reports), "medication error" (10.5%), and "post-induction hypotension" (6.9%). The accuracy of the AI model reached 70.9% sensitivity, 96.6% specificity for "difficult intubation", 43.2% sensitivity, and 98.9% specificity for "medication error."

CONCLUSIONS

This unsupervised ML method provides an accurate, automated, AI-supported search algorithm that ranks and helps to understand complex risk patterns and has greater speed, precision, and clarity when compared to manual human data extraction. Machine learning and Natural language processing (NLP) models can effectively be used to process natural language AE reports and augment expert clinician input. This model can support clinical applications and methodological standards and used to better inform and enhance decision-making for improved risk management and patient safety.

TRIAL REGISTRATION

The study was approved by the ethics committee of the French Society of Anesthesiology (IRB 00010254-2020-20) and the CNIL (CNIL: 118 58 95) and the study was registered with ClinicalTrials.gov (NCT: NCT05185479).

Developing Validated Tools to Identify Pulmonary Embolism in Electronic Databases: Rationale and Design of the PE-EHR+ Study

BACKGROUND

 Contemporary pulmonary embolism (PE) research, in many cases, relies on data from electronic health records (EHRs) and administrative databases that use International Classification of Diseases (ICD) codes. Natural language processing (NLP) tools can be used for automated chart review and patient identification. However, there remains uncertainty with the validity of ICD-10 codes or NLP algorithms for patient identification.

METHODS

 The PE-EHR+ study has been designed to validate ICD-10 codes as Principal Discharge Diagnosis, or Secondary Discharge Diagnoses, as well as NLP tools set out in prior studies to identify patients with PE within EHRs. Manual chart review by two independent abstractors by predefined criteria will be the reference standard. Sensitivity, specificity, and positive and negative predictive values will be determined. We will assess the discriminatory function of code subgroups for intermediate- and high-risk PE. In addition, accuracy of NLP algorithms to identify PE from radiology reports will be assessed.

RESULTS

 A total of 1,734 patients from the Mass General Brigham health system have been identified. These include 578 with ICD-10 Principal Discharge Diagnosis codes for PE, 578 with codes in the secondary position, and 578 without PE codes during the index hospitalization. Patients within each group were selected randomly from the entire pool of patients at the Mass General Brigham health system. A smaller subset of patients will also be identified from the Yale-New Haven Health System. Data validation and analyses will be forthcoming.

CONCLUSIONS

 The PE-EHR+ study will help validate efficient tools for identification of patients with PE in EHRs, improving the reliability of efficient observational studies or randomized trials of patients with PE using electronic databases.

Natural Language Processing in a Clinical Decision Support System for the Identification of Venous Thromboembolism: Algorithm Development and Validation

BACKGROUND

It remains unknown whether capturing data from electronic health records (EHRs) using natural language processing (NLP) can improve venous thromboembolism (VTE) detection in different clinical settings.

OBJECTIVE

The aim of this study was to validate the NLP algorithm in a clinical decision support system for VTE risk assessment and integrated care (DeVTEcare) to identify VTEs from EHRs.

METHODS

All inpatients aged ≥18 years in the Sixth Medical Center of the Chinese People's Liberation Army General Hospital from January 1 to December 31, 2021, were included as the validation cohort. The sensitivity, specificity, positive and negative likelihood ratios (LR+ and LR-, respectively), area under the receiver operating characteristic curve (AUC), and F1-scores along with their 95% CIs were used to analyze the performance of the NLP tool, with manual review of medical records as the reference standard for detecting deep vein thrombosis (DVT) and pulmonary embolism (PE). The primary end point was the performance of the NLP approach embedded into the EHR for VTE identification. The secondary end points were the performances to identify VTE among different hospital departments with different VTE risks. Subgroup analyses were performed among age, sex, and the study season.

RESULTS

Among 30,152 patients (median age 56 [IQR 41-67] years; 14,247/30,152, 47.3% females), the prevalence of VTE, PE, and DVT was 2.1% (626/30,152), 0.6% (177/30,152), and 1.8% (532/30,152), respectively. The sensitivity, specificity, LR+, LR-, AUC, and F1-score of NLP-facilitated VTE detection were 89.9% (95% CI 87.3%-92.2%), 99.8% (95% CI 99.8%-99.9%), 483 (95% CI 370-629), 0.10 (95% CI 0.08-0.13), 0.95 (95% CI 0.94-0.96), and 0.90 (95% CI 0.90-0.91), respectively. Among departments of surgery, internal medicine, and intensive care units, the highest specificity (100% vs 99.7% vs 98.8%, respectively), LR+ (3202 vs 321 vs 77, respectively), and F1-score (0.95 vs 0.89 vs 0.92, respectively) were in the surgery department (all P<.001). Among low, intermediate, and high VTE risks in hospital departments, the low-risk department had the highest AUC (1.00 vs 0.94 vs 0.96, respectively) and F1-score (0.97 vs 0.90 vs 0.90, respectively) as well as the lowest LR- (0.00 vs 0.13 vs 0.08, respectively) (DeLong test for AUC; all P<.001). Subgroup analysis of the age, sex, and season demonstrated consistently good performance of VTE detection with >87% sensitivity and specificity and >89% AUC and F1-score. The NLP algorithm performed better among patients aged ≤65 years than among those aged >65 years (F1-score 0.93 vs 0.89, respectively; P<.001).

CONCLUSIONS

The NLP algorithm in our DeVTEcare identified VTE well across different clinical settings, especially in patients in surgery units, departments with low-risk VTE, and patients aged ≤65 years. This algorithm can help to inform accurate in-hospital VTE rates and enhance risk-classified VTE integrated care in future research.

IKAR: An Interdisciplinary Knowledge-Based Automatic Retrieval Method from Chinese Electronic Medical Record

To date, information retrieval methods in the medical field have mainly focused on English medical reports, but little work has studied Chinese electronic medical reports, especially in the field of obstetrics and gynecology. In this paper, a dataset of 180,000 complete Chinese ultrasound reports in obstetrics and gynecology was established and made publicly available. Based on the ultrasound reports in the dataset, a new information retrieval method (IKAR) is proposed to extract key information from the ultrasound reports and automatically generate the corresponding ultrasound diagnostic results. The model can both extract what is already in the report and analyze what is not in the report by inference. After applying the IKAR method to the dataset, it is proved that the method could achieve 89.38% accuracy, 91.09% recall, and 90.23% F-score. Moreover, the method achieves an F-score of over 90% on 50% of the 10 components of the report. This study provides a quality dataset for the field of electronic medical records and offers a reference for information retrieval methods in the field of obstetrics and gynecology or in other fields.

Understanding short-term transmission dynamics of methicillin-resistant Staphylococcus aureus in the patient room

OBJECTIVE

Little is known about the short-term dynamics of methicillin-resistant Staphylococcus aureus (MRSA) transmission between patients and their immediate environment. We conducted a real-life microbiological evaluation of environmental MRSA contamination in hospital rooms in relation to recent patient activity.

DESIGN

Observational pilot study.

SETTING

Two hospitals, hospital 1 in Zurich, Switzerland, and hospital 2 in Ann Arbor, Michigan, United States.

PATIENTS

Inpatients with MRSA colonization or infection.

METHODS

At baseline, the groin, axilla, nares, dominant hands of 10 patients and 6 environmental high-touch surfaces in their rooms were sampled. Cultures were then taken of the patient hand and high-touch surfaces 3 more times at 90-minute intervals. After each swabbing, patients' hands and surfaces were disinfected. Patient activity was assessed by interviews at hospital 1 and analysis of video footage at hospital 2. A contamination pressure score was created by multiplying the number of colonized body sites with the activity level of the patient.

RESULTS

In total, 10 patients colonized and/or infected with MRSA were enrolled; 40 hand samples and 240 environmental samples were collected. At baseline, 30% of hands and 20% of high-touch surfaces yielded MRSA. At follow-up intervals, 8 (27%) of 30 patient hands, and 10 (6%) of 180 of environmental sites were positive. Activity of the patient explained 7 of 10 environmental contaminations. Patients with higher contamination pressure score showed a trend toward higher environmental contamination.

CONCLUSION

Environmental MRSA contamination in patient rooms was highly dynamic and was likely driven by the patient's MRSA body colonization pattern and the patient activity.

Natural Language Processing: from Bedside to Everywhere

OBJECTIVES

Owing to the rapid progress of natural language processing (NLP), the role of NLP in the medical field has radically gained considerable attention from both NLP and medical informatics. Although numerous medical NLP papers are published annually, there is still a gap between basic NLP research and practical product development. This gap raises questions, such as what has medical NLP achieved in each medical field, and what is the burden for the practical use of NLP? This paper aims to clarify the above questions.

METHODS

We explore the literature on potential NLP products/services applied to various medical/clinical/healthcare areas.

RESULTS

This paper introduces clinical applications (bedside applications), in which we introduce the use of NLP for each clinical department, internal medicine, pre-surgery, post-surgery, oncology, radiology, pathology, psychiatry, rehabilitation, obstetrics, and gynecology. Also, we clarify technical problems to be addressed for encouraging bedside applications based on NLP.

CONCLUSIONS

These results contribute to discussions regarding potentially feasible NLP applications and highlight research gaps for future studies.

Exploring the Applicability of Using Natural Language Processing to Support Nationwide Venous Thromboembolism Surveillance: Model Evaluation Study

Background

Venous thromboembolism (VTE) is a preventable, common vascular disease that has been estimated to affect up to 900,000 people per year. It has been associated with risk factors such as recent surgery, cancer, and hospitalization. VTE surveillance for patient management and safety can be improved via natural language processing (NLP). NLP tools have the ability to access electronic medical records, identify patients that meet the VTE case definition, and subsequently enter the relevant information into a database for hospital review.

Objective

We aimed to evaluate the performance of a VTE identification model of IDEAL-X (Information and Data Extraction Using Adaptive Learning; Emory University)-an NLP tool-in automatically classifying cases of VTE by "reading" unstructured text from diagnostic imaging records collected from 2012 to 2014.

Methods

After accessing imaging records from pilot surveillance systems for VTE from Duke University and the University of Oklahoma Health Sciences Center (OUHSC), we used a VTE identification model of IDEAL-X to classify cases of VTE that had previously been manually classified. Experts reviewed the technicians' comments in each record to determine if a VTE event occurred. The performance measures calculated (with 95% CIs) were accuracy, sensitivity, specificity, and positive and negative predictive values. Chi-square tests of homogeneity were conducted to evaluate differences in performance measures by site, using a significance level of .05.

Results

The VTE model of IDEAL-X "read" 1591 records from Duke University and 1487 records from the OUHSC, for a total of 3078 records. The combined performance measures were 93.7% accuracy (95% CI 93.7%-93.8%), 96.3% sensitivity (95% CI 96.2%-96.4%), 92% specificity (95% CI 91.9%-92%), an 89.1% positive predictive value (95% CI 89%-89.2%), and a 97.3% negative predictive value (95% CI 97.3%-97.4%). The sensitivity was higher at Duke University (97.9%, 95% CI 97.8%-98%) than at the OUHSC (93.3%, 95% CI 93.1%-93.4%; P<.001), but the specificity was higher at the OUHSC (95.9%, 95% CI 95.8%-96%) than at Duke University (86.5%, 95% CI 86.4%-86.7%; P<.001).

Conclusions

The VTE model of IDEAL-X accurately classified cases of VTE from the pilot surveillance systems of two separate health systems in Durham, North Carolina, and Oklahoma City, Oklahoma. NLP is a promising tool for the design and implementation of an automated, cost-effective national surveillance system for VTE. Conducting public health surveillance at a national scale is important for measuring disease burden and the impact of prevention measures. We recommend additional studies to identify how integrating IDEAL-X in a medical record system could further automate the surveillance process.

Patient Safety Monitoring in Acute Care in a Decentralized National Health Care System: Conceptual Framework and Initial Set of Actionable Indicators

OBJECTIVES

Monitoring patient safety is critical for continuous quality improvement in acute care. We carried out a national project to identify a conceptual framework with core indicators that could be uniformly applied in the decentralized health system of Italy.

METHODS

We used key international references to identify a framework with a core list of indicators and data sources for calculation in 4 hospitals in the Lombardy region. Two different data processing methods were applied: (a) centralized analysis of national databases and (b) decentralized data extraction and calculation using different hospital data available in Lombardy.

RESULTS

Agreement was reached on a conceptual framework for patient safety monitoring in acute care, including structures, processes, and outcomes as vertical dimensions and health care needs as horizontal axes. We were able to compute 15 of 32 indicators through the application of a range of methods. The calculation of indicators using national databases was based on international standards. The consistency of the estimates obtained through the use of different methods and data sources seemed limited.

CONCLUSIONS

We successfully identified a conceptual framework for patient safety in acute care including actionable indicators that can be calculated routinely using different data sources at national, regional, and hospital levels. Further work is required to compare methods and understand whether a combination of strategies at national and local levels could be proven effective.

Natural Language Processing of Large-Scale Structured Radiology Reports to Identify Oncologic Patients With or Without Splenomegaly Over a 10-Year Period

PURPOSE

To assess the accuracy of a natural language processing (NLP) model in extracting splenomegaly described in patients with cancer in structured computed tomography radiology reports.

METHODS

In this retrospective study between July 2009 and April 2019, 3,87,359 consecutive structured radiology reports for computed tomography scans of the chest, abdomen, and pelvis from 91,665 patients spanning 30 types of cancer were included. A randomized sample of 2,022 reports from patients with colorectal cancer, hepatobiliary cancer (HB), leukemia, Hodgkin lymphoma (HL), and non-HL patients was manually annotated as positive or negative for splenomegaly. NLP model training/testing was performed on 1,617/405 reports, and a new validation set of 400 reports from all cancer subtypes was used to test NLP model accuracy, precision, and recall. Overall survival was compared between the patient groups (with and without splenomegaly) using Kaplan-Meier curves.

RESULTS

The final cohort included 3,87,359 reports from 91,665 patients (mean age 60.8 years; 51.2% women). In the testing set, the model achieved accuracy of 92.1%, precision of 92.2%, and recall of 92.1% for splenomegaly. In the validation set, accuracy, precision, and recall were 93.8%, 92.9%, and 86.7%, respectively. In the entire cohort, splenomegaly was most frequent in patients with leukemia (32.5%), HB (17.4%), non-HL (9.1%), colorectal cancer (8.5%), and HL (5.6%). A splenomegaly label was associated with an increased risk of mortality in the entire cohort (hazard ratio 2.10; 95% CI, 1.98 to 2.22; P < .001).

CONCLUSION

Automated splenomegaly labeling by NLP of radiology report demonstrates good accuracy, precision, and recall. Splenomegaly is most frequently reported in patients with leukemia, followed by patients with HB.

Prediction and Diagnosis of Venous Thromboembolism Using Artificial Intelligence Approaches: A Systematic Review and Meta-Analysis

Venous thromboembolism (VTE) is a fatal disease and has become a burden on the global health system. Recent studies have suggested that artificial intelligence (AI) could be used to make a diagnosis and predict venous thrombosis more accurately. Thus, we performed a meta-analysis to better evaluate the performance of AI in the prediction and diagnosis of venous thrombosis. PubMed, Web of Science, and EMBASE were used to identify relevant studies. Of the 741 studies, 12 met the inclusion criteria and were included in the meta-analysis. Among them, 5 studies included a training set and test set, and 7 studies included only a training set. In the training set, the pooled sensitivity was 0.87 (95% CI 0.79-0.92), the pooled specificity was 0.95 (95% CI 0.89-0.97), and the area under the summary receiver operating characteristic (SROC) curve was 0.97 (95% CI 0.95-0.98). In the test set, the pooled sensitivity was 0.87 (95% CI 0.74-0.93), the pooled specificity was 0.96 (95% CI 0.79-0.99), and the area under the SROC curve was 0.98 (95% CI 0.97-0.99). The combined results remained significant in the subgroup analyzes, which included venous thrombosis type, AI type, model type (diagnosis/prediction), and whether the period was perioperative. In conclusion, AI may aid in the diagnosis and prediction of venous thrombosis, demonstrating high sensitivity, specificity and area under the SROC curve values. Thus, AI has important clinical value.

Natural Language Processing Performance for the Identification of Venous Thromboembolism in an Integrated Healthcare System

Real-time identification of venous thromboembolism (VTE), defined as deep vein thrombosis (DVT) and pulmonary embolism (PE), can inform a healthcare organization's understanding of these events and be used to improve care. In a former publication, we reported the performance of an electronic medical record (EMR) interrogation tool that employs natural language processing (NLP) of imaging studies for the diagnosis of venous thromboembolism. Because we transitioned from the legacy electronic medical record to the Cerner product, iCentra, we now report the operating characteristics of the NLP EMR interrogation tool in the new EMR environment. Two hundred randomly selected patient encounters for which the imaging report assessed by NLP that revealed VTE was present were reviewed. These included one hundred imaging studies for which PE was identified. These included computed tomography pulmonary angiography-CTPA, ventilation perfusion-V/Q scan, and CT angiography of the chest/ abdomen/pelvis. One hundred randomly selected comprehensive ultrasound (CUS) that identified DVT were also obtained. For comparison, one hundred patient encounters in which PE was suspected and imaging was negative for PE (CTPA or V/Q) and 100 cases of suspected DVT with negative CUS as reported by NLP were also selected. Manual chart review of the 400 charts was performed and we report the sensitivity, specificity, positive and negative predictive values of NLP compared with manual chart review. NLP and manual review agreed on the presence of PE in 99 of 100 cases, the presence of DVT in 96 of 100 cases, the absence of PE in 99 of 100 cases and the absence of DVT in all 100 cases. When compared with manual chart review, NLP interrogation of CUS, CTPA, CT angiography of the chest, and V/Q scan yielded a sensitivity = 93.3%, specificity = 99.6%, positive predictive value = 97.1%, and negative predictive value = 99%.

Clinical information extraction for preterm birth risk prediction

This paper contributes to the pursuit of leveraging unstructured medical notes to structured clinical decision making. In particular, we present a pipeline for clinical information extraction from medical notes related to preterm birth, and discuss the main challenges as well as its potential for clinical practice. A large collection of medical notes, created by staff during hospitalizations of patients who were at risk of delivering preterm, was gathered and analyzed. Based on an annotated collection of notes, we trained and evaluated information extraction components to discover clinical entities such as symptoms, events, anatomical sites and procedures, as well as attributes linked to these clinical entities. In a retrospective study, we show that these are highly informative for clinical decision support models that are trained to predict whether delivery is likely to occur within specific time windows, in combination with structured information from electronic health records.

Deep Learning for Natural Language Processing in Urology: State-of-the-Art Automated Extraction of Detailed Pathologic Prostate Cancer Data From Narratively Written Electronic Health Records

PURPOSE

Entering all information from narrative documentation for clinical research into databases is time consuming, costly, and nearly impossible. Even high-volume databases do not cover all patient characteristics and drawn results may be limited. A new viable automated solution is machine learning based on deep neural networks applied to natural language processing (NLP), extracting detailed information from narratively written (eg, pathologic radical prostatectomy [RP]) electronic health records (EHRs).

METHODS

Within an RP pathologic database, 3,679 RP EHRs were randomly split into 70% training and 30% test data sets. Training EHRs were automatically annotated, providing a semiautomatically annotated corpus of narratively written pathologic reports with initially context-free gold standard encodings. Primary and secondary Gleason pattern, corresponding percentages, tumor stage, nodal stage, total volume, tumor volume and diameter, and surgical margin were variables of interest. Second, state-of-the-art NLP techniques were used to train an industry-standard language model for pathologic EHRs by transfer learning. Finally, accuracy of the named entity extractors was compared with the gold standard encodings.

RESULTS

Agreement rates (95% confidence interval) for primary and secondary Gleason patterns each were 91.3% (89.4 to 93.0), corresponding to the following: Gleason percentages, 70.5% (67.6 to 73.3) and 80.9% (78.4 to 83.3); tumor stage, 99.3% (98.6 to 99.7); nodal stage, 98.7% (97.8 to 99.3); total volume, 98.3% (97.3 to 99.0); tumor volume, 93.3% (91.6 to 94.8); maximum diameter, 96.3% (94.9 to 97.3); and surgical margin, 98.7% (97.8 to 99.3). Cumulative agreement was 91.3%.

CONCLUSION

Our proposed NLP pipeline offers new abilities for precise and efficient data management from narrative documentation for clinical research. The scalable approach potentially allows the NLP pipeline to be generalized to other genitourinary EHRs, tumor entities, and other medical disciplines.

Evidence-based Clinical Decision Support Systems for the prediction and detection of three disease states in critical care: A systematic literature review

Background: Clinical decision support (CDS) systems have emerged as tools providing intelligent decision making to address challenges of critical care. CDS systems can be based on existing guidelines or best practices; and can also utilize machine learning to provide a diagnosis, recommendation, or therapy course. Methods: This research aimed to identify evidence-based study designs and outcome measures to determine the clinical effectiveness of clinical decision support systems in the detection and prediction of hemodynamic instability, respiratory distress, and infection within critical care settings. PubMed, ClinicalTrials.gov and Cochrane Database of Systematic Reviews were systematically searched to identify primary research published in English between 2013 and 2018. Studies conducted in the USA, Canada, UK, Germany and France with more than 10 participants per arm were included. Results: In studies on hemodynamic instability, the prediction and management of septic shock were the most researched topics followed by the early prediction of heart failure. For respiratory distress, the most popular topics were pneumonia detection and prediction followed by pulmonary embolisms. Given the importance of imaging and clinical notes, this area combined Machine Learning with image analysis and natural language processing. In studies on infection, the most researched areas were the detection, prediction, and management of sepsis, surgical site infections, as well as acute kidney injury. Overall, a variety of Machine Learning algorithms were utilized frequently, particularly support vector machines, boosting techniques, random forest classifiers and neural networks. Sensitivity, specificity, and ROC AUC were the most frequently reported performance measures. Conclusion: This review showed an increasing use of Machine Learning for CDS in all three areas. Large datasets are required for training these algorithms; making it imperative to appropriately address, challenges such as class imbalance, correct labelling of data and missing data. Recommendations are formulated for the development and successful adoption of CDS systems.

Evidence-based Clinical Decision Support Systems for the prediction and detection of three disease states in critical care: A systematic literature review

Background: Clinical decision support (CDS) systems have emerged as tools providing intelligent decision making to address challenges of critical care. CDS systems can be based on existing guidelines or best practices; and can also utilize machine learning to provide a diagnosis, recommendation, or therapy course. Methods: This research aimed to identify evidence-based study designs and outcome measures to determine the clinical effectiveness of clinical decision support systems in the detection and prediction of hemodynamic instability, respiratory distress, and infection within critical care settings. PubMed, ClinicalTrials.gov and Cochrane Database of Systematic Reviews were systematically searched to identify primary research published in English between 2013 and 2018. Studies conducted in the USA, Canada, UK, Germany and France with more than 10 participants per arm were included. Results: In studies on hemodynamic instability, the prediction and management of septic shock were the most researched topics followed by the early prediction of heart failure. For respiratory distress, the most popular topics were pneumonia detection and prediction followed by pulmonary embolisms. Given the importance of imaging and clinical notes, this area combined Machine Learning with image analysis and natural language processing. In studies on infection, the most researched areas were the detection, prediction, and management of sepsis, surgical site infections, as well as acute kidney injury. Overall, a variety of Machine Learning algorithms were utilized frequently, particularly support vector machines, boosting techniques, random forest classifiers and neural networks. Sensitivity, specificity, and ROC AUC were the most frequently reported performance measures. Conclusion: This review showed an increasing use of Machine Learning for CDS in all three areas. Large datasets are required for training these algorithms; making it imperative to appropriately address, challenges such as class imbalance, correct labelling of data and missing data. Recommendations are formulated for the development and successful adoption of CDS systems.

A systematic review of natural language processing for classification tasks in the field of incident reporting and adverse event analysis

CONTEXT

Adverse events in healthcare are often collated in incident reports which contain unstructured free text. Learning from these events may improve patient safety. Natural language processing (NLP) uses computational techniques to interrogate free text, reducing the human workload associated with its analysis. There is growing interest in applying NLP to patient safety, but the evidence in the field has not been summarised and evaluated to date.

OBJECTIVE

To perform a systematic literature review and narrative synthesis to describe and evaluate NLP methods for classification of incident reports and adverse events in healthcare.

METHODS

Data sources included Medline, Embase, The Cochrane Library, CINAHL, MIDIRS, ISI Web of Science, SciELO, Google Scholar, PROSPERO, hand searching of key articles, and OpenGrey. Data items were manually abstracted to a standardised extraction form.

RESULTS

From 428 articles screened for eligibility, 35 met the inclusion criteria of using NLP to perform a classification task on incident reports, or with the aim of detecting adverse events. The majority of studies used free text from incident reporting systems or electronic health records. Models were typically designed to classify by type of incident, type of medication error, or harm severity. A broad range of NLP techniques are demonstrated to perform these classification tasks with favourable performance outcomes. There are methodological challenges in how these results can be interpreted in a broader context.

CONCLUSION

NLP can generate meaningful information from unstructured data in the specific domain of the classification of incident reports and adverse events. Understanding what or why incidents are occurring is important in adverse event analysis. If NLP enables these insights to be drawn from larger datasets it may improve the learning from adverse events in healthcare.

Design and Implementation of a Comprehensive Surveillance System for Venous Thromboembolism in a Defined Region Using Electronic and Manual Approaches

BACKGROUND

Systematic surveillance for venous thromboembolism (VTE) in the United States has been recommended by several organizations. Despite adoption of electronic medical records (EMRs) by most health care providers and facilities, however, systematic surveillance for VTE is not available.

OBJECTIVES

This article develops a comprehensive, population-based surveillance strategy for VTE in a defined geographical region.

METHODS

The primary surveillance strategy combined computerized searches of the EMR with a manual review of imaging data at the Duke University Health System in Durham County, North Carolina, United States. Different strategies of searching the EMR were explored. Consolidation of results with autopsy reports (nonsearchable in the EMR) and with results from the Durham Veterans' Administration Medical Center was performed to provide a comprehensive report of new VTE from the defined region over a 2-year timeframe.

RESULTS

Monthly searches of the primary EMR missed a significant number of patients with new VTE who were identified by a separate manual search of radiology records, apparently related to delays in data entry and coding into the EMR. Comprehensive searches incorporating a location-restricted strategy were incomplete due to the assigned residence reflecting the current address and not the address at the time of event. The most comprehensive strategy omitted the geographic restriction step and identified all patients with VTE followed by manual review of individual records to remove incorrect entries (e.g., outside the surveillance time period or geographic location; no evidence for VTE). Consolidation of results from the EMR searches with results from autopsy reports and the separate facility identified additional patients not diagnosed within the Duke system.

CONCLUSION

We identified several challenges with implementing a comprehensive VTE surveillance program that could limit accuracy of the results. Improved electronic strategies are needed to cross-reference patients across multiple health systems and to minimize the need for manual review and confirmation of results.

Natural Language Processing of Clinical Notes on Chronic Diseases: Systematic Review

BACKGROUND

Novel approaches that complement and go beyond evidence-based medicine are required in the domain of chronic diseases, given the growing incidence of such conditions on the worldwide population. A promising avenue is the secondary use of electronic health records (EHRs), where patient data are analyzed to conduct clinical and translational research. Methods based on machine learning to process EHRs are resulting in improved understanding of patient clinical trajectories and chronic disease risk prediction, creating a unique opportunity to derive previously unknown clinical insights. However, a wealth of clinical histories remains locked behind clinical narratives in free-form text. Consequently, unlocking the full potential of EHR data is contingent on the development of natural language processing (NLP) methods to automatically transform clinical text into structured clinical data that can guide clinical decisions and potentially delay or prevent disease onset.

OBJECTIVE

The goal of the research was to provide a comprehensive overview of the development and uptake of NLP methods applied to free-text clinical notes related to chronic diseases, including the investigation of challenges faced by NLP methodologies in understanding clinical narratives.

METHODS

Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines were followed and searches were conducted in 5 databases using "clinical notes," "natural language processing," and "chronic disease" and their variations as keywords to maximize coverage of the articles.

RESULTS

Of the 2652 articles considered, 106 met the inclusion criteria. Review of the included papers resulted in identification of 43 chronic diseases, which were then further classified into 10 disease categories using the International Classification of Diseases, 10th Revision. The majority of studies focused on diseases of the circulatory system (n=38) while endocrine and metabolic diseases were fewest (n=14). This was due to the structure of clinical records related to metabolic diseases, which typically contain much more structured data, compared with medical records for diseases of the circulatory system, which focus more on unstructured data and consequently have seen a stronger focus of NLP. The review has shown that there is a significant increase in the use of machine learning methods compared to rule-based approaches; however, deep learning methods remain emergent (n=3). Consequently, the majority of works focus on classification of disease phenotype with only a handful of papers addressing extraction of comorbidities from the free text or integration of clinical notes with structured data. There is a notable use of relatively simple methods, such as shallow classifiers (or combination with rule-based methods), due to the interpretability of predictions, which still represents a significant issue for more complex methods. Finally, scarcity of publicly available data may also have contributed to insufficient development of more advanced methods, such as extraction of word embeddings from clinical notes.

CONCLUSIONS

Efforts are still required to improve (1) progression of clinical NLP methods from extraction toward understanding; (2) recognition of relations among entities rather than entities in isolation; (3) temporal extraction to understand past, current, and future clinical events; (4) exploitation of alternative sources of clinical knowledge; and (5) availability of large-scale, de-identified clinical corpora.

A Computerized Method for Measuring Computed Tomography Pulmonary Angiography Yield in the Emergency Department: Validation Study

Background

Use of computed tomography pulmonary angiography (CTPA) in the assessment of pulmonary embolism (PE) has markedly increased over the past two decades. While this technology has improved the accuracy of radiological testing for PE, CTPA also carries the risk of substantial iatrogenic harm. Each CTPA carries a 14% risk of contrast-induced nephropathy and a lifetime malignancy risk that can be as high as 2.76%. The appropriate use of CTPA can be estimated by monitoring the CTPA yield, the percentage of tests positive for PE. This is the first study to propose and validate a computerized method for measuring the CTPA yield in the emergency department (ED).

Objective

The objective of our study was to assess the validity of a novel computerized method of calculating the CTPA yield in the ED.

Methods

The electronic health record databases at two tertiary care academic hospitals were queried for CTPA orders completed in the ED over 1-month periods. These visits were linked with an inpatient admission with a discharge diagnosis of PE based on the International Classification of Diseases codes. The computerized the CTPA yield was calculated as the number of CTPA orders with an associated inpatient discharge diagnosis of PE divided by the total number of orders for completed CTPA. This computerized method was then validated by 2 independent reviewers performing a manual chart review, which included reading the free-text radiology reports for each CTPA.

Results

A total of 349 CTPA orders were completed during the 1-month periods at the two institutions. Of them, acute PE was diagnosed on CTPA in 28 studies, with a CTPA yield of 7.7%. The computerized method correctly identified 27 of 28 scans positive for PE. The one discordant scan was tied to a patient who was discharged directly from the ED and, as a result, never received an inpatient discharge diagnosis.

Conclusions

This is the first successful validation study of a computerized method for calculating the CTPA yield in the ED. This method for data extraction allows for an accurate determination of the CTPA yield and is more efficient than manual chart review. With this ability, health care systems can monitor the appropriate use of CTPA and the effect of interventions to reduce overuse and decrease preventable iatrogenic harm.

The use of natural language processing on pediatric diagnostic radiology reports in the electronic health record to identify deep venous thrombosis in children

Venous thromboembolism (VTE) is a potentially life-threatening condition that includes both deep vein thrombosis (DVT) and pulmonary embolism. We sought to improve detection and reporting of children with a new diagnosis of VTE by applying natural language processing (NLP) tools to radiologists' reports. We validated an NLP tool, Reveal NLP (Health Fidelity Inc, San Mateo, CA) and inference rules engine's performance in identifying reports with deep venous thrombosis using a curated set of ultrasound reports. We then configured the NLP tool to scan all available radiology reports on a daily basis for studies that met criteria for VTE between July 1, 2015, and March 31, 2016. The NLP tool and inference rules engine correctly identified 140 out of 144 reports with positive DVT findings and 98 out of 106 negative reports in the validation set. The tool's sensitivity was 97.2% (95% CI 93-99.2%), specificity was 92.5% (95% CI 85.7-96.7%). Subsequently, the NLP tool and inference rules engine processed 6373 radiology reports from 3371 hospital encounters. The NLP tool and inference rules engine identified 178 positive reports and 3193 negative reports with a sensitivity of 82.9% (95% CI 74.8-89.2) and specificity of 97.5% (95% CI 96.9-98). The system functions well as a safety net to screen patients for HA-VTE on a daily basis and offers value as an automated, redundant system. To our knowledge, this is the first pediatric study to apply NLP technology in a prospective manner for HA-VTE identification.

Clinical information extraction applications: A literature review

BACKGROUND

With the rapid adoption of electronic health records (EHRs), it is desirable to harvest information and knowledge from EHRs to support automated systems at the point of care and to enable secondary use of EHRs for clinical and translational research. One critical component used to facilitate the secondary use of EHR data is the information extraction (IE) task, which automatically extracts and encodes clinical information from text.

OBJECTIVES

In this literature review, we present a review of recent published research on clinical information extraction (IE) applications.

METHODS

A literature search was conducted for articles published from January 2009 to September 2016 based on Ovid MEDLINE In-Process & Other Non-Indexed Citations, Ovid MEDLINE, Ovid EMBASE, Scopus, Web of Science, and ACM Digital Library.

RESULTS

A total of 1917 publications were identified for title and abstract screening. Of these publications, 263 articles were selected and discussed in this review in terms of publication venues and data sources, clinical IE tools, methods, and applications in the areas of disease- and drug-related studies, and clinical workflow optimizations.

CONCLUSIONS

Clinical IE has been used for a wide range of applications, however, there is a considerable gap between clinical studies using EHR data and studies using clinical IE. This study enabled us to gain a more concrete understanding of the gap and to provide potential solutions to bridge this gap.

The effects of patient cost sharing on inpatient utilization, cost, and outcome

Background

Health insurance and provider payment reforms all over the world beg a key empirical question: what are the potential impacts of patient cost-sharing on health care utilization, cost and outcomes? The unique health insurance system and rich electronic medical record (EMR) data in China provides us a unique opportunity to study this topic.

Methods

Four years (2010 to 2014) of EMR data from one medical center in China were utilized, including 10,858 adult patients with liver diseases. We measured patient cost-sharing using actual reimbursement ratio (RR) which is allowed us to better capture financial incentive than using type of health insurance. A rigorous risk adjustment method was employed with both comorbidities and disease severity measures acting as risk adjustors. Associations between RR and health use, costs and outcome were analyzed by multivariate analyses.

Results

After risk adjustment, patients with more generous health insurance coverage (higher RR) were found to have longer hospital stay, higher total cost, higher medication cost, and higher ratio of medication to total cost, as well as higher number and likelihood that specific procedures were performed.

Conclusion

Our study implied that patient cost-sharing affects health care services use and cost. This reflects how patients and physicians respond to financial incentives in the current healthcare system in China, and the responses could be a joint effect of both demand and supply side moral hazard. In order to contain cost and improve efficiency in the system, reforming provide payment and insurance scheme is urgently needed.

Creation of a simple natural language processing tool to support an imaging utilization quality dashboard

BACKGROUND

Testing for venous thromboembolism (VTE) is associated with cost and risk to patients (e.g. radiation). To assess the appropriateness of imaging utilization at the provider level, it is important to know that provider's diagnostic yield (percentage of tests positive for the diagnostic entity of interest). However, determining diagnostic yield typically requires either time-consuming, manual review of radiology reports or the use of complex and/or proprietary natural language processing software.

OBJECTIVES

The objectives of this study were twofold: 1) to develop and implement a simple, user-configurable, and open-source natural language processing tool to classify radiology reports with high accuracy and 2) to use the results of the tool to design a provider-specific VTE imaging dashboard, consisting of both utilization rate and diagnostic yield.

METHODS

Two physicians reviewed a training set of 400 lower extremity ultrasound (UTZ) and computed tomography pulmonary angiogram (CTPA) reports to understand the language used in VTE-positive and VTE-negative reports. The insights from this review informed the arguments to the five modifiable parameters of the NLP tool. A validation set of 2,000 studies was then independently classified by the reviewers and by the tool; the classifications were compared and the performance of the tool was calculated.

RESULTS

The tool was highly accurate in classifying the presence and absence of VTE for both the UTZ (sensitivity 95.7%; 95% CI 91.5-99.8, specificity 100%; 95% CI 100-100) and CTPA reports (sensitivity 97.1%; 95% CI 94.3-99.9, specificity 98.6%; 95% CI 97.8-99.4). The diagnostic yield was then calculated at the individual provider level and the imaging dashboard was created.

CONCLUSIONS

We have created a novel NLP tool designed for users without a background in computer programming, which has been used to classify venous thromboembolism reports with a high degree of accuracy. The tool is open-source and available for download at http://iturrate.com/simpleNLP. Results obtained using this tool can be applied to enhance quality by presenting information about utilization and yield to providers via an imaging dashboard.

Accuracy and generalizability of using automated methods for identifying adverse events from electronic health record data: a validation study protocol

Background

Adverse events (AEs) in acute care hospitals are frequent and associated with significant morbidity, mortality, and costs. Measuring AEs is necessary for quality improvement and benchmarking purposes, but current detection methods lack in accuracy, efficiency, and generalizability. The growing availability of electronic health records (EHR) and the development of natural language processing techniques for encoding narrative data offer an opportunity to develop potentially better methods. The purpose of this study is to determine the accuracy and generalizability of using automated methods for detecting three high-incidence and high-impact AEs from EHR data: a) hospital-acquired pneumonia, b) ventilator-associated event and, c) central line-associated bloodstream infection.

Methods

This validation study will be conducted among medical, surgical and ICU patients admitted between 2013 and 2016 to the Centre hospitalier universitaire de Sherbrooke (CHUS) and the McGill University Health Centre (MUHC), which has both French and English sites. A random 60% sample of CHUS patients will be used for model development purposes (cohort 1, development set). Using a random sample of these patients, a reference standard assessment of their medical chart will be performed. Multivariate logistic regression and the area under the curve (AUC) will be employed to iteratively develop and optimize three automated AE detection models (i.e., one per AE of interest) using EHR data from the CHUS. These models will then be validated on a random sample of the remaining 40% of CHUS patients (cohort 1, internal validation set) using chart review to assess accuracy. The most accurate models developed and validated at the CHUS will then be applied to EHR data from a random sample of patients admitted to the MUHC French site (cohort 2) and English site (cohort 3)—a critical requirement given the use of narrative data –, and accuracy will be assessed using chart review. Generalizability will be determined by comparing AUCs from cohorts 2 and 3 to those from cohort 1.

Discussion

This study will likely produce more accurate and efficient measures of AEs. These measures could be used to assess the incidence rates of AEs, evaluate the success of preventive interventions, or benchmark performance across hospitals.

Development and validation of method for defining conditions using Chinese electronic medical record

Background

The adoption of the electronic medical record (EMR) is rapidly growing in China. Constantly evolving, Chinese EMRs contain vast amounts of clinical and financial data, providing tremendous potential for research and policy use; however, they are only partially standardized and contain free text or unstructured data. To utilize the information contained in Chinese EMRs, the development of data extraction methodology is urgently needed. The purpose of this study is to develop and validate methods to extract clinical information from the Chinese EMR for research use.

Methods

Using 2010 to 2014 EMR data from YouAn Hospital, a large teaching hospital affiliated with Capital Medical University in Beijing, China, we developed extraction methods including 40 EMR definitions for defining 6 liver disease, 5 disease severity conditions, and 29 comorbidities and treatments. We conducted a chart review of 450 randomly selected EMRs. Using physician chart review results as a reference, sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) were calculated to validate each EMR definition.

Results

The sensitivity of the 6 EMR definitions for liver diseases ranged from 78.9 to 100.0 %, and PPV ranged from 82.1 to 100.0 %. The sensitivity of the 5 definitions on disease severity conditions ranged from 91.0 to 100.0 %, and PPV ranged from 79.2 to 100.0 %. Among the 29 EMR definitions for comorbidities and treatments, 23 had sensitivity over 90.0 % and 25 had PPV over 80.0 %. The specificity and NPV for all 40 EMR definitions were over 90.0 %.

Conclusion

The extraction method developed is a valid way of extracting information on liver diseases, comorbidities and related treatments from YouAn hospital EMRs. Our method should be modified for application to other Chinese EMR systems, following our framework for extracting conditions.

Electronic supplementary material

The online version of this article (doi:10.1186/s12911-016-0348-6) contains supplementary material, which is available to authorized users.

Using automatically extracted information from mammography reports for decision-support

OBJECTIVE

To evaluate a system we developed that connects natural language processing (NLP) for information extraction from narrative text mammography reports with a Bayesian network for decision-support about breast cancer diagnosis. The ultimate goal of this system is to provide decision support as part of the workflow of producing the radiology report.

MATERIALS AND METHODS

We built a system that uses an NLP information extraction system (which extract BI-RADS descriptors and clinical information from mammography reports) to provide the necessary inputs to a Bayesian network (BN) decision support system (DSS) that estimates lesion malignancy from BI-RADS descriptors. We used this integrated system to predict diagnosis of breast cancer from radiology text reports and evaluated it with a reference standard of 300 mammography reports. We collected two different outputs from the DSS: (1) the probability of malignancy and (2) the BI-RADS final assessment category. Since NLP may produce imperfect inputs to the DSS, we compared the difference between using perfect ("reference standard") structured inputs to the DSS ("RS-DSS") vs NLP-derived inputs ("NLP-DSS") on the output of the DSS using the concordance correlation coefficient. We measured the classification accuracy of the BI-RADS final assessment category when using NLP-DSS, compared with the ground truth category established by the radiologist.

RESULTS

The NLP-DSS and RS-DSS had closely matched probabilities, with a mean paired difference of 0.004±0.025. The concordance correlation of these paired measures was 0.95. The accuracy of the NLP-DSS to predict the correct BI-RADS final assessment category was 97.58%.

CONCLUSION

The accuracy of the information extracted from mammography reports using the NLP system was sufficient to provide accurate DSS results. We believe our system could ultimately reduce the variation in practice in mammography related to assessment of malignant lesions and improve management decisions.

Natural Language Processing Techniques for Extracting and Categorizing Finding Measurements in Narrative Radiology Reports

BACKGROUND

Accumulating quantitative outcome parameters may contribute to constructing a healthcare organization in which outcomes of clinical procedures are reproducible and predictable. In imaging studies, measurements are the principal category of quantitative para meters.

OBJECTIVES

The purpose of this work is to develop and evaluate two natural language processing engines that extract finding and organ measurements from narrative radiology reports and to categorize extracted measurements by their "temporality".

METHODS

The measurement extraction engine is developed as a set of regular expressions. The engine was evaluated against a manually created ground truth. Automated categorization of measurement temporality is defined as a machine learning problem. A ground truth was manually developed based on a corpus of radiology reports. A maximum entropy model was created using features that characterize the measurement itself and its narrative context. The model was evaluated in a ten-fold cross validation protocol.

RESULTS

The measurement extraction engine has precision 0.994 and recall 0.991. Accuracy of the measurement classification engine is 0.960.

CONCLUSIONS

The work contributes to machine understanding of radiology reports and may find application in software applications that process medical data.