Automated identification of pneumonia in chest radiograph reports in critically ill patients

A methodological approach to validate pneumonia encounters from radiology reports using Natural Language Processing (NLP)

INTRODUCTION

Pneumonia is caused by microbes that establish an infectious process in the lungs. The gold standard for pneumonia diagnosis is radiologist-documented pneumonia-related features in radiology notes that are captured in electronic health records in an unstructured format.

OBJECTIVE

The study objective was to develop a methodological approach for assessing validity of a pneumonia diagnosis based on identifying presence or absence of key radiographic features in radiology reports with subsequent rendering of diagnostic decisions into a structured format.

METHODS

A pneumonia-specific Natural Language Processing (NLP) pipeline was strategically developed applying cTAKES to validate pneumonia diagnoses following development of a pneumonia feature-specific lexicon. Radiographic reports of study-eligible subjects identified by International Classification of Diseases (ICD) codes were parsed through the NLP pipeline. Classification rules were developed to assign each pneumonia episode into one of three categories: "positive", "negative" or "not classified: requires manual review" based on tagged concepts that support or refute diagnostic codes.

RESULTS

A total of 91,998 pneumonia episodes diagnosed in 65,904 patients were retrieved retrospectively. Approximately 89% (81,707/91,998) of the total pneumonia episodes were documented by 225,893 chest x-ray reports. NLP classified and validated 33% (26,800/81,707) of pneumonia episodes classified as 'Pneumonia-positive', 19% as (15401/81,707) as 'Pneumonia-negative' and 48% (39,209/81,707) as ''episode classification pending further manual review'. NLP pipeline performance metrics included accuracy (76.3%), sensitivity (88%), and specificity (75%).

CONCLUSION

The pneumonia-specific NLP pipeline exhibited good performance comparable to other pneumonia-specific NLP systems developed to date.

Predictive Risk Models for Wound Infection-Related Hospitalization or ED Visits in Home Health Care Using Machine-Learning Algorithms

OBJECTIVE

Wound infection is prevalent in home healthcare (HHC) and often leads to hospitalizations. However, none of the previous studies of wounds in HHC have used data from clinical notes. Therefore, the authors created a more accurate description of a patient's condition by extracting risk factors from clinical notes to build predictive models to identify a patient's risk of wound infection in HHC.

METHODS

The structured data (eg, standardized assessments) and unstructured information (eg, narrative-free text charting) were retrospectively reviewed for HHC patients with wounds who were served by a large HHC agency in 2014. Wound infection risk factors were identified through bivariate analysis and stepwise variable selection. Risk predictive performance of three machine learning models (logistic regression, random forest, and artificial neural network) was compared.

RESULTS

A total of 754 of 54,316 patients (1.39%) had a hospitalization or ED visit related to wound infection. In the bivariate logistic regression, language describing wound type in the patient's clinical notes was strongly associated with risk (odds ratio, 9.94; P < .05). The areas under the curve were 0.82 in logistic regression, 0.75 in random forest, and 0.78 in artificial neural network. Risk prediction performance of the models improved (by up to 13.2%) after adding risk factors extracted from clinical notes.

CONCLUSIONS

Logistic regression showed the best risk prediction performance in prediction of wound infection-related hospitalization or ED visits in HHC. The use of data extracted from clinical notes can improve the performance of risk prediction models.

Improving reference standards for validation of AI-based radiography

Objective:

Demonstrate the importance of combining multiple readers' opinions, in a context-aware manner, when establishing the reference standard for validation of artificial intelligence (AI) applications for, e.g. chest radiographs. By comparing individual readers, majority vote of a panel, and panel-based discussion, we identify methods which maximize interobserver agreement and label reproducibility.

Methods:

1100 frontal chest radiographs were evaluated for 6 findings: airspace opacity, cardiomegaly, pulmonary edema, fracture, nodules, and pneumothorax. Each image was reviewed by six radiologists, first individually and then via asynchronous adjudication (web-based discussion) in two panels of three readers to resolve disagreements within each panel. We quantified the reproducibility of each method by measuring interreader agreement.

Results:

Panel-based majority vote improved agreement relative to individual readers for all findings. Most disagreements were resolved with two rounds of adjudication, which further improved reproducibility for some findings, particularly reducing misses. Improvements varied across finding categories, with adjudication improving agreement for cardiomegaly, fractures, and pneumothorax.

Conclusion:

The likelihood of interreader agreement, even within panels of US board-certified radiologists, must be considered before reads can be used as a reference standard for validation of proposed AI tools. Agreement and, by extension, reproducibility can be improved by applying majority vote, maximum sensitivity, or asynchronous adjudication for different findings, which supports the development of higher quality clinical research.

Advances in knowledge:

A panel of three experts is a common technique for establishing reference standards when ground truth is not available for use in AI validation. The manner in which differing opinions are resolved is shown to be important, and has not been previously explored.

Utility of radiographic keyword abstraction for identification of misdiagnosed pneumonia

Misdiagnosis of bacterial pneumonia is a leading cause of inappropriate antimicrobial use in hospitalized patients. We report a novel strategy of keyword abstraction from chest radiography transcripts that reliably identified patients with pneumonia misdiagnosis and opportunities for antibiotic discontinuation and/or de-escalation.

The Presentation, Pace, and Profile of Infection and Sepsis Patients Hospitalized Through the Emergency Department: An Exploratory Analysis

To characterize the signs and symptoms of sepsis, compare them with those from simple infection and other emergent conditions and evaluate their association with hospital outcomes.

Design Setting Participants and INTERVENTION

A multicenter, retrospective cohort study of 408,377 patients hospitalized through the emergency department from 2012 to 2017 with sepsis, suspected infection, heart failure, or stroke. Infected patients were identified based on Sepsis-3 criteria, whereas noninfected patients were identified through diagnosis codes.

MEASUREMENTS AND MAIN RESULTS

Signs and symptoms were identified within physician clinical documentation in the first 24 hours of hospitalization using natural language processing. The time of sign and symptom onset prior to presentation was quantified, and sign and symptom prevalence was assessed. Using multivariable logistic regression, the association of each sign and symptom with four outcomes was evaluated: sepsis versus suspected infection diagnosis, hospital mortality, ICU admission, and time of first antibiotics (> 3 vs ≤ 3 hr from presentation). A total of 10,825 signs and symptoms were identified in 6,148,348 clinical documentation fragments. The most common symptoms overall were as follows: dyspnea (35.2%), weakness (27.2%), altered mental status (24.3%), pain (23.9%), cough (19.7%), edema (17.8%), nausea (16.9%), hypertension (15.6%), fever (13.9%), and chest pain (12.1%). Compared with predominant signs and symptoms in heart failure and stroke, those present in infection were heterogeneous. Signs and symptoms indicative of neurologic dysfunction, significant respiratory conditions, and hypotension were strongly associated with sepsis diagnosis, hospital mortality, and intensive care. Fever, present in only a minority of patients, was associated with improved mortality (odds ratio, 0.67, 95% CI, 0.64-0.70; p < 0.001). For common symptoms, the peak time of symptom onset before sepsis was 2 days, except for altered mental status, which peaked at 1 day prior to presentation.

Conclusions

The clinical presentation of sepsis was heterogeneous and occurred with rapid onset prior to hospital presentation. These findings have important implications for improving public education, clinical treatment, and quality measures of sepsis care.

Natural Language Processing and Machine Learning to Enable Clinical Decision Support for Treatment of Pediatric Pneumonia

Pneumonia is the most frequent cause of infectious disease-related deaths in children worldwide. Clinical decision support (CDS) applications can guide appropriate treatment, but the system must first recognize the appropriate diagnosis. To enable CDS for pediatric pneumonia, we developed an algorithm integrating natural language processing (NLP) and random forest classifiers to identify potential pediatric pneumonia from radiology reports. We deployed the algorithm in the EHR of a large children's hospital using real-time NLP. We describe the development and deployment of the algorithm, and evaluate our approach using 9-months of data gathered while the system was in use. Our model, trained on individual radiology reports, had an AUC of 0.954. The intervention, evaluated on patient encounters that could include multiple radiology reports, achieved a sensitivity, specificity, and positive predictive value of0.899, 0.949, and 0.781, respectively.

Evidence-based Clinical Decision Support Systems for the prediction and detection of three disease states in critical care: A systematic literature review

Background: Clinical decision support (CDS) systems have emerged as tools providing intelligent decision making to address challenges of critical care. CDS systems can be based on existing guidelines or best practices; and can also utilize machine learning to provide a diagnosis, recommendation, or therapy course. Methods: This research aimed to identify evidence-based study designs and outcome measures to determine the clinical effectiveness of clinical decision support systems in the detection and prediction of hemodynamic instability, respiratory distress, and infection within critical care settings. PubMed, ClinicalTrials.gov and Cochrane Database of Systematic Reviews were systematically searched to identify primary research published in English between 2013 and 2018. Studies conducted in the USA, Canada, UK, Germany and France with more than 10 participants per arm were included. Results: In studies on hemodynamic instability, the prediction and management of septic shock were the most researched topics followed by the early prediction of heart failure. For respiratory distress, the most popular topics were pneumonia detection and prediction followed by pulmonary embolisms. Given the importance of imaging and clinical notes, this area combined Machine Learning with image analysis and natural language processing. In studies on infection, the most researched areas were the detection, prediction, and management of sepsis, surgical site infections, as well as acute kidney injury. Overall, a variety of Machine Learning algorithms were utilized frequently, particularly support vector machines, boosting techniques, random forest classifiers and neural networks. Sensitivity, specificity, and ROC AUC were the most frequently reported performance measures. Conclusion: This review showed an increasing use of Machine Learning for CDS in all three areas. Large datasets are required for training these algorithms; making it imperative to appropriately address, challenges such as class imbalance, correct labelling of data and missing data. Recommendations are formulated for the development and successful adoption of CDS systems.

Evidence-based Clinical Decision Support Systems for the prediction and detection of three disease states in critical care: A systematic literature review

Background: Clinical decision support (CDS) systems have emerged as tools providing intelligent decision making to address challenges of critical care. CDS systems can be based on existing guidelines or best practices; and can also utilize machine learning to provide a diagnosis, recommendation, or therapy course. Methods: This research aimed to identify evidence-based study designs and outcome measures to determine the clinical effectiveness of clinical decision support systems in the detection and prediction of hemodynamic instability, respiratory distress, and infection within critical care settings. PubMed, ClinicalTrials.gov and Cochrane Database of Systematic Reviews were systematically searched to identify primary research published in English between 2013 and 2018. Studies conducted in the USA, Canada, UK, Germany and France with more than 10 participants per arm were included. Results: In studies on hemodynamic instability, the prediction and management of septic shock were the most researched topics followed by the early prediction of heart failure. For respiratory distress, the most popular topics were pneumonia detection and prediction followed by pulmonary embolisms. Given the importance of imaging and clinical notes, this area combined Machine Learning with image analysis and natural language processing. In studies on infection, the most researched areas were the detection, prediction, and management of sepsis, surgical site infections, as well as acute kidney injury. Overall, a variety of Machine Learning algorithms were utilized frequently, particularly support vector machines, boosting techniques, random forest classifiers and neural networks. Sensitivity, specificity, and ROC AUC were the most frequently reported performance measures. Conclusion: This review showed an increasing use of Machine Learning for CDS in all three areas. Large datasets are required for training these algorithms; making it imperative to appropriately address, challenges such as class imbalance, correct labelling of data and missing data. Recommendations are formulated for the development and successful adoption of CDS systems.

Identifying pulmonary nodules or masses on chest radiography using deep learning: external validation and strategies to improve clinical practice

AIM

To test the diagnostic performance of a deep learning-based system for the detection of clinically significant pulmonary nodules/masses on chest radiographs.

MATERIALS AND METHODS

Using a retrospective study of 100 patients (47 with clinically significant pulmonary nodules/masses and 53 control subjects without pulmonary nodules), two radiologists verified clinically significantly pulmonary nodules/masses according to chest computed tomography (CT) findings. A computer-aided diagnosis (CAD) software using a deep-learning approach was used to detect pulmonary nodules/masses to determine the diagnostic performance in four algorithms (heat map, abnormal probability, nodule probability, and mass probability).

RESULTS

A total of 100 cases were included in the analysis. Among the four algorithms, mass algorithm could achieve a 76.6% sensitivity (36/47, 11 false negative) and 88.68% specificity (47/53, six false-positive) in the detection of pulmonary nodules/masses at the optimal probability score cut-off of 0.2884. Compared to the other three algorithms, mass probability algorithm had best predictive ability for pulmonary nodule/mass detection at the optimal probability score cut-off of 0.2884 (AUC^Mass: 0.916 versus AUC^{Heat map}: 0.682, p<0.001; AUC^Mass: 0.916 versus AUC^Abnormal: 0.810, p=0.002; AUC^Mass: 0.916 versus AUC^Nodule: 0.813, p=0.014).

CONCLUSION

In conclusion, the deep-learning based computer-aided diagnosis system will likely play a vital role in the early detection and diagnosis of pulmonary nodules/masses on chest radiographs. In future applications, these algorithms could support triage workflow via double reading to improve sensitivity and specificity during the diagnostic process.

Accurate Identification of Colonoscopy Quality and Polyp Findings Using Natural Language Processing

OBJECTIVES

The aim of this study was to test the ability of a commercially available natural language processing (NLP) tool to accurately extract examination quality-related and large polyp information from colonoscopy reports with varying report formats.

BACKGROUND

Colonoscopy quality reporting often requires manual data abstraction. NLP is another option for extracting information; however, limited data exist on its ability to accurately extract examination quality and polyp findings from unstructured text in colonoscopy reports with different reporting formats.

STUDY DESIGN

NLP strategies were developed using 500 colonoscopy reports from Kaiser Permanente Northern California and then tested using 300 separate colonoscopy reports that underwent manual chart review. Using findings from manual review as the reference standard, we evaluated the NLP tool's sensitivity, specificity, positive predictive value (PPV), and accuracy for identifying colonoscopy examination indication, cecal intubation, bowel preparation adequacy, and polyps ≥10 mm.

RESULTS

The NLP tool was highly accurate in identifying examination quality-related variables from colonoscopy reports. Compared with manual review, sensitivity for screening indication was 100% (95% confidence interval: 95.3%-100%), PPV was 90.6% (82.3%-95.8%), and accuracy was 98.2% (97.0%-99.4%). For cecal intubation, sensitivity was 99.6% (98.0%-100%), PPV was 100% (98.5%-100%), and accuracy was 99.8% (99.5%-100%). For bowel preparation adequacy, sensitivity was 100% (98.5%-100%), PPV was 100% (98.5%-100%), and accuracy was 100% (100%-100%). For polyp(s) ≥10 mm, sensitivity was 90.5% (69.6%-98.8%), PPV was 100% (82.4%-100%), and accuracy was 95.2% (88.8%-100%).

CONCLUSION

NLP yielded a high degree of accuracy for identifying examination quality-related and large polyp information from diverse types of colonoscopy reports.

Reinventing Radiology: Big Data and the Future of Medical Imaging

PURPOSE

Today, data surrounding most of our lives are collected and stored. Data scientists are beginning to explore applications that could harness this information and make sense of it.

MATERIALS AND METHODS

In this review, the topic of Big Data is explored, and applications in modern health care are considered.

RESULTS

Big Data is a concept that has evolved from the modern trend of "scientism." One of the primary goals of data scientists is to develop ways to discover new knowledge from the vast quantities of increasingly available information.

CONCLUSIONS

Current and future opportunities and challenges with respect to radiology are provided with emphasis on cardiothoracic imaging.

Development and Validation of a Natural Language Processing Tool to Identify Patients Treated for Pneumonia across VA Emergency Departments

BACKGROUND

Identifying pneumonia using diagnosis codes alone may be insufficient for research on clinical decision making. Natural language processing (NLP) may enable the inclusion of cases missed by diagnosis codes.

OBJECTIVES

This article (1) develops a NLP tool that identifies the clinical assertion of pneumonia from physician emergency department (ED) notes, and (2) compares classification methods using diagnosis codes versus NLP against a gold standard of manual chart review to identify patients initially treated for pneumonia.

METHODS

Among a national population of ED visits occurring between 2006 and 2012 across the Veterans Affairs health system, we extracted 811 physician documents containing search terms for pneumonia for training, and 100 random documents for validation. Two reviewers annotated span- and document-level classifications of the clinical assertion of pneumonia. An NLP tool using a support vector machine was trained on the enriched documents. We extracted diagnosis codes assigned in the ED and upon hospital discharge and calculated performance characteristics for diagnosis codes, NLP, and NLP plus diagnosis codes against manual review in training and validation sets.

RESULTS

Among the training documents, 51% contained clinical assertions of pneumonia; in the validation set, 9% were classified with pneumonia, of which 100% contained pneumonia search terms. After enriching with search terms, the NLP system alone demonstrated a recall/sensitivity of 0.72 (training) and 0.55 (validation), and a precision/positive predictive value (PPV) of 0.89 (training) and 0.71 (validation). ED-assigned diagnostic codes demonstrated lower recall/sensitivity (0.48 and 0.44) but higher precision/PPV (0.95 in training, 1.0 in validation); the NLP system identified more "possible-treated" cases than diagnostic coding. An approach combining NLP and ED-assigned diagnostic coding classification achieved the best performance (sensitivity 0.89 and PPV 0.80).

CONCLUSION

System-wide application of NLP to clinical text can increase capture of initial diagnostic hypotheses, an important inclusion when studying diagnosis and clinical decision-making under uncertainty.

Fifty Years of Research in ARDS. The Epidemiology of Acute Respiratory Distress Syndrome. A 50th Birthday Review

Since its first description 50 years ago, no other intensive care syndrome has been as extensively studied as acute respiratory distress syndrome (ARDS). Despite this extensive body of research, many basic epidemiologic questions remain unsolved. The lack of gold standard tests jeopardizes accurate diagnosis and translational research. Wide variation in the population incidence has been reported, making even simple estimates of the burden of disease problematic. Despite these limitations, there has been an increase in the understanding of pathophysiology and important risk factors both for the development of ARDS and for important patient-centered outcomes like mortality. In this Critical Care Perspective, we discuss the historical context of ARDS description and attempts at its definition. We highlight the epidemiologic challenges of studying ARDS, as well as other intensive care syndromes, and propose solutions to address them. We update the current knowledge of ARDS trends in incidence and mortality, risk factors, and recently described endotypes.

Using multiclass classification to automate the identification of patient safety incident reports by type and severity

BACKGROUND

Approximately 10% of admissions to acute-care hospitals are associated with an adverse event. Analysis of incident reports helps to understand how and why incidents occur and can inform policy and practice for safer care. Unfortunately our capacity to monitor and respond to incident reports in a timely manner is limited by the sheer volumes of data collected. In this study, we aim to evaluate the feasibility of using multiclass classification to automate the identification of patient safety incidents in hospitals.

METHODS

Text based classifiers were applied to identify 10 incident types and 4 severity levels. Using the one-versus-one (OvsO) and one-versus-all (OvsA) ensemble strategies, we evaluated regularized logistic regression, linear support vector machine (SVM) and SVM with a radial-basis function (RBF) kernel. Classifiers were trained and tested with "balanced" datasets (n_ _Type  = 2860, n_ _{SeverityLevel}  = 1160) from a state-wide incident reporting system. Testing was also undertaken with imbalanced "stratified" datasets (n_ _Type  = 6000, n_ _{SeverityLevel} =5950) from the state-wide system and an independent hospital reporting system. Classifier performance was evaluated using a confusion matrix, as well as F-score, precision and recall.

RESULTS

The most effective combination was a OvsO ensemble of binary SVM RBF classifiers with binary count feature extraction. For incident type, classifiers performed well on balanced and stratified datasets (F-score: 78.3, 73.9%), but were worse on independent datasets (68.5%). Reports about falls, medications, pressure injury, aggression and blood products were identified with high recall and precision. "Documentation" was the hardest type to identify. For severity level, F-score for severity assessment code (SAC) 1 (extreme risk) was 87.3 and 64% for SAC4 (low risk) on balanced data. With stratified data, high recall was achieved for SAC1 (82.8-84%) but precision was poor (6.8-11.2%). High risk incidents (SAC2) were confused with medium risk incidents (SAC3).

CONCLUSIONS

Binary classifier ensembles appear to be a feasible method for identifying incidents by type and severity level. Automated identification should enable safety problems to be detected and addressed in a more timely manner. Multi-label classifiers may be necessary for reports that relate to more than one incident type.

Enhancing Comparative Effectiveness Research With Automated Pediatric Pneumonia Detection in a Multi-Institutional Clinical Repository: A PHIS+ Pilot Study

Background

Community-acquired pneumonia is a leading cause of pediatric morbidity. Administrative data are often used to conduct comparative effectiveness research (CER) with sufficient sample sizes to enhance detection of important outcomes. However, such studies are prone to misclassification errors because of the variable accuracy of discharge diagnosis codes.

Objective

The aim of this study was to develop an automated, scalable, and accurate method to determine the presence or absence of pneumonia in children using chest imaging reports.

Methods

The multi-institutional PHIS+ clinical repository was developed to support pediatric CER by expanding an administrative database of children’s hospitals with detailed clinical data. To develop a scalable approach to find patients with bacterial pneumonia more accurately, we developed a Natural Language Processing (NLP) application to extract relevant information from chest diagnostic imaging reports. Domain experts established a reference standard by manually annotating 282 reports to train and then test the NLP application. Findings of pleural effusion, pulmonary infiltrate, and pneumonia were automatically extracted from the reports and then used to automatically classify whether a report was consistent with bacterial pneumonia.

Results

Compared with the annotated diagnostic imaging reports reference standard, the most accurate implementation of machine learning algorithms in our NLP application allowed extracting relevant findings with a sensitivity of .939 and a positive predictive value of .925. It allowed classifying reports with a sensitivity of .71, a positive predictive value of .86, and a specificity of .962. When compared with each of the domain experts manually annotating these reports, the NLP application allowed for significantly higher sensitivity (.71 vs .527) and similar positive predictive value and specificity .

Conclusions

NLP-based pneumonia information extraction of pediatric diagnostic imaging reports performed better than domain experts in this pilot study. NLP is an efficient method to extract information from a large collection of imaging reports to facilitate CER.

Natural Language Processing in Radiology: A Systematic Review

Radiological reporting has generated large quantities of digital content within the electronic health record, which is potentially a valuable source of information for improving clinical care and supporting research. Although radiology reports are stored for communication and documentation of diagnostic imaging, harnessing their potential requires efficient and automated information extraction: they exist mainly as free-text clinical narrative, from which it is a major challenge to obtain structured data. Natural language processing (NLP) provides techniques that aid the conversion of text into a structured representation, and thus enables computers to derive meaning from human (ie, natural language) input. Used on radiology reports, NLP techniques enable automatic identification and extraction of information. By exploring the various purposes for their use, this review examines how radiology benefits from NLP. A systematic literature search identified 67 relevant publications describing NLP methods that support practical applications in radiology. This review takes a close look at the individual studies in terms of tasks (ie, the extracted information), the NLP methodology and tools used, and their application purpose and performance results. Additionally, limitations, future challenges, and requirements for advancing NLP in radiology will be discussed.

Developing a web-based SKOS editor

Background

The Simple Knowledge Organization System (SKOS) was introduced to the wider research community by a 2005 World Wide Web Consortium (W3C) working draft, and further developed and refined in a 2009 W3C recommendation. Since then, SKOS has become the de facto standard for representing and sharing thesauri, lexicons, vocabularies, taxonomies, and classification schemes. In this paper, we describe the development of a web-based, free, open-source SKOS editor built for the development, curation, and management of small to medium-sized lexicons for health-related Natural Language Processing (NLP).

Results

The web-based SKOS editor allows users to create, curate, version, manage, and visualise SKOS resources. We tested the system against five widely-used, publicly-available SKOS vocabularies of various sizes and found that the editor is suitable for the development and management of small to medium-size lexicons. Qualitative testing has focussed on using the editor to develop lexical resources to drive NLP applications in two domains. First, developing a lexicon to support an Electronic Health Record-based NLP system for the automatic identification of pneumonia symptoms. Second, creating a taxonomy of lexical cues associated with Diagnostic and Statistical Manual of Mental Disorders (DSM-5) diagnoses with the goal of facilitating the automatic identification of symptoms associated with depression from short, informal texts.

Conclusions

The SKOS editor we have developed is — to the best of our knowledge — the first free, open-source, web-based, SKOS editor capable of creating, curating, versioning, managing, and visualising SKOS lexicons.

Electronic supplementary material

The online version of this article (doi:10.1186/s13326-015-0043-z) contains supplementary material, which is available to authorized users.

Agile text mining for the 2014 i2b2/UTHealth Cardiac risk factors challenge

This paper describes the use of an agile text mining platform (Linguamatics' Interactive Information Extraction Platform, I2E) to extract document-level cardiac risk factors in patient records as defined in the i2b2/UTHealth 2014 challenge. The approach uses a data-driven rule-based methodology with the addition of a simple supervised classifier. We demonstrate that agile text mining allows for rapid optimization of extraction strategies, while post-processing can leverage annotation guidelines, corpus statistics and logic inferred from the gold standard data. We also show how data imbalance in a training set affects performance. Evaluation of this approach on the test data gave an F-Score of 91.7%, one percent behind the top performing system.

Risk prediction for chronic kidney disease progression using heterogeneous electronic health record data and time series analysis

BACKGROUND

As adoption of electronic health records continues to increase, there is an opportunity to incorporate clinical documentation as well as laboratory values and demographics into risk prediction modeling.

OBJECTIVE

The authors develop a risk prediction model for chronic kidney disease (CKD) progression from stage III to stage IV that includes longitudinal data and features drawn from clinical documentation.

METHODS

The study cohort consisted of 2908 primary-care clinic patients who had at least three visits prior to January 1, 2013 and developed CKD stage III during their documented history. Development and validation cohorts were randomly selected from this cohort and the study datasets included longitudinal inpatient and outpatient data from these populations. Time series analysis (Kalman filter) and survival analysis (Cox proportional hazards) were combined to produce a range of risk models. These models were evaluated using concordance, a discriminatory statistic.

RESULTS

A risk model incorporating longitudinal data on clinical documentation and laboratory test results (concordance 0.849) predicts progression from state III CKD to stage IV CKD more accurately when compared to a similar model without laboratory test results (concordance 0.733, P<.001), a model that only considers the most recent laboratory test results (concordance 0.819, P < .031) and a model based on estimated glomerular filtration rate (concordance 0.779, P < .001).

CONCLUSIONS

A risk prediction model that takes longitudinal laboratory test results and clinical documentation into consideration can predict CKD progression from stage III to stage IV more accurately than three models that do not take all of these variables into consideration.