Detecting inpatient falls by using natural language processing of electronic medical records

The use of natural language processing for the identification of ageing syndromes including sarcopenia, frailty and falls in electronic healthcare records: a systematic review

BACKGROUND

Recording and coding of ageing syndromes in hospital records is known to be suboptimal. Natural Language Processing algorithms may be useful to identify diagnoses in electronic healthcare records to improve the recording and coding of these ageing syndromes, but the feasibility and diagnostic accuracy of such algorithms are unclear.

METHODS

We conducted a systematic review according to a predefined protocol and in line with Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) guidelines. Searches were run from the inception of each database to the end of September 2023 in PubMed, Medline, Embase, CINAHL, ACM digital library, IEEE Xplore and Scopus. Eligible studies were identified via independent review of search results by two coauthors and data extracted from each study to identify the computational method, source of text, testing strategy and performance metrics. Data were synthesised narratively by ageing syndrome and computational method in line with the Studies Without Meta-analysis guidelines.

RESULTS

From 1030 titles screened, 22 studies were eligible for inclusion. One study focussed on identifying sarcopenia, one frailty, twelve falls, five delirium, five dementia and four incontinence. Sensitivity (57.1%-100%) of algorithms compared with a reference standard was reported in 20 studies, and specificity (84.0%-100%) was reported in only 12 studies. Study design quality was variable with results relevant to diagnostic accuracy not always reported, and few studies undertaking external validation of algorithms.

CONCLUSIONS

Current evidence suggests that Natural Language Processing algorithms can identify ageing syndromes in electronic health records. However, algorithms require testing in rigorously designed diagnostic accuracy studies with appropriate metrics reported.

Monitoring falls in residential aged care facilities: Agreement between falls incident reports and progress notes

OBJECTIVES

Accurate fall reporting is essential for assessing the effectiveness of fall prevention strategies. This study aimed to investigate the level of agreement between incident reports and resident progress notes as data sources for falls monitoring in residential aged care facilities.

METHODS

A retrospective observational study was conducted involving 46 older people from six residential aged care facilities who had consented to join the broader TOP UP trial. Fall events documented in the incident report system and resident progress notes over 12 months before randomisation were extracted by two independent reviewers using a standardised Excel form. Agreement between the two data collection methods was calculated using Cohen's kappa coefficient.

RESULTS

A total of 75 falls were recorded from 27 (59%) of the 46 participants who were 65% female, with an average age of 83 [SD 9] years. The incident reports captured 68 (90.7%) falls, while the progress notes captured 73 (97.3%) falls. Overall, there was a 75% agreement between falls recorded in progress notes and incident reports. Perfect agreement was identified for five facilities (n = 35), while one facility had a lower agreement rate of 29% (n = 11), which appeared to be attributable to staff shortages linked to the COVID-19 pandemic.

CONCLUSIONS

There was substantial agreement between incident reports and progress records. These findings support the use of incident reports for identifying falls in research or to investigate the effectiveness of fall prevention strategies in residential aged care facilities.

BERT-Based Neural Network for Inpatient Fall Detection From Electronic Medical Records: Retrospective Cohort Study

BACKGROUND

Inpatient falls are a substantial concern for health care providers and are associated with negative outcomes for patients. Automated detection of falls using machine learning (ML) algorithms may aid in improving patient safety and reducing the occurrence of falls.

OBJECTIVE

This study aims to develop and evaluate an ML algorithm for inpatient fall detection using multidisciplinary progress record notes and a pretrained Bidirectional Encoder Representation from Transformers (BERT) language model.

METHODS

A cohort of 4323 adult patients admitted to 3 acute care hospitals in Calgary, Alberta, Canada from 2016 to 2021 were randomly sampled. Trained reviewers determined falls from patient charts, which were linked to electronic medical records and administrative data. The BERT-based language model was pretrained on clinical notes, and a fall detection algorithm was developed based on a neural network binary classification architecture.

RESULTS

To address various use scenarios, we developed 3 different Alberta hospital notes-specific BERT models: a high sensitivity model (sensitivity 97.7, IQR 87.7-99.9), a high positive predictive value model (positive predictive value 85.7, IQR 57.2-98.2), and the high F1-score model (F1=64.4). Our proposed method outperformed 3 classical ML algorithms and an International Classification of Diseases code-based algorithm for fall detection, showing its potential for improved performance in diverse clinical settings.

CONCLUSIONS

The developed algorithm provides an automated and accurate method for inpatient fall detection using multidisciplinary progress record notes and a pretrained BERT language model. This method could be implemented in clinical practice to improve patient safety and reduce the occurrence of falls in hospitals.

The use of natural language processing in detecting and predicting falls within the healthcare setting: a systematic review

Falls are a common problem associated with significant morbidity, mortality, and economic costs. Current fall prevention policies in local healthcare settings are often guided by information provided by fall risk assessment tools, incident reporting, and coding data. This review was conducted with the aim of identifying studies which utilized natural language processing (NLP) for the automated detection and prediction of falls in the healthcare setting. The databases Ovid Medline, Ovid Embase, Ovid Emcare, PubMed, CINAHL, IEEE Xplore, and Ei Compendex were searched from 2012 until April 2023. Retrospective derivation, validation, and implementation studies wherein patients experienced falls within a healthcare setting were identified for inclusion. The initial search yielded 2611 publications for title and abstract screening. Full-text screening was conducted on 105 publications, resulting in 26 unique studies that underwent qualitative analyses. Studies applied NLP towards falls risk factor identification, known falls detection, future falls prediction, and falls severity stratification with reasonable success. The NLP pipeline was reviewed in detail between studies and models utilizing rule-based, machine learning (ML), deep learning (DL), and hybrid approaches were examined. With a growing literature surrounding falls prediction in both inpatient and outpatient environments, the absence of studies examining the impact of these models on patient and system outcomes highlights the need for further implementation studies. Through an exploration of the application of NLP techniques, it may be possible to develop models with higher performance in automated falls prediction and detection.

External validation of the Johns Hopkins Fall Risk Assessment Tool in older Dutch hospitalized patients

PURPOSE

Fall prevention is a safety goal in many hospitals. The performance of the Johns Hopkins Fall Risk Assessment Tool (JHFRAT) in older inpatients is largely unknown. We aimed to assess the JHFRAT performance in a large sample of Dutch older inpatients, including its trend over time.

METHODS

We used an Electronic Health Records (EHR) dataset with hospitalized patients (≥ 70), admitted for ≥ 24 h between 2016 and 2021. Inpatient falls were extracted from structured and free-text data. We assessed the association between JHFRAT and falls using logistic regression. For test accuracy, we calculated sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV). Discrimination was measured by the AUC. For calibration, we plotted the predicted fall probability with the actual probability of falls. For time-related effects, we calculated the AUC per 6 months (using data of patients admitted during the 6 months' time interval) and plotted these different AUC values over time. Furthermore, we compared the model (JHFRAT and falls) with and without adjusting for seasonal influenza, COVID-19, spring, summer, fall or winter periods.

RESULTS

Data included 17,263 admissions with at least 1 JHFRAT measurement, a median age of 76 and a percentage female of 47%. The in-hospital fall prevalence was 2.5%. JHFRAT [OR = 1.11 (1.03-1.20)] and its subcategories were significantly associated with falls. For medium/high risk of falls (JHFRAT > 5), sensitivity was 73%, specificity 51%, PPV 4% and NPV 99%. The overall AUC was 0.67, varying over time between 0.62 and 0.71 (for 6 months' time intervals). Seasonal influenza did affect the association between JHFRAT and falls. COVID-19, spring, summer, fall or winter did not affect the association.

CONCLUSIONS

Our results show an association between JHFRAT and falls, a low discrimination by JHFRAT for older inpatients and over-prediction in the calibration. Improvements in the fall-risk assessment are warranted to improve efficiency.

Developing EMR-based algorithms to Identify hospital adverse events for health system performance evaluation and improvement: Study protocol

Background

Measurement of care quality and safety mainly relies on abstracted administrative data. However, it is well studied that administrative data-based adverse event (AE) detection methods are suboptimal due to lack of clinical information. Electronic medical records (EMR) have been widely implemented and contain detailed and comprehensive information regarding all aspects of patient care, offering a valuable complement to administrative data. Harnessing the rich clinical data in EMRs offers a unique opportunity to improve detection, identify possible risk factors of AE and enhance surveillance. However, the methodological tools for detection of AEs within EMR need to be developed and validated. The objectives of this study are to develop EMR-based AE algorithms from hospital EMR data and assess AE algorithm’s validity in Canadian EMR data.

Methods

Patient EMR structured and text data from acute care hospitals in Calgary, Alberta, Canada will be linked with discharge abstract data (DAD) between 2010 and 2020 (n~1.5 million). AE algorithms development. First, a comprehensive list of AEs will be generated through a systematic literature review and expert recommendations. Second, these AEs will be mapped to EMR free texts using Natural Language Processing (NLP) technologies. Finally, an expert panel will assess the clinical relevance of the developed NLP algorithms. AE algorithms validation: We will test the newly developed AE algorithms on 10,000 randomly selected EMRs between 2010 to 2020 from Calgary, Alberta. Trained reviewers will review the selected 10,000 EMR charts to identify AEs that had occurred during hospitalization. Performance indicators (e.g., sensitivity, specificity, positive predictive value, negative predictive value, F₁ score, etc.) of the developed AE algorithms will be assessed using chart review data as the reference standard.

Discussion

The results of this project can be widely implemented in EMR based healthcare system to accurately and timely detect in-hospital AEs.

Evaluating Patients' Experiences with Healthcare Services: Extracting Domain and Language-Specific Information from Free-Text Narratives

Evaluating patients’ experience and satisfaction often calls for analyses of free-text data. Language and domain-specific information extraction can reduce costly manual preprocessing and enable the analysis of extensive collections of experience-based narratives. The research aims were to (1) elicit free-text narratives about experiences with health services of international students in Poland, (2) develop domain- and language-specific algorithms for the extraction of information relevant for the evaluation of quality and safety of health services, and (3) test the performance of information extraction algorithms’ on questions about the patients’ experiences with health services. The materials were free-text narratives about health clinic encounters produced by English-speaking foreigners recalling their experiences (n = 104) in healthcare facilities in Poland. A linguistic analysis of the text collection led to constructing a semantic−syntactic lexicon and a set of lexical-syntactic frames. These were further used to develop rule-based information extraction algorithms in the form of Python scripts. The extraction algorithms generated text classifications according to predefined queries. In addition, the narratives were classified by human readers. The algorithm-based and the human readers’ classifications were highly correlated and significant (p < 0.01), indicating an excellent performance of the automatic query algorithms. The study results demonstrate that domain-specific and language-specific information extraction from free-text narratives can be used as an efficient and low-cost method for evaluating patient experiences and satisfaction with health services and built into software solutions for the quality evaluation in health care.

A hybrid model to identify fall occurrence from electronic health records

INTRODUCTION

Falls are a leading cause of unintentional injury in the elderly. Electronic health records (EHRs) offer the unique opportunity to develop models that can identify fall events. However, identifying fall events in clinical notes requires advanced natural language processing (NLP) to simultaneously address multiple issues because the word "fall" is a typical homonym.

METHODS

We implemented a context-aware language model, Bidirectional Encoder Representations from Transformers (BERT) to identify falls from the EHR text and further fused the BERT model into a hybrid architecture coupled with post-hoc heuristic rules to enhance the performance. The models were evaluated on real world EHR data and were compared to conventional rule-based and deep learning models (CNN and Bi-LSTM). To better understand the ability of each approach to identify falls, we further categorize fall-related concepts (i.e., risk of fall, prevention of fall, homonym) and performed a detailed error analysis.

RESULTS

The hybrid model achieved the highest f1-score on sentence (0.971), document (0.985), and patient (0.954) level. At the sentence level (basic data unit in the model), the hybrid model had 0.954, 1.000, 0.988, and 0.999 in sensitivity, specificity, positive predictive value, and negative predictive value, respectively. The error analysis showed that that machine learning-based approaches demonstrated higher performance than a rule-based approach in challenging cases that required contextual understanding. The context-aware language model (BERT) slightly outperformed the word embedding approach trained on Bi-LSTM. No single model yielded the best performance for all fall-related semantic categories.

CONCLUSION

A context-aware language model (BERT) was able to identify challenging fall events that requires context understanding in EHR free text. The hybrid model combined with post-hoc rules allowed a custom fix on the BERT outcomes and further improved the performance of fall detection.

Potentially inappropriate medications and their effect on falls during hospital admission

OBJECTIVE

to investigate the effect of potentially inappropriate medications (PIMs) on inpatient falls and to identify whether PIMs as defined by STOPPFall or the designated section K for falls of STOPP v2 have a stronger association with inpatient falls when compared to the general tool STOPP v2.

METHODS

a retrospective observational matching study using an electronic health records dataset of patients (≥70 years) admitted to an academic hospital (2015-19), including free text to identify inpatient falls. PIMs were identified using the STOPP v2, section K of STOPP v2 and STOPPFall. We first matched admissions with PIMs to those without PIMs on confounding factors. We then applied multinomial logistic regression analysis and Cox proportional hazards analysis on the matched datasets to identify effects of PIMs on inpatient falls.

RESULTS

the dataset included 16,678 hospital admissions, with a mean age of 77.2 years. Inpatient falls occurred during 446 (2.7%) admissions. Adjusted odds ratio (OR) (95% confidence interval (CI)) for the association between PIM exposure and falls were 7.9 (6.1-10.3) for STOPP section K, 2.2 (2.0-2.5) for STOPP and 1.4 (1.3-1.5) for STOPPFall. Adjusted hazard ratio (HR) (95% CI) for the effect on time to first fall were 2.8 (2.3-3.5) for STOPP section K, 1.5 (1.3-1.6) for STOPP and 1.3 (1.2-1.5) for STOPPFall.

CONCLUSIONS

we identified an independent association of PIMs on inpatient falls for all applied (de)prescribing tools. The strongest effect was identified for STOPP section K, which is restricted to high-risk medication for falls. Our results suggest that decreasing PIM exposure during hospital stay might benefit fall prevention, but intervention studies are warranted.

A Keyword Approach to Identify Adverse Events Within Narrative Documents From 4 Italian Institutions

OBJECTIVES

Existing methods for measuring adverse events in hospitals intercept a restricted number of events. Text mining refers to a range of techniques to extract data from narrative sources. The goal of this study was to evaluate the performance of an automated approach for extracting adverse event keywords from within electronic health records.

METHODS

The study involved 4 medical centers in the Region of Lombardy. A starting set of keywords was trained in an iterative process to develop queries for 7 adverse events, including those used by the Agency for Healthcare Research and Quality as patient safety indicators. We calculated positive predictive values of the 7 queries and performed an error analysis to detect reasons for false-positive cases of pulmonary embolism, deep vein thrombosis, and urinary tract infection.

RESULTS

Overall, 397,233 records were collected (34,805 discharge summaries, 292,593 emergency department notes, and 69,835 operation reports). Positive predictive values were higher for postoperative wound dehiscence (83.83%) and urinary tract infection (73.07%), whereas they were lower for deep vein thrombosis (5.37%), pulmonary embolism (13.63%), and postoperative sepsis (12.28%). The most common reasons for false positives were reporting of past events (42.25%), negations (22.80%), and conditions suspected by physicians but not confirmed by a diagnostic test (11.25%).

CONCLUSIONS

The results of our study demonstrated the feasibility of using an automated approach to detect multiple adverse events in several data sources. More sophisticated techniques, such as natural language processing, should be tested to evaluate the feasibility of using text mining as a routine method for monitoring adverse events in hospitals.

Machine learning and natural language processing to identify falls in electronic patient care records from ambulance attendances

We derived machine learning models utilizing features generated by natural language processing (NLP) of free-text data from an ambulance services provider to identify fall cases. The data comprised samples of electronic patient care records care records (ePCRs) from St John Western Australia (WA), the sole ambulance services provider in most of WA. We manually labeled fall cases by reviewing the free-text summary. The models used features including case characteristics (e.g., age) and text frequency-inverse document frequency (tf-idf) of each word of the free-text generated by NLP. Support vector machine (SVM) and random forest were used as classifiers. We compared the performance of the models against the manual identification of falls by recall, precision, and F-measure. A total of 9,447 cases (1%) were randomly sampled, of which 1,648 (17%) were labeled as fall. The best model was an SVM model using case characteristics and tf-idf's of the first 100 words of free-text, with recall of 0.84, precision of 0.86, and F-measure of 0.85. This performance was better than an SVM model with only case characteristics. Machine-learning models incorporated with features generated by NLP improved the performance of classifying fall cases compared with models without such features. Scope remains for further improvement.

Content Analysis of Patient Safety Incident Reports for Older Adult Patient Transfers, Handovers, and Discharges: Do They Serve Organizations, Staff, or Patients?

OBJECTIVE

The aim of the study was to analyze content of incident reports during patient transitions in the context of care of older people, cardiology, orthopedics, and stroke.

METHODS

A structured search strategy identified incident reports involving patient transitions (March 2014-August 2014, January 2015-June 2015) within 2 National Health Service Trusts (in upper and lower quartiles of incident reports/100 admissions) in care of older people, cardiology, orthopedics, and stroke. Content analysis identified the following: incident classifications; active failures; latent conditions; patient/relative involvement; and evidence of individual or organizational learning. Reported harm was interpreted with reference to National Reporting and Learning System criteria.

RESULTS

A total 278 incident reports were analyzed. Fourteen incident classifications were identified, with pressure ulcers the modal category (n = 101,36%), followed by falls (n = 32, 12%), medication (n = 31, 11%), and documentation (n = 29, 10%). Half (n = 139, 50%) of incident reports related to interunit/department/team transfers. Latent conditions were explicit in 33 (12%) reports; most frequently, these related to inadequate resources/staff and concomitant time pressures (n = 13). Patient/family involvement was explicit in 61 (22%) reports. Patient well-being was explicit in 24 (9%) reports. Individual and organizational learning was evident in 3% and 7% of reports, respectively. Reported harm was significantly lower than coder-interpreted harm (P < 0.0001).

CONCLUSIONS

Incident report quality was suboptimal for individual and organizational learning. Underreporting level of harm suggests reporter bias, which requires reducing as much as practicable. System-level interventions are warranted to encourage use of staff reflective skills, emphasizing joint ownership of incidents. Co-producing incident reports with other clinicians involved in the transition and patients/relatives could optimize organizational learning.

Reliability of Patient-Report, Physician-Report, and Medical Record Review to Identify Hospital-Acquired Complications: A Prospective Cohort Study

This prospective study of internal medicine inpatients treated at 2 hospitals in Toronto, Canada, between September 1, 2016, and September 1, 2017, compared patient-report, physician-report, and detailed medical record review to identify specific hospital-acquired complications. Six complications were assessed: delirium, catheter-associated urinary tract infection, acute kidney injury, deep vein thrombosis/pulmonary embolism, hospital-acquired pneumonia, or fall. The study included 207 patients and physician responses were obtained for 156 (75%). Complications were identified in 28 (14%) patients by medical record review, 30 (14%) patients by patient-report, and 11 (7%) patients by physician-report. Fifty-four (26%) patients experienced a complication as identified through at least one of the 3 methods. There was little agreement between the 3 methods (Fleiss' ĸ 0.15, P < 0.001). All 3 sources agreed on the occurrence of a specific complication in only 1 patient (1%). Multiple approaches likely are needed to adequately measure hospital-acquired complications.

The incidence and nature of adverse events in rehabilitation inpatients with acquired brain injuries

INTRODUCTION

Patient safety is important in all healthcare settings. Few studies have examined the state of patient safety in rehabilitation and none have examined patient safety in the setting of acquired brain injury (ABI) rehabilitation.

OBJECTIVES

To determine the incidence, most common types, and severities of adverse events among inpatients undergoing ABI rehabilitation.

DESIGN

Retrospective case series descriptive study.

SETTING

The inpatient ABI rehabilitation program at an academic, tertiary rehabilitation hospital in Canada.

PARTICIPANTS

108 consecutive inpatients with acquired brain injuries.

INTERVENTIONS

Not applicable.

MAIN OUTCOME MEASURE(S)

Patient charts and incident reports from the hospital's voluntary reporting system were reviewed by three board-certified physiatrists to determine the incidence, type, severity and preventability of adverse events. Adverse events were identified and classified for severity and type using the WHO International Classification for Patient Safety. Preventability was rated on a 6-point Likert scale.

RESULTS

During the study period, the incidence of adverse events was 17.42 ± 3.86 per 1000 patient days. Adverse events affected 52.8% of patients. Most adverse events identified were mild in severity (98.51%) and the rest were of moderate severity. The two most common types of adverse events were 1) patient incidents (56.72%) such as falls, pressure ulcers and skin tears, and 2) patient behaviors such as missing patient, assault, or sexual behaviors (16.42%). Of the 80 adverse events identified in the study, 44.78% were preventable. The hospital's voluntary reporting system did not capture 57.9% of the adverse events identified.

CONCLUSIONS

Future efforts to improve patient safety in ABI rehab should focus on reducing falls, skin injuries and behaviors, and removing barriers to voluntary incident reporting. Detecting adverse events through chart reviews provide a more complete understanding of patient safety risks in ABI rehab than relying on incident reporting alone. This article is protected by copyright. All rights reserved.

Identification of Inpatient Falls Using Automated Review of Text-Based Medical Records

OBJECTIVES

Although falls are among the most common adverse event in hospitals, they are difficult to measure and often unreported. Mechanisms to track falls include incident reporting and medical records review. Because of limitations of each method, researchers suggest multimodal approaches. Although incident reporting is commonly used, medical records review is limited by the need to read a high volume of clinical notes. Natural language processing (NLP) is 1 potential mechanism to automate this process.

METHOD

We compared automated NLP to manual chart review and incident reporting as a method to detect falls among inpatients. First, we developed an NLP algorithm to identify inpatient progress notes describing falls. Second, we compared the NLP algorithm to manual records review in identifying inpatient progress notes that describe falls. Third, we compared the NLP algorithm to the incident reporting system in identifying falls.

RESULTS

When examining individual inpatient notes, our NLP algorithm was highly specific (0.97) but had low sensitivity (0.44) when compared with our manual records review. However, when considering groups of inpatient notes, all describing the same fall, our NLP algorithm had a large improvement in sensitivity (0.80) with some loss of specificity (0.65) compared with incident reporting.

CONCLUSIONS

National language processing represents a promising method to automate review of inpatient medical records to identify falls.

Nurse staffing practices and adverse events in acute care hospitals: The research protocol of a multisite patient-level longitudinal study

Aims

We describe an innovative research protocol to: (a) examine patient‐level longitudinal associations between nurse staffing practices and the risk of adverse events in acute care hospitals and; (b) determine possible thresholds for safe nurse staffing.

Design

A dynamic cohort of adult medical, surgical and intensive care unit patients admitted to 16 hospitals in Quebec (Canada) between January 2015–December 2019.

Methods

Patients in the cohort will be followed from admission until 30‐day postdischarge to assess exposure to selected nurse staffing practices in relation to the subsequent occurrence of adverse events. Five staffing practices will be measured for each shift of an hospitalization episode, using electronic payroll data, with the following time‐varying indicators: (a) nursing worked hours per patient; (b) skill mix; (c) overtime use; (d) education mix and; and (e) experience. Four high‐impact adverse events, presumably associated with nurse staffing practices, will be measured from electronic health record data retrieved at the participating sites: (a) failure‐to‐rescue; (b) in‐hospital falls; (c) hospital‐acquired pneumonia and; and (d) venous thromboembolism. To examine the associations between the selected nurse staffing exposures and the risk of each adverse event, separate multivariable Cox proportional hazards frailty regression models will be fitted, while adjusting for patient, nursing unit and hospital characteristics, and for clustering. To assess for possible staffing thresholds, flexible non‐linear spline functions will be fitted. Funding for the study began in October 2019 and research ethics/institutional approval was granted in February 2020.

Discussion

To our knowledge, this study is the first multisite patient‐level longitudinal investigation of the associations between common nurse staffing practices and the risk of adverse events. It is hoped that our results will assist hospital managers in making the most effective use of the scarce nursing resources and in identifying staffing practices that minimize the occurrence of adverse events.

Automated Fall Detection Algorithm With Global Trigger Tool, Incident Reports, Manual Chart Review, and Patient-Reported Falls: Algorithm Development and Validation With a Retrospective Diagnostic Accuracy Study

BACKGROUND

Falls are common adverse events in hospitals, frequently leading to additional health costs due to prolonged stays and extra care. Therefore, reliable fall detection is vital to develop and test fall prevention strategies. However, conventional methods-voluntary incident reports and manual chart reviews-are error-prone and time consuming, respectively. Using a search algorithm to examine patients' electronic health record data and flag fall indicators offers an inexpensive, sensitive, cost-effective alternative.

OBJECTIVE

This study's purpose was to develop a fall detection algorithm for use with electronic health record data, then to evaluate it alongside the Global Trigger Tool, incident reports, a manual chart review, and patient-reported falls.

METHODS

Conducted on 2 campuses of a large hospital system in Switzerland, this retrospective diagnostic accuracy study consisted of 2 substudies: the first, targeting 240 patients, for algorithm development and the second, targeting 298 patients, for validation. In the development study, we compared the new algorithm's in-hospital fall rates with those indicated by the Global Trigger Tool and incident reports; in the validation study, we compared the algorithm's in-hospital fall rates with those from patient-reported falls and manual chart review. We compared the various methods by calculating sensitivity, specificity, and predictive values.

RESULTS

Twenty in-hospital falls were discovered in the development study sample. Of these, the algorithm detected 19 (sensitivity 95%), the Global Trigger Tool detected 18 (90%), and incident reports detected 14 (67%). Of the 15 falls found in the validation sample, the algorithm identified all 15 (100%), the manual chart review identified 14 (93%), and the patient-reported fall measure identified 5 (33%). Owing to relatively high numbers of false positives based on falls present on admission, the algorithm's positive predictive values were 50% (development sample) and 47% (validation sample). Instead of requiring 10 minutes per case for a full manual review or 20 minutes to apply the Global Trigger Tool, the algorithm requires only a few seconds, after which only the positive results (roughly 11% of the full case number) require review.

CONCLUSIONS

The newly developed electronic health record algorithm demonstrated very high sensitivity for fall detection. Applied in near real time, the algorithm can record in-hospital falls events effectively and help to develop and test fall prevention measures.

Comparison of deep learning models for natural language processing-based classification of non-English head CT reports

PURPOSE

Natural language processing (NLP) can be used for automatic flagging of radiology reports. We assessed deep learning models for classifying non-English head CT reports.

METHODS

We retrospectively collected head CT reports (2011-2018). Reports were signed in Hebrew. Emergency department (ED) reports of adult patients from January to February for each year (2013-2018) were manually labeled. All other reports were used to pre-train an embedding layer. We explored two use cases: (1) general labeling use case, in which reports were labeled as normal vs. pathological; (2) specific labeling use case, in which reports were labeled as with and without intra-cranial hemorrhage. We tested long short-term memory (LSTM) and LSTM-attention (LSTM-ATN) networks for classifying reports. We also evaluated the improvement of adding Word2Vec word embedding. Deep learning models were compared with a bag-of-words (BOW) model.

RESULTS

We retrieved 176,988 head CT reports for pre-training. We manually labeled 7784 reports as normal (46.3%) or pathological (53.7%), and 7.1% with intra-cranial hemorrhage. For the general labeling, LSTM-ATN-Word2Vec showed the best results (AUC = 0.967 ± 0.006, accuracy 90.8% ± 0.01). For the specific labeling, all methods showed similar accuracies between 95.0 and 95.9%. Both LSTM-ATN-Word2Vec and BOW had the highest AUC (0.970).

CONCLUSION

For a general use case, word embedding using a large cohort of non-English head CT reports and ATN improves NLP performance. For a more specific task, BOW and deep learning showed similar results. Models should be explored and tailored to the NLP task.

A systematic review of natural language processing for classification tasks in the field of incident reporting and adverse event analysis

CONTEXT

Adverse events in healthcare are often collated in incident reports which contain unstructured free text. Learning from these events may improve patient safety. Natural language processing (NLP) uses computational techniques to interrogate free text, reducing the human workload associated with its analysis. There is growing interest in applying NLP to patient safety, but the evidence in the field has not been summarised and evaluated to date.

OBJECTIVE

To perform a systematic literature review and narrative synthesis to describe and evaluate NLP methods for classification of incident reports and adverse events in healthcare.

METHODS

Data sources included Medline, Embase, The Cochrane Library, CINAHL, MIDIRS, ISI Web of Science, SciELO, Google Scholar, PROSPERO, hand searching of key articles, and OpenGrey. Data items were manually abstracted to a standardised extraction form.

RESULTS

From 428 articles screened for eligibility, 35 met the inclusion criteria of using NLP to perform a classification task on incident reports, or with the aim of detecting adverse events. The majority of studies used free text from incident reporting systems or electronic health records. Models were typically designed to classify by type of incident, type of medication error, or harm severity. A broad range of NLP techniques are demonstrated to perform these classification tasks with favourable performance outcomes. There are methodological challenges in how these results can be interpreted in a broader context.

CONCLUSION

NLP can generate meaningful information from unstructured data in the specific domain of the classification of incident reports and adverse events. Understanding what or why incidents are occurring is important in adverse event analysis. If NLP enables these insights to be drawn from larger datasets it may improve the learning from adverse events in healthcare.

Improving Prediction of Fall Risk Using Electronic Health Record Data With Various Types and Sources at Multiple Times

Inpatient falls are among the most common adverse events threatening patient safety. Although many studies have developed predictive models for fall risk, there are some drawbacks. First, most previous studies have relied on an incident-reporting system alone to identify fall events. Thus, it has been found that falls are more likely to be underreported. Second, there has been a controversy on how to select accurate representative values for patient status data across multiple times and various data sources in electronic health records. Given this background, this study used nurses' progress notes as a complementary data source to detect fall events. In addition, we developed criteria including coverage, currency, and granularity in order to integrate electronic health records data documented at multiple times in various data types and sources. Based on this methodology, we developed three models, logistic regression, Cox proportional hazard regression, and decision tree, to predict risk of patient falls and evaluate the predictive performance of these models by comparing the results to results from the Hendrich II Fall Risk Model. The findings of this study will be used in a clinical decision support system to predict risk of falling and provide evidence-based tailored recommendations in the future.

Mining fall-related information in clinical notes: Comparison of rule-based and novel word embedding-based machine learning approaches

BACKGROUND

Natural language processing (NLP) of health-related data is still an expertise demanding, and resource expensive process. We created a novel, open source rapid clinical text mining system called NimbleMiner. NimbleMiner combines several machine learning techniques (word embedding models and positive only labels learning) to facilitate the process in which a human rapidly performs text mining of clinical narratives, while being aided by the machine learning components.

OBJECTIVE

This manuscript describes the general system architecture and user Interface and presents results of a case study aimed at classifying fall-related information (including fall history, fall prevention interventions, and fall risk) in homecare visit notes.

METHODS

We extracted a corpus of homecare visit notes (n = 1,149,586) for 89,459 patients from a large US-based homecare agency. We used a gold standard testing dataset of 750 notes annotated by two human reviewers to compare the NimbleMiner's ability to classify documents regarding whether they contain fall-related information with a previously developed rule-based NLP system.

RESULTS

NimbleMiner outperformed the rule-based system in almost all domains. The overall F- score was 85.8% compared to 81% by the rule based-system with the best performance for identifying general fall history (F = 89% vs. F = 85.1% rule-based), followed by fall risk (F = 87% vs. F = 78.7% rule-based), fall prevention interventions (F = 88.1% vs. F = 78.2% rule-based) and fall within 2 days of the note date (F = 83.1% vs. F = 80.6% rule-based). The rule-based system achieved slightly better performance for fall within 2 weeks of the note date (F = 81.9% vs. F = 84% rule-based).

DISCUSSION & CONCLUSIONS

NimbleMiner outperformed other systems aimed at fall information classification, including our previously developed rule-based approach. These promising results indicate that clinical text mining can be implemented without the need for large labeled datasets necessary for other types of machine learning. This is critical for domains with little NLP developments, like nursing or allied health professions.

Identifying Falls Risk Screenings Not Documented with Administrative Codes Using Natural Language Processing

Quality reporting that relies on coded administrative data alone may not completely and accurately depict providers' performance. To assess this concern with a test case, we developed and evaluated a natural language processing (NLP) approach to identify falls risk screenings documented in clinical notes of patients without coded falls risk screening data. Extracting information from 1,558 clinical notes (mainly progress notes) from 144 eligible patients, we generated a lexicon of 38 keywords relevant to falls risk screening, 26 terms for pre-negation, and 35 terms for post-negation. The NLP algorithm identified 62 (out of the 144) patients who falls risk screening documented only in clinical notes and not coded. Manual review confirmed 59 patients as true positives and 77 patients as true negatives. Our NLP approach scored 0.92 for precision, 0.95 for recall, and 0.93 for F-measure. These results support the concept of utilizing NLP to enhance healthcare quality reporting.

Characteristics of Inpatient Falls not Reported in an Incident Reporting System

An incident reporting system is the most commonly used method to identify patient safety incidents in a hospital. However, non-reporting of incidents for various reasons is a serious problem. We studied the rate of inpatient falls that were not reported in an incident reporting system but were recorded in medical charts and we evaluated characteristics of those falls by comparing with the falls reported in incident reports in a Japanese acute care hospital setting. Falls recorded in medical charts were detected by using a text mining method followed by a manual chart review. About 25% of the recorded falls were not reported in incident reports. Male patients, first fall, long lag time until recording, no witness at the time of the fall and physician profession were shown to be significant factors associated with non-reporting. Our results show that the rate of non-reporting of inpatient falls in a Japanese acute care hospital is compable to that shown in previous studies in other conutries and that the same barriers to incident reporting as those found in previous studies exist in the medical staff.

Improving patient safety by optimizing the use of nursing human resources

Background

Recent ecological studies have suggested that inadequate nurse staffing may contribute to the incidence of adverse events in acute care hospitals. However, longitudinal studies are needed to further examine these associations and to identify the staffing patterns that are of greatest risk. The aims of this study are to determine if (a) nurse staffing levels are associated with an increased risk of adverse events, (b) the risk of adverse events in relationship to nurse staffing levels is modified by the complexity of patient requirements, and (c) optimal nurse staffing levels can be established.

Methods/design

A dynamic cohort of all adult medical, surgical, and intensive care unit patients admitted between 2010 and 2015 to a Canadian academic health center will be followed during the inpatient and 7-day post-discharge period to assess the occurrence and frequency of adverse events in relationship to antecedent nurse staffing levels. Four potentially preventable adverse events will be measured: (a) hospital-acquired pneumonia, (b) ventilator-associated pneumonia, (c) venous thromboembolism, and (d) in-hospital fall. These events were selected for their high incidence, morbidity and mortality rates, and because they are hypothesized to be related to nurse staffing levels. Adverse events will be ascertained from electronic health record data using validated automated detection algorithms. Patient exposure to nurse staffing will be measured on every shift of the hospitalization using electronic payroll records. To examine the association between nurse staffing levels and the risk of adverse events, four Cox proportional hazards regression models will be used (one for each adverse event), while adjusting for patient characteristics and risk factors of adverse event occurrence. To determine if the association between nurse staffing levels and the occurrence of adverse events is modified by the complexity of patient requirements, interaction terms will be included in the regression models, and their significance assessed. To assess for the presence of optimal nurse staffing levels, flexible nonlinear spline functions will be fitted.

Discussion

This study will likely generate evidence-based information that will assist managers in making the most effective use of scarce nursing resources and in identifying staffing patterns that minimize the risk of adverse events.

Electronic supplementary material

The online version of this article (doi:10.1186/s13012-015-0278-1) contains supplementary material, which is available to authorized users.

Accuracy of using automated methods for detecting adverse events from electronic health record data: a research protocol

Background

Adverse events are associated with significant morbidity, mortality and cost in hospitalized patients. Measuring adverse events is necessary for quality improvement, but current detection methods are inaccurate, untimely and expensive. The advent of electronic health records and the development of automated methods for encoding and classifying electronic narrative data, such as natural language processing, offer an opportunity to identify potentially better methods. The objective of this study is to determine the accuracy of using automated methods for detecting three highly prevalent adverse events: a) hospital-acquired pneumonia, b) catheter-associated bloodstream infections, and c) in-hospital falls.

Methods/design

This validation study will be conducted at two large Canadian academic health centres: the McGill University Health Centre (MUHC) and The Ottawa Hospital (TOH). The study population consists of all medical, surgical and intensive care unit patients admitted to these centres between 2008 and 2014. An automated detection algorithm will be developed and validated for each of the three adverse events using electronic data extracted from multiple clinical databases. A random sample of MUHC patients will be used to develop the automated detection algorithms (cohort 1, development set). The accuracy of these algorithms will be assessed using chart review as the reference standard. Then, receiver operating characteristic curves will be used to identify optimal cut points for each of the data sources. Multivariate logistic regression and the areas under curve (AUC) will be used to identify the optimal combination of data sources that maximize the accuracy of adverse event detection. The most accurate algorithms will then be validated on a second random sample of MUHC patients (cohort 1, validation set), and accuracy will be measured using chart review as the reference standard. The most accurate algorithms validated at the MUHC will then be applied to TOH data (cohort 2), and their accuracy will be assessed using a reference standard assessment of the medical chart.

Discussion

There is a need for more accurate, timely and efficient measures of adverse events in acute care hospitals. This is a critical requirement for evaluating the effectiveness of preventive interventions and for tracking progress in patient safety through time.

Assisted and unassisted falls: different events, different outcomes, different implications for quality of hospital care

BACKGROUND

Many hospitals classify inpatient falls as assisted (if a staff member is present to ease the patient's descent or break the fall) or unassisted for quality measurement purposes. Unassisted falls are more likely to result in injury, but there is limited research quantifying this effect or linking the assisted/unassisted classification to processes of care. A study was conducted to link the assisted/unassisted fall classification to both processes and outcomes of care, thereby demonstrating its suitability for use in quality measurement. This was only the second known published study to quantify the increased risk of injury associated with falling unassisted (versus assisted), and the first to estimate the effects of falling unassisted (versus assisted) on the likelihood of specific levels of injury.

METHODS

A cross-sectional analysis of falls from all available 2011 data for 6,539 adult medical, surgical, and medical-surgical units in 1,464 general hospitals participating in the National Database of Nursing Quality Indicators" (NDNQI) was performed.

RESULTS

Participating units reported 166,883 falls (3.44 falls per 1,000 patient-days). Excluding repeat falls, 85.5% of falls were unassisted. Assisted and unassisted falls were associated with different processes and outcomes: Fallers for whom a fall prevention protocol was not in place were more likely to fall unassisted than those for whom one was in place (adjusted odds ratio [aOR], 1.39 [95% confidence interval (CI), 1.32, 1.46]), and unassisted falls were more likely to result in injury (aOR, 1.59 [95% CI, 1.52, 1.67]).

CONCLUSIONS

The assisted/unassisted fall classification is associated with care processes and patient outcomes, making it suitable for quality measurement. Unassisted falls are more likely than assisted falls to result in injury and should be considered a target for future prevention efforts.

Agreement of Medicaid claims and electronic health records for assessing preventive care quality among adults

To compare the agreement of electronic health record (EHR) data versus Medicaid claims data in documenting adult preventive care. Insurance claims are commonly used to measure care quality. EHR data could serve this purpose, but little information exists about how this source compares in service documentation. For 13 101 Medicaid-insured adult patients attending 43 Oregon community health centers, we compared documentation of 11 preventive services, based on EHR versus Medicaid claims data. Documentation was comparable for most services. Agreement was highest for influenza vaccination (κ =  0.77; 95% CI 0.75 to 0.79), cholesterol screening (κ = 0.80; 95% CI 0.79 to 0.81), and cervical cancer screening (κ = 0.71; 95% CI 0.70 to 0.73), and lowest on services commonly referred out of primary care clinics and those that usually do not generate claims. EHRs show promise for use in quality reporting. Strategies to maximize data capture in EHRs are needed to optimize the use of EHR data for service documentation.

Development of a risk assessment tool to predict fall-related severe injuries occurring in a hospital

Inpatient falls are the most common adverse events that occur in a hospital, and about 3 to 10% of falls result in serious injuries such as bone fractures and intracranial haemorrhages. We previously reported that bone fractures and intracranial haemorrhages were two major fall-related injuries and that risk assessment score for osteoporotic bone fracture was significantly associated not only with bone fractures after falls but also with intracranial haemorrhage after falls. Based on the results, we tried to establish a risk assessment tool for predicting fall-related severe injuries in a hospital. Possible risk factors related to fall-related serious injuries were extracted from data on inpatients that were admitted to a tertiary-care university hospital by using multivariate Cox' s regression analysis and multiple logistic regression analysis. We found that fall risk score and fracture risk score were the two significant factors, and we constructed models to predict fall-related severe injuries incorporating these factors. When the prediction model was applied to another independent dataset, the constructed model could detect patients with fall-related severe injuries efficiently. The new assessment system could identify patients prone to severe injuries after falls in a reproducible fashion.

Public hospital quality report awareness: evidence from National and Californian Internet searches and social media mentions, 2012

OBJECTIVES

Publicly available hospital quality reports seek to inform consumers of important healthcare quality and affordability attributes, and may inform consumer decision-making. To understand how much consumers search for such information online on one Internet search engine, whether they mention such information in social media and how positively they view this information.

SETTING AND DESIGN

A leading Internet search engine (Google) was the main focus of the study. Google Trends and Google Adwords keyword analyses were performed for national and Californian searches between 1 August 2012 and 31 July 2013 for keywords related to 'top hospital', best hospital', and 'hospital quality', as well as for six specific hospital quality reports. Separately, a proprietary social media monitoring tool was used to investigate blog, forum, social media and traditional media mentions of, and sentiment towards, major public reports of hospital quality in California in 2012.

PRIMARY OUTCOME MEASURES

(1) Counts of searches for keywords performed on Google; (2) counts of and (3) sentiment of mentions of public reports on social media.

RESULTS

National Google search volume for 75 hospital quality-related terms averaged 610 700 searches per month with strong variation by keyword and by state. A commercial report (Healthgrades) was more commonly searched for nationally on Google than the federal government's Hospital Compare, which otherwise dominated quality-related search terms. Social media references in California to quality reports were generally few, and commercially produced hospital quality reports were more widely mentioned than state (Office of Statewide Healthcare Planning and Development (OSHPD)), or non-profit (CalHospitalCompare) reports.

CONCLUSIONS

Consumers are somewhat aware of hospital quality based on Internet search activity and social media disclosures. Public stakeholders may be able to broaden their quality dissemination initiatives by advertising on Google or Twitter and using social media interactively with consumers looking for relevant information.