Systematic review and longitudinal analysis of implementing Artificial Intelligence to predict clinical deterioration in adult hospitals: what is known and what remains uncertain

OBJECTIVE

To identify factors influencing implementation of machine learning algorithms (MLAs) that predict clinical deterioration in hospitalized adult patients and relate these to a validated implementation framework.

MATERIALS AND METHODS

A systematic review of studies of implemented or trialed real-time clinical deterioration prediction MLAs was undertaken, which identified: how MLA implementation was measured; impact of MLAs on clinical processes and patient outcomes; and barriers, enablers and uncertainties within the implementation process. Review findings were then mapped to the SALIENT end-to-end implementation framework to identify the implementation stages at which these factors applied.

RESULTS

Thirty-seven articles relating to 14 groups of MLAs were identified, each trialing or implementing a bespoke algorithm. One hundred and seven distinct implementation evaluation metrics were identified. Four groups reported decreased hospital mortality, 1 significantly. We identified 24 barriers, 40 enablers, and 14 uncertainties and mapped these to the 5 stages of the SALIENT implementation framework.

DISCUSSION

Algorithm performance across implementation stages decreased between in silico and trial stages. Silent plus pilot trial inclusion was associated with decreased mortality, as was the use of logistic regression algorithms that used less than 39 variables. Mitigation of alert fatigue via alert suppression and threshold configuration was commonly employed across groups.

CONCLUSIONS

: There is evidence that real-world implementation of clinical deterioration prediction MLAs may improve clinical outcomes. Various factors identified as influencing success or failure of implementation can be mapped to different stages of implementation, thereby providing useful and practical guidance for implementers.

Surgery's Rosetta Stone: Natural language processing to predict discharge and readmission after general surgery

BACKGROUND

This study aimed to examine the accuracy with which multiple natural language processing artificial intelligence models could predict discharge and readmissions after general surgery.

METHODS

Natural language processing models were derived and validated to predict discharge within the next 48 hours and 7 days and readmission within 30 days (based on daily ward round notes and discharge summaries, respectively) for general surgery inpatients at 2 South Australian hospitals. Natural language processing models included logistic regression, artificial neural networks, and Bidirectional Encoder Representations from Transformers.

RESULTS

For discharge prediction analyses, 14,690 admissions were included. For readmission prediction analyses, 12,457 patients were included. For prediction of discharge within 48 hours, derivation and validation data set area under the receiver operator characteristic curves were, respectively: 0.86 and 0.86 for Bidirectional Encoder Representations from Transformers, 0.82 and 0.81 for logistic regression, and 0.82 and 0.81 for artificial neural networks. For prediction of discharge within 7 days, derivation and validation data set area under the receiver operator characteristic curves were, respectively: 0.82 and 0.81 for Bidirectional Encoder Representations from Transformers, 0.75 and 0.72 for logistic regression, and 0.68 and 0.67 for artificial neural networks. For readmission prediction within 30 days, derivation and validation data set area under the receiver operator characteristic curves were, respectively: 0.55 and 0.59 for Bidirectional Encoder Representations from Transformers and 0.77 and 0.62 for logistic regression.

CONCLUSION

Modern natural language processing models, particularly Bidirectional Encoder Representations from Transformers, can effectively and accurately identify general surgery patients who will be discharged in the next 48 hours. However, these approaches are less capable of identifying general surgery patients who will be discharged within the next 7 days or who will experience readmission within 30 days of discharge.

The Adelaide Score: An artificial intelligence measure of readiness for discharge after general surgery

BACKGROUND

This study aimed to examine the performance of machine learning algorithms for the prediction of discharge within 12 and 24 h to produce a measure of readiness for discharge after general surgery.

METHODS

Consecutive general surgery patients at two tertiary hospitals, over a 2-year period, were included. Observation and laboratory parameter data were stratified into training, testing and validation datasets. Random forest, XGBoost and logistic regression models were evaluated. Each ward round note time was taken as a different event. Primary outcome was classification accuracy of the algorithmic model able to predict discharge within the next 12 h on the validation data set.

RESULTS

42 572 ward round note timings were included from 8826 general surgery patients. Discharge occurred within 12 h for 8800 times (20.7%), and within 24 h for 9885 (23.2%). For predicting discharge within 12 h, model classification accuracies for derivation and validation data sets were: 0.84 and 0.85 random forest, 0.84 and 0.83 XGBoost, 0.80 and 0.81 logistic regression. For predicting discharge within 24 h, model classification accuracies for derivation and validation data sets were: 0.83 and 0.84 random forest, 0.82 and 0.81 XGBoost, 0.78 and 0.79 logistic regression. Algorithms generated a continuous number between 0 and 1 (or 0 and 100), representing readiness for discharge after general surgery.

CONCLUSIONS

A derived artificial intelligence measure (the Adelaide Score) successfully predicts discharge within the next 12 and 24 h in general surgery patients. This may be useful for both treating teams and allied health staff within surgical systems.

Single count breath test for the evaluation of respiratory function in Myasthenia Gravis: A systematic review

BACKGROUND

Myasthenia gravis (MG) can have a variety of respiratory presentations, ranging from mild symptoms through to respiratory failure. The evaluation of respiratory function in MG can be limited by accessibility to testing facilities, availability of medical equipment, and facial weakness. The single count breath test (SCBT) may be a useful adjunct in the evaluation of respiratory function in MG.

METHOD

A systematic review of the databases PubMed, EMBASE, and the Cochrane Library was conducted from inception to October 2022 in accordance with PRISMA guidelines and was registered on PROSPERO.

RESULTS

There were 6 studies that fulfilled the inclusion criteria. The described method of evaluating SCBT involves inhaling deeply, then counting at two counts per second, in English or Spanish, sitting upright, with normal vocal register, until another breath needs to be taken. The identified studies support that the SCBT has a moderate correlation with forced vital capacity. These results also support that SCBT can assist the identification of MG exacerbation, including via assessment over the telephone. The included studies support a threshold count of ≥ 25 as consistent with normal respiratory muscle function. Although further analysis is needed, the included studies describe the SCBT as a quick bedside tool that is inexpensive and well tolerated.

CONCLUSIONS

The results of this review support the clinical utility of the SCBT in assessing respiratory function in MG and describe the most current and effective methods of administration.

Protocol describing a systematic review and mixed methods consensus process to define the deteriorated ward patient

INTRODUCTION

Most patients admitted to hospital recover with treatments that can be administered on the general ward. A small but important group deteriorate however and require augmented organ support in areas with increased nursing to patient ratios. In observational studies evaluating this cohort, proxy outcomes such as unplanned intensive care unit admission, cardiac arrest and death are used. These outcome measures introduce subjectivity and variability, which in turn hinders the development and accuracy of the increasing numbers of electronic medical record (EMR) linked digital tools designed to predict clinical deterioration. Here, we describe a protocol for developing a new outcome measure using mixed methods to address these limitations.

METHODS AND ANALYSIS

We will undertake firstly, a systematic literature review to identify existing generic, syndrome-specific and organ-specific definitions for clinically deteriorated, hospitalised adult patients. Secondly, an international modified Delphi study to generate a short list of candidate definitions. Thirdly, a nominal group technique (NGT) (using a trained facilitator) will take a diverse group of stakeholders through a structured process to generate a consensus definition. The NGT process will be informed by the data generated from the first two stages. The definition(s) for the deteriorated ward patient will be readily extractable from the EMR.

ETHICS AND DISSEMINATION

This study has ethics approval (reference 16399) from the Central Adelaide Local Health Network Human Research Ethics Committee. Results generated from this study will be disseminated through publication and presentation at national and international scientific meetings.

Artificial intelligence and clinical deterioration

PURPOSE OF REVIEW

To provide an overview of the systems being used to identify and predict clinical deterioration in hospitalised patients, with focus on the current and future role of artificial intelligence (AI).

RECENT FINDINGS

There are five leading AI driven systems in this field: the Advanced Alert Monitor (AAM), the electronic Cardiac Arrest Risk Triage (eCART) score, Hospital wide Alert Via Electronic Noticeboard, the Mayo Clinic Early Warning Score, and the Rothman Index (RI). Each uses Electronic Patient Record (EPR) data and machine learning to predict adverse events. Less mature but relevant evolutions are occurring in the fields of Natural Language Processing, Time and Motion Studies, AI Sepsis and COVID-19 algorithms.

SUMMARY

Research-based AI-driven systems to predict clinical deterioration are increasingly being developed, but few are being implemented into clinical workflows. Escobar et al. (AAM) provide the current gold standard for robust model development and implementation methodology. Multiple technologies show promise, however, the pathway to meaningfully affect patient outcomes remains challenging.

Evaluation of a digital system to predict unplanned admissions to the intensive care unit: A mixed-methods approach

Background

We have developed the Hospital Alerting Via Electronic Noticeboard (HAVEN) which aims to identify hospitalised patients most at risk of reversible deterioration. HAVEN combines patients’ vital-sign measurements with laboratory results, demographics and comorbidities using a machine learnt algorithm.

Objectives

The aim of this study was to identify variables or concepts that could improve HAVEN predictive performance.

Methods

This was an embedded, mixed methods study. Eligible patients with the five highest HAVEN scores in the hospital (i.e., ‘HAVEN Top 5′) had their medical identification details recorded. We conducted a structured medical note review on these patients 48 hours post their identifiers being recorded. Methods of constant comparison were used during data collection and to analyse patient data.

Results

The 129 patients not admitted to ICU then underwent constant comparison review, which produced three main groups. Group 1 were patients referred to specialist services (n = 37). Group 2 responded to ward-based treatment, (n = 38). Group 3 were frail and had documented treatment limitations (n = 47).

Conclusions

Digital-only validation methods code the cohort not admitted to ICU as ‘falsely positive’ in sensitivity analyses however this approach limits the evaluation of model performance. Our study suggested that coding for patients referred to other specialist teams, those with treatment limitations in place, along with those who are deteriorating but then respond to ward-based therapies, would give a more accurate measure of the value of the scores, especially in relation to cost-effectiveness of resource utilisation.

Systematic review of applied usability metrics within usability evaluation methods for hospital electronic healthcare record systems: Metrics and Evaluation Methods for eHealth Systems

BACKGROUND AND OBJECTIVES

Electronic healthcare records have become central to patient care. Evaluation of new systems include a variety of usability evaluation methods or usability metrics (often referred to interchangeably as usability components or usability attributes). This study reviews the breadth of usability evaluation methods, metrics, and associated measurement techniques that have been reported to assess systems designed for hospital staff to assess inpatient clinical condition.

METHODS

Following Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) methodology, we searched Medline, EMBASE, CINAHL, Cochrane Database of Systematic Reviews, and Open Grey from 1986 to 2019. For included studies, we recorded usability evaluation methods or usability metrics as appropriate, and any measurement techniques applied to illustrate these. We classified and described all usability evaluation methods, usability metrics, and measurement techniques. Study quality was evaluated using a modified Downs and Black checklist.

RESULTS

The search identified 1336 studies. After abstract screening, 130 full texts were reviewed. In the 51 included studies 11 distinct usability evaluation methods were identified. Within these usability evaluation methods, seven usability metrics were reported. The most common metrics were ISO9241-11 and Nielsen's components. An additional "usefulness" metric was reported in almost 40% of included studies. We identified 70 measurement techniques used to evaluate systems. Overall study quality was reflected in a mean modified Downs and Black checklist score of 6.8/10 (range 1-9) 33% studies classified as "high-quality" (scoring eight or higher), 51% studies "moderate-quality" (scoring 6-7), and the remaining 16% (scoring below five) were "low-quality."

CONCLUSION

There is little consistency within the field of electronic health record systems evaluation. This review highlights the variability within usability methods, metrics, and reporting. Standardized processes may improve evaluation and comparison electronic health record systems and improve their development and implementation.

Detecting Deteriorating Patients in the Hospital: Development and Validation of a Novel Scoring System

Rationale: Late recognition of patient deterioration in hospital is associated with worse outcomes, including higher mortality. Despite the widespread introduction of early warning score (EWS) systems and electronic health records, deterioration still goes unrecognized. Objectives: To develop and externally validate a Hospital- wide Alerting via Electronic Noticeboard (HAVEN) system to identify hospitalized patients at risk of reversible deterioration. Methods: This was a retrospective cohort study of patients 16 years of age or above admitted to four UK hospitals. The primary outcome was cardiac arrest or unplanned admission to the ICU. We used patient data (vital signs, laboratory tests, comorbidities, and frailty) from one hospital to train a machine-learning model (gradient boosting trees). We internally and externally validated the model and compared its performance with existing scoring systems (including the National EWS, laboratory-based acute physiology score, and electronic cardiac arrest risk triage score). Measurements and Main Results: We developed the HAVEN model using 230,415 patient admissions to a single hospital. We validated HAVEN on 266,295 admissions to four hospitals. HAVEN showed substantially higher discrimination (c-statistic, 0.901 [95% confidence interval, 0.898-0.903]) for the primary outcome within 24 hours of each measurement than other published scoring systems (which range from 0.700 [0.696-0.704] to 0.863 [0.860-0.865]). With a precision of 10%, HAVEN was able to identify 42% of cardiac arrests or unplanned ICU admissions with a lead time of up to 48 hours in advance, compared with 22% by the next best system. Conclusions: The HAVEN machine-learning algorithm for early identification of in-hospital deterioration significantly outperforms other published scores such as the National EWS.

Testing a digital system that ranks the risk of unplanned intensive care unit admission in all ward patients: protocol for a prospective observational cohort study

INTRODUCTION

Traditional early warning scores (EWSs) use vital sign derangements to detect clinical deterioration in patients treated on hospital wards. Combining vital signs with demographics and laboratory results improves EWS performance. We have developed the Hospital Alerting Via Electronic Noticeboard (HAVEN) system. HAVEN uses vital signs, as well as demographic, comorbidity and laboratory data from the electronic patient record, to quantify and rank the risk of unplanned admission to an intensive care unit (ICU) within 24 hours for all ward patients. The primary aim of this study is to find additional variables, potentially missed during development, which may improve HAVEN performance. These variables will be sought in the medical record of patients misclassified by the HAVEN risk score during testing.

METHODS

This will be a prospective, observational, cohort study conducted at the John Radcliffe Hospital, part of the Oxford University Hospitals NHS Foundation Trust in the UK. Each day during the study periods, we will document all highly ranked patients (ie, those with the highest risk for unplanned ICU admission) identified by the HAVEN system. After 48 hours, we will review the progress of the identified patients. Patients who were subsequently admitted to the ICU will be removed from the study (as they will have been correctly classified by HAVEN). Highly ranked patients not admitted to ICU will undergo a structured medical notes review. Additionally, at the end of the study periods, all patients who had an unplanned ICU admission but whom HAVEN failed to rank highly will have a structured medical notes review. The review will identify candidate variables, likely associated with unplanned ICU admission, not included in the HAVEN risk score.

ETHICS AND DISSEMINATION

Approval has been granted for gathering the data used in this study from the South Central Oxford C Research Ethics Committee (16/SC/0264, 13 June 2016) and the Confidentiality Advisory Group (16/CAG/0066).

DISCUSSION

Our study will use a clinical expert conducting a structured medical notes review to identify variables, associated with unplanned ICU admission, not included in the development of the HAVEN risk score. These variables will then be added to the risk score and evaluated for potential performance gain. To the best of our knowledge, this is the first study of this type. We anticipate that documenting the HAVEN development methods will assist other research groups developing similar technology.

TRIAL REGISTRATION NUMBER

ISRCTN12518261.

The effect of fractional inspired oxygen concentration on early warning score performance: A database analysis

OBJECTIVES

To calculate fractional inspired oxygen concentration (FiO2) thresholds in ward patients and add these to the National Early Warning Score (NEWS). To evaluate the performance of NEWS-FiO2 against NEWS when predicting in-hospital death and unplanned intensive care unit (ICU) admission.

METHODS

A multi-centre, retrospective, observational cohort study was carried out in five hospitals from two UK NHS Trusts. Adult admissions with at least one complete set of vital sign observations recorded electronically were eligible. The primary outcome measure was an 'adverse event' which comprised either in-hospital death or unplanned ICU admission. Discrimination was assessed using the Area Under the Receiver Operating Characteristic curve (AUROC).

RESULTS

A cohort of 83,304 patients from a total of 271,363 adult admissions were prescribed oxygen. In this cohort, NEWS-FiO2 (AUROC 0.823, 95% CI 0.819-0.824) outperformed NEWS (AUORC 0.811, 95% CI 0.809-0.814) when predicting in-hospital death or unplanned ICU admission within 24 h of a complete set of vital sign observations.

CONCLUSIONS

NEWS-FiO2 generates a performance gain over NEWS when studied in ward patients requiring oxygen. This warrants further study, particularly in patients with respiratory disorders.

Patient centred variables with univariate associations with unplanned ICU admission: a systematic review

Background

Multiple predictive scores using Electronic Patient Record data have been developed for hospitalised patients at risk of clinical deterioration. Methods used to select patient centred variables for inclusion in these scores varies. We performed a systematic review to describe univariate associations with unplanned Intensive Care Unit (ICU) admission with the aim of assisting model development for future scores that predict clinical deterioration.

Methods

Data sources were MEDLINE, EMBASE, CINAHL, CENTRAL and the Cochrane Database of Systematic Reviews. Included studies were published since 2000 describing an association between patient centred variables and unplanned ICU admission determined using univariate analysis. Two authors independently screened titles, abstracts and full texts against inclusion and exclusion criteria. DistillerSR (Evidence Partners, Canada, Ottawa, Ontario) software was used to manage the data and identify duplicate search results. All screening and data extraction forms were implemented within DistillerSR. Study quality was assessed using an adapted version of the Newcastle-Ottawa Scale. Variables were analysed for strength of association with unplanned ICU admission.

Results

The database search yielded 1520 unique studies; 1462 were removed after title and abstract review; 57 underwent full text screening; 16 studies were included. One hundred and eighty nine variables with an evaluated univariate association with unplanned ICU admission were described.

Discussion

Being male, increasing age, a history of congestive cardiac failure or diabetes, a diagnosis of hepatic disease or having abnormal vital signs were all strongly associated with ICU admission.

Conclusion

These findings will assist variable selection during the development of future models predicting unplanned ICU admission.

Trial registration

This study is a component of a larger body of work registered in the ISRCTN registry (ISRCTN12518261).

Electronic supplementary material

The online version of this article (10.1186/s12911-019-0820-1) contains supplementary material, which is available to authorized users.

A comparison of the ability of the National Early Warning Score and the National Early Warning Score 2 to identify patients at risk of in-hospital mortality: A multi-centre database study

Aims

To compare the ability of the National Early Warning Score (NEWS) and the National Early Warning Score 2 (NEWS2) to identify patients at risk of in-hospital mortality and other adverse outcomes.

Methods

We undertook a multi-centre retrospective observational study at five acute hospitals from two UK NHS Trusts. Data were obtained from completed adult admissions who were not fit enough to be discharged alive on the day of admission. Diagnostic coding and oxygen prescriptions were used to identify patients with type II respiratory failure (T2RF). The primary outcome was in-hospital mortality within 24 h of a vital signs observation. Secondary outcomes included unanticipated intensive care unit admission or cardiac arrest within 24 h of a vital signs observation. Discrimination was assessed using the c-statistic.

Results

Among 251,266 adult admissions, 48,898 were identified to be at risk of T2RF by diagnostic coding. In this group, NEWS2 showed statistically significant lower discrimination (c-statistic, 95% CI) for identifying in-hospital mortality within 24 h (0.860, 0.857–0.864) than NEWS (0.881, 0.878-0.884). For 1394 admissions with documented T2RF, discrimination was similar for both systems: NEWS2 (0.841, 0.827-0.855), NEWS (0.862, 0.848–0.875). For all secondary endpoints, NEWS2 showed no improvements in discrimination.

Conclusions

NEWS2 modifications to NEWS do not improve discrimination of adverse outcomes in patients with documented T2RF and decrease discrimination in patients at risk of T2RF. Further evaluation of the relationship between SpO₂ values, oxygen therapy and risk should be investigated further before wide-scale adoption of NEWS2.

Variables associated with unplanned general adult ICU admission in hospitalised patients: protocol for a systematic review

BACKGROUND

Failure to promptly identify deterioration in hospitalised patients is associated with delayed admission to intensive care units (ICUs) and poor outcomes. Existing vital sign-based Early Warning Score (EWS) algorithms do not have a sufficiently high positive predictive value to be used for automated activation of an ICU outreach team. Incorporating additional patient data might improve the predictive power of EWS algorithms; however, it is currently not known which patient data (or variables) are most predictive of ICU admission. We describe the protocol for a systematic review of variables associated with ICU admission.

METHODS/DESIGN

MEDLINE, EMBASE, CINAHL and the Cochrane Library, including Cochrane Database of Systematic Reviews and the Cochrane Central Register of Controlled Trials (CENTRAL) will be searched for studies that assess the association of routinely recorded variables associated with subsequent unplanned ICU admission. Only studies involving adult patients admitted to general ICUs will be included. We will extract data relating to the statistical association between ICU admission and predictor variables, the quality of the studies and the generalisability of the findings.

DISCUSSION

The results of this review will aid the development of future models which predict the risk of unplanned ICU admission.

SYSTEMATIC REVIEW REGISTRATION

PROSPERO: CRD42015029617.