A scoping review of real-time automated clinical deterioration alerts and evidence of impacts on hospitalised patient outcomes

A prediction model for prehospital clinical deterioration: The use of early warning scores

BACKGROUND

Various prognosticative approaches to assist in recognizing clinical deterioration have been proposed. To date, early warning scores (EWSs) have been evaluated in hospital with limited research investigating their suitability in the prehospital setting. This study evaluated the predictive ability of established EWSs and other clinical factors for prehospital clinical deterioration.

METHODS

A retrospective cohort study investigating adult patients of all etiologies attended by Queensland Ambulance Service paramedics between January 1, 2018, and December 31, 2020, was conducted. With logistic regression, several models were developed to predict adverse event outcomes. The National Early Warning Score (NEWS), Modified Early Warning Score (MEWS), Queensland Adult Deterioration Detection System (Q-ADDS), and shock index were calculated from vital signs taken by paramedics.

RESULTS

A total of 1,422,046 incidents met the inclusion criteria. NEWS, MEWS, and Q-ADDS were found to have comparably high predictive ability with area under the receiver operating characteristic curve (AUC-ROC) between 70% and 90%, whereas shock index had relatively low AUC-ROC. Sensitivity was lower than specificity for all models. Although established EWSs performed well when predicting adverse events, these scores require complex calculations requiring multiple vital signs that may not be suitable for the prehospital setting.

CONCLUSIONS

This study found NEWS, MEWS, and Q-ADDS all performed well in the prehospital setting. Although a simple shock index is easier for paramedics to use in the prehospital environment, it did not perform comparably to established EWSs. Further research is required to develop suitably performing parsimonious solutions until established EWSs are integrated into technological solutions to be used by prehospital clinicians in real time.

Discrepancies between Promised and Actual AI Capabilities in the Continuous Vital Sign Monitoring of In-Hospital Patients: A Review of the Current Evidence

Continuous vital sign monitoring (CVSM) with wireless sensors in general hospital wards can enhance patient care. An artificial intelligence (AI) layer is crucial to allow sensor data to be managed by clinical staff without over alerting from the sensors. With the aim of summarizing peer-reviewed evidence for AI support in CVSM sensors, we searched PubMed and Embase for studies on adult patients monitored with CVSM sensors in general wards. Peer-reviewed evidence and white papers on the official websites of CVSM solutions were also included. AI classification was based on standard definitions of simple AI, as systems with no memory or learning capabilities, and advanced AI, as systems with the ability to learn from past data to make decisions. Only studies evaluating CVSM algorithms for improving or predicting clinical outcomes (e.g., adverse events, intensive care unit admission, mortality) or optimizing alarm thresholds were included. We assessed the promised level of AI for each CVSM solution based on statements from the official product websites. In total, 467 studies were assessed; 113 were retrieved for full-text review, and 26 studies on four different CVSM solutions were included. Advanced AI levels were indicated on the websites of all four CVSM solutions. Five studies assessed algorithms with potential for applications as advanced AI algorithms in two of the CVSM solutions (50%), while 21 studies assessed algorithms with potential as simple AI in all four CVSM solutions (100%). Evidence on algorithms for advanced AI in CVSM is limited, revealing a discrepancy between promised AI levels and current algorithm capabilities.

A Scoping Review of Fall-Risk Screening Tools in the Emergency Department for Future Falls in Older Adults

BACKGROUND

Approximately one-third of adults over the age of 65 experience falls annually, with half resulting in injury. Peak bodies have recommended the use of fall-risk screening tools in the emergency department (ED) to identify patients requiring in-depth assessment and potential fall-prevention intervention. This study aimed to examine the scope of published studies on fall-risk screening tools used in the ED and evidence of associations between screening and future falls.

SUMMARY

PubMed, Embase and CINAHL were searched for peer-reviewed journal articles published since 2012 that examined one or more screening tools to identify patient-level fall risk. Eligible studies described fall-risk tools applied in the ED. Data extracted included sample information, variables measured, and statistical analysis. Sixteen studies published since 2012 were included after full-text review. Fourteen unique screening tools were found. Eight tools were fall-risk screening tools, one tool was a functional screening tool, one tool was a frailty-screening tool, two tools were rapid physical tests, one tool was a trauma triage tool, and one tool was a component of a health-related quality-of-life measure. Studies that evaluated prognostic performance (n = 11) generally reported sensitivity higher than specificity. Previous falls (n = 10) and high-risk medications (n = 6) were consistently associated with future falls. Augmentation with additional variables from the electronic medical record (EMR) improved screening tool prognostic performance in one study.

KEY MESSAGES

Current evidence on the association between the use of fall-risk screening tools in the ED for future falls consistently identifies previous falls and high-risk medications as associated with future falls. Comparison between tools is difficult due to different evaluation methods and different covariates measured. Augmentation of fall-risk screening using the EMR in the ED requires further investigation.

Clinician perspectives and recommendations regarding design of clinical prediction models for deteriorating patients in acute care

BACKGROUND

Successful deployment of clinical prediction models for clinical deterioration relates not only to predictive performance but to integration into the decision making process. Models may demonstrate good discrimination and calibration, but fail to match the needs of practising acute care clinicians who receive, interpret, and act upon model outputs or alerts. We sought to understand how prediction models for clinical deterioration, also known as early warning scores (EWS), influence the decision-making of clinicians who regularly use them and elicit their perspectives on model design to guide future deterioration model development and implementation.

METHODS

Nurses and doctors who regularly receive or respond to EWS alerts in two digital metropolitan hospitals were interviewed for up to one hour between February 2022 and March 2023 using semi-structured formats. We grouped interview data into sub-themes and then into general themes using reflexive thematic analysis. Themes were then mapped to a model of clinical decision making using deductive framework mapping to develop a set of practical recommendations for future deterioration model development and deployment.

RESULTS

Fifteen nurses (n = 8) and doctors (n = 7) were interviewed for a mean duration of 42 min. Participants emphasised the importance of using predictive tools for supporting rather than supplanting critical thinking, avoiding over-protocolising care, incorporating important contextual information and focusing on how clinicians generate, test, and select diagnostic hypotheses when managing deteriorating patients. These themes were incorporated into a conceptual model which informed recommendations that clinical deterioration prediction models demonstrate transparency and interactivity, generate outputs tailored to the tasks and responsibilities of end-users, avoid priming clinicians with potential diagnoses before patients were physically assessed, and support the process of deciding upon subsequent management.

CONCLUSIONS

Prediction models for deteriorating inpatients may be more impactful if they are designed in accordance with the decision-making processes of acute care clinicians. Models should produce actionable outputs that assist with, rather than supplant, critical thinking.

Prioritising deteriorating patients using time-to-event analysis: prediction model development and internal-external validation

BACKGROUND

Binary classification models are frequently used to predict clinical deterioration, however they ignore information on the timing of events. An alternative is to apply time-to-event models, augmenting clinical workflows by ranking patients by predicted risks. This study examines how and why time-to-event modelling of vital signs data can help prioritise deterioration assessments using lift curves, and develops a prediction model to stratify acute care inpatients by risk of clinical deterioration.

METHODS

We developed and validated a Cox regression for time to in-hospital mortality. The model used time-varying covariates to estimate the risk of clinical deterioration. Adult inpatient medical records from 5 Australian hospitals between 1 January 2019 and 31 December 2020 were used for model development and validation. Model discrimination and calibration were assessed using internal-external cross validation. A discrete-time logistic regression model predicting death within 24 h with the same covariates was used as a comparator to the Cox regression model to estimate differences in predictive performance between the binary and time-to-event outcome modelling approaches.

RESULTS

Our data contained 150,342 admissions and 1016 deaths. Model discrimination was higher for Cox regression than for discrete-time logistic regression, with cross-validated AUCs of 0.96 and 0.93, respectively, for mortality predictions within 24 h, declining to 0.93 and 0.88, respectively, for mortality predictions within 1 week. Calibration plots showed that calibration varied by hospital, but this can be mitigated by ranking patients by predicted risks.

CONCLUSION

Time-varying covariate Cox models can be powerful tools for triaging patients, which may lead to more efficient and effective care in time-poor environments when the times between observations are highly variable.

Systematic review and longitudinal analysis of implementing Artificial Intelligence to predict clinical deterioration in adult hospitals: what is known and what remains uncertain

OBJECTIVE

To identify factors influencing implementation of machine learning algorithms (MLAs) that predict clinical deterioration in hospitalized adult patients and relate these to a validated implementation framework.

MATERIALS AND METHODS

A systematic review of studies of implemented or trialed real-time clinical deterioration prediction MLAs was undertaken, which identified: how MLA implementation was measured; impact of MLAs on clinical processes and patient outcomes; and barriers, enablers and uncertainties within the implementation process. Review findings were then mapped to the SALIENT end-to-end implementation framework to identify the implementation stages at which these factors applied.

RESULTS

Thirty-seven articles relating to 14 groups of MLAs were identified, each trialing or implementing a bespoke algorithm. One hundred and seven distinct implementation evaluation metrics were identified. Four groups reported decreased hospital mortality, 1 significantly. We identified 24 barriers, 40 enablers, and 14 uncertainties and mapped these to the 5 stages of the SALIENT implementation framework.

DISCUSSION

Algorithm performance across implementation stages decreased between in silico and trial stages. Silent plus pilot trial inclusion was associated with decreased mortality, as was the use of logistic regression algorithms that used less than 39 variables. Mitigation of alert fatigue via alert suppression and threshold configuration was commonly employed across groups.

CONCLUSIONS

: There is evidence that real-world implementation of clinical deterioration prediction MLAs may improve clinical outcomes. Various factors identified as influencing success or failure of implementation can be mapped to different stages of implementation, thereby providing useful and practical guidance for implementers.

Information displays for automated surveillance algorithms of in-hospital patient deterioration: a scoping review

OBJECTIVE

Surveillance algorithms that predict patient decompensation are increasingly integrated with clinical workflows to help identify patients at risk of in-hospital deterioration. This scoping review aimed to identify the design features of the information displays, the types of algorithm that drive the display, and the effect of these displays on process and patient outcomes.

MATERIALS AND METHODS

The scoping review followed Arksey and O'Malley's framework. Five databases were searched with dates between January 1, 2009 and January 26, 2022. Inclusion criteria were: participants-clinicians in inpatient settings; concepts-intervention as deterioration information displays that leveraged automated AI algorithms; comparison as usual care or alternative displays; outcomes as clinical, workflow process, and usability outcomes; and context as simulated or real-world in-hospital settings in any country. Screening, full-text review, and data extraction were reviewed independently by 2 researchers in each step. Display categories were identified inductively through consensus.

RESULTS

Of 14 575 articles, 64 were included in the review, describing 61 unique displays. Forty-one displays were designed for specific deteriorations (eg, sepsis), 24 provided simple alerts (ie, text-based prompts without relevant patient data), 48 leveraged well-accepted score-based algorithms, and 47 included nurses as the target users. Only 1 out of the 10 randomized controlled trials reported a significant effect on the primary outcome.

CONCLUSIONS

Despite significant advancements in surveillance algorithms, most information displays continue to leverage well-understood, well-accepted score-based algorithms. Users' trust, algorithmic transparency, and workflow integration are significant hurdles to adopting new algorithms into effective decision support tools.

Forecasting algorithms in the ICU

Despite significant advances in modeling methods and access to large datasets, there are very few real-time forecasting systems deployed in highly monitored environment such as the intensive care unit. Forecasting models may be developed as classification, regression or time-to-event tasks; each could be using a variety of machine learning algorithms. An accurate and useful forecasting systems include several components beyond a forecasting model, and its performance is assessed using end-user-centered metrics. Several barriers to implementation and acceptance persist and clinicians will play an active role in the successful deployment of this promising technology.

Vital signs-based deterioration prediction model assumptions can lead to losses in prediction performance

OBJECTIVE

Vital signs-based models are complicated by repeated measures per patient and frequently missing data. This paper investigated the impacts of common vital signs modeling assumptions during clinical deterioration prediction model development.

STUDY DESIGN AND SETTING

Electronic medical record (EMR) data from five Australian hospitals (1 January 2019-31 December 2020) were used. Summary statistics for each observation's prior vital signs were created. Missing data patterns were investigated using boosted decision trees, then imputed with common methods. Two example models predicting in-hospital mortality were developed, as follows: logistic regression and eXtreme Gradient Boosting. Model discrimination and calibration were assessed using the C-statistic and nonparametric calibration plots.

RESULTS

The data contained 5,620,641 observations from 342,149 admissions. Missing vitals were associated with observation frequency, vital sign variability, and patient consciousness. Summary statistics improved discrimination slightly for logistic regression and markedly for eXtreme Gradient Boosting. Imputation method led to notable differences in model discrimination and calibration. Model calibration was generally poor.

CONCLUSION

Summary statistics and imputation methods can improve model discrimination and reduce bias during model development, but it is questionable whether these differences are clinically significant. Researchers should consider why data are missing during model development and how this may impact clinical utility.

Evaluating the costs and consequences of computerized clinical decision support systems in hospitals: a scoping review and recommendations for future practice

OBJECTIVE

Sustainable investment in computerized decision support systems (CDSS) requires robust evaluation of their economic impacts compared with current clinical workflows. We reviewed current approaches used to evaluate the costs and consequences of CDSS in hospital settings and presented recommendations to improve the generalizability of future evaluations.

MATERIALS AND METHODS

A scoping review of peer-reviewed research articles published since 2010. Searches were completed in the PubMed, Ovid Medline, Embase, and Scopus databases (last searched February 14, 2023). All studies reported the costs and consequences of a CDSS-based intervention compared with current hospital workflows. Findings were summarized using narrative synthesis. Individual studies were further appraised against the Consolidated Health Economic Evaluation and Reporting (CHEERS) 2022 checklist.

RESULTS

Twenty-nine studies published since 2010 were included. Studies evaluated CDSS for adverse event surveillance (5 studies), antimicrobial stewardship (4 studies), blood product management (8 studies), laboratory testing (7 studies), and medication safety (5 studies). All studies evaluated costs from a hospital perspective but varied based on the valuation of resources affected by CDSS implementation, and the measurement of consequences. We recommend future studies follow guidance from the CHEERS checklist; use study designs that adjust for confounders; consider both the costs of CDSS implementation and adherence; evaluate consequences that are directly or indirectly affected by CDSS-initiated behavior change; examine the impacts of uncertainty and differences in outcomes across patient subgroups.

DISCUSSION AND CONCLUSION

Improving consistency in the conduct and reporting of evaluations will enable detailed comparisons between promising initiatives, and their subsequent uptake by decision-makers.

Developing and Validating a Prediction Model For Death or Critical Illness in Hospitalized Adults, an Opportunity for Human-Computer Collaboration

Hospital early warning systems that use machine learning (ML) to predict clinical deterioration are increasingly being used to aid clinical decision-making. However, it is not known how ML predictions complement physician and nurse judgment. Our objective was to train and validate a ML model to predict patient deterioration and compare model predictions with real-world physician and nurse predictions.

DESIGN

Retrospective and prospective cohort study.

SETTING

Academic tertiary care hospital.

PATIENTS

Adult general internal medicine hospitalizations.

MEASUREMENTS AND MAIN RESULTS

We developed and validated a neural network model to predict in-hospital death and ICU admission in 23,528 hospitalizations between April 2011 and April 2019. We then compared model predictions with 3,374 prospectively collected predictions from nurses, residents, and attending physicians about their own patients in 960 hospitalizations between April 30, and August 28, 2019. ML model predictions achieved clinician-level accuracy for predicting ICU admission or death (ML median F1 score 0.32 [interquartile range (IQR) 0.30-0.34], AUC 0.77 [IQ 0.76-0.78]; clinicians median F1-score 0.33 [IQR 0.30-0.35], AUC 0.64 [IQR 0.63-0.66]). ML predictions were more accurate than clinicians for ICU admission. Of all ICU admissions and deaths, 36% occurred in hospitalizations where the model and clinicians disagreed. Combining human and model predictions detected 49% of clinical deterioration events, improving sensitivity by 16% compared with clinicians alone and 24% compared with the model alone while maintaining a positive predictive value of 33%, thus keeping false alarms at a clinically acceptable level.

CONCLUSIONS

ML models can complement clinician judgment to predict clinical deterioration in hospital. These findings demonstrate important opportunities for human-computer collaboration to improve prognostication and personalized medicine in hospital.