Towards development of alert thresholds for clinical deterioration using continuous predictive analytics monitoring

Prospective validation of clinical deterioration predictive models prior to intensive care unit transfer among patients admitted to acute care cardiology wards

Objective. Very few predictive models have been externally validated in a prospective cohort following the implementation of an artificial intelligence analytic system. This type of real-world validation is critically important due to the risk of data drift, or changes in data definitions or clinical practices over time, that could impact model performance in contemporaneous real-world cohorts. In this work, we report the model performance of a predictive analytics tool developed before COVID-19 and demonstrate model performance during the COVID-19 pandemic.Approach. The analytic system (CoMETⓇ, Nihon Kohden Digital Health Solutions LLC, Irvine, CA) was implemented in a randomized controlled trial that enrolled 10 422 patient visits in a 1:1 display-on display-off design. The CoMET scores were calculated for all patients but only displayed in the display-on arm. Only the control/display-off group is reported here because the scores could not alter care patterns.Main results.Of the 5184 visits in the display-off arm, 311 experienced clinical deterioration and care escalation, resulting in transfer to the intensive care unit, primarily due to respiratory distress. The model performance of CoMET was assessed based on areas under the receiver operating characteristic curve, which ranged from 0.725 to 0.737.Significance.The models were well-calibrated, and there were dynamic increases in the model scores in the hours preceding the clinical deterioration events. A hypothetical alerting strategy based on a rise in score and duration of the rise would have had good performance, with a positive predictive value more than 10-fold the event rate. We conclude that predictive statistical models developed five years before study initiation had good model performance despite the passage of time and the impact of the COVID-19 pandemic.

Wearable wireless continuous vital signs monitoring on the general ward

PURPOSE OF REVIEW

Wearable wireless sensors for continuous vital signs monitoring (CVSM) offer the potential for early identification of patient deterioration, especially in low-intensity care settings like general wards. This study aims to review advances in wearable CVSM - with a focus on the general ward - highlighting the technological characteristics of CVSM systems, user perspectives and impact on patient outcomes by exploring recent evidence.

RECENT FINDINGS

The accuracy of wearable sensors measuring vital signs exhibits variability, especially notable in ambulatory patients within hospital settings, and standard validation protocols are lacking. Usability of CMVS systems is critical for nurses and patients, highlighting the need for easy-to-use wearable sensors, and expansion of the number of measured vital signs. Current software systems lack integration with hospital IT infrastructures and workflow automation. Imperative enhancements involve nurse-friendly, less intrusive alarm strategies, and advanced decision support systems. Despite observed reductions in ICU admissions and Rapid Response Team calls, the impact on patient outcomes lacks robust statistical significance.

SUMMARY

Widespread implementation of CVSM systems on the general ward and potentially outside the hospital seems inevitable. Despite the theoretical benefits of CVSM systems in improving clinical outcomes, and supporting nursing care by optimizing clinical workflow efficiency, the demonstrated effects in clinical practice are mixed. This review highlights the existing challenges related to data quality, usability, implementation, integration, interpretation, and user perspectives, as well as the need for robust evidence to support their impact on patient outcomes, workflow and cost-effectiveness.

Predicting emergency department visits and hospitalizations for patients with heart failure in home healthcare using a time series risk model

OBJECTIVES

Little is known about proactive risk assessment concerning emergency department (ED) visits and hospitalizations in patients with heart failure (HF) who receive home healthcare (HHC) services. This study developed a time series risk model for predicting ED visits and hospitalizations in patients with HF using longitudinal electronic health record data. We also explored which data sources yield the best-performing models over various time windows.

MATERIALS AND METHODS

We used data collected from 9362 patients from a large HHC agency. We iteratively developed risk models using both structured (eg, standard assessment tools, vital signs, visit characteristics) and unstructured data (eg, clinical notes). Seven specific sets of variables included: (1) the Outcome and Assessment Information Set, (2) vital signs, (3) visit characteristics, (4) rule-based natural language processing-derived variables, (5) term frequency-inverse document frequency variables, (6) Bio-Clinical Bidirectional Encoder Representations from Transformers variables, and (7) topic modeling. Risk models were developed for 18 time windows (1-15, 30, 45, and 60 days) before an ED visit or hospitalization. Risk prediction performances were compared using recall, precision, accuracy, F1, and area under the receiver operating curve (AUC).

RESULTS

The best-performing model was built using a combination of all 7 sets of variables and the time window of 4 days before an ED visit or hospitalization (AUC = 0.89 and F1 = 0.69).

DISCUSSION AND CONCLUSION

This prediction model suggests that HHC clinicians can identify patients with HF at risk for visiting the ED or hospitalization within 4 days before the event, allowing for earlier targeted interventions.

Interpretation and Use of Applied/Operational Machine Learning and Artificial Intelligence in Surgery

Applications for artificial intelligence (AI) and machine learning in surgery include image interpretation, data summarization, automated narrative construction, trajectory and risk prediction, and operative navigation and robotics. The pace of development has been exponential, and some AI applications are working well. However, demonstrations of clinical utility, validity, and equity have lagged algorithm development and limited widespread adoption of AI into clinical practice. Outdated computing infrastructure and regulatory challenges which promote data silos are key barriers. Multidisciplinary teams will be needed to address these challenges and to build AI systems that are relevant, equitable, and dynamic.

Association of alternative polysomnographic features with patient outcomes in obstructive sleep apnea: a systematic review

STUDY OBJECTIVES

Polysomnograms (PSGs) collect a plethora of physiologic signals across the night. However, few of these PSG data are incorporated into standard reports, and hence, ultimately, under-utilized in clinical decision making. Recently, there has been substantial interest regarding novel alternative PSG metrics that may help to predict obstructive sleep apnea (OSA)-related outcomes better than standard PSG metrics such as the apnea-hypopnea index. We systematically review the recent literature for studies that examined the use of alternative PSG metrics in the context of OSA and their association with health outcomes.

METHODS

We systematically searched EMBASE, MEDLINE, and the Cochrane Database of Systematic Reviews for studies published between 2000 and 2022 for those that reported alternative metrics derived from PSG in adults and related them to OSA-related outcomes.

RESULTS

Of the 186 initial studies identified by the original search, data from 31 studies were ultimately included in the final analysis. Numerous metrics were identified that were significantly related to a broad range of outcomes. We categorized the outcomes into 2 main subgroups: (1) cardiovascular/metabolic outcomes and mortality and (2) cognitive function- and vigilance-related outcomes. Four general categories of alternative metrics were identified based on signals analyzed: autonomic/hemodynamic metrics, electroencephalographic metrics, oximetric metrics, and respiratory event-related metrics.

CONCLUSIONS

We have summarized the current landscape of literature for alternative PSG metrics relating to risk prediction in OSA. Although promising, further prospective observational studies are needed to verify findings from other cohorts, and to assess the clinical utility of these metrics.

CITATION

Hajipour M, Baumann B, Azarbarzin A, et al. Association of alternative polysomnographic features with patient outcomes in obstructive sleep apnea: a systematic review. J Clin Sleep Med. 2023;19(2):225-242.

A wearable patch based remote early warning score (REWS) in major abdominal cancer surgery patients

INTRODUCTION

The shift toward remote patient monitoring methods to detect clinical deterioration requires testing of wearable devices in real-life clinical settings. This study aimed to develop a remote early warning scoring (REWS) system based on continuous measurements using a wearable device, and compare its diagnostic performance for the detection of deterioration to the diagnostic performance of the conventional modified early warning score (MEWS).

MATERIALS AND METHODS

The study population of this prospective, single center trial consisted of patients who underwent major abdominal cancer surgery and were monitored using routine in-hospital spotcheck measurements of the vital parameters. Heart and respiratory rates were measured continuously using a wireless accelerometer patch (HealthDot). The prediction by MEWS of deterioration toward a complication graded Clavien-Dindo of 2 or higher was compared to the REWS derived from continuous measurements by the wearable patch.

MAIN RESULTS

A total of 103 patients and 1909 spot-check measurements were included in the analysis. Postoperative deterioration was observed in 29 patients. For both EWS systems, the sensitivity (MEWS: 0.20 95% CI: [0.13-0.29], REWS: 0.20 95% CI: [0.13-0.29]) and specificity (MEWS: 0.96 95% CI: [0.95-0.97], REWS: 0.96 95% CI: [0.95-0.97]) were assessed.

CONCLUSIONS

The diagnostic value of the REWS method, based on continuous measurements of the heart and respiratory rates, is comparable to that of the MEWS in patients following major abdominal cancer surgery. The wearable patch could detect the same amount of deteriorations, without requiring manual spot check measurements.

Review of the Use of Liquid Chromatography-Tandem Mass Spectrometry in Clinical Laboratories: Part II-Operations

Liquid chromatography-tandem mass spectrometry (LC-MS/MS) is increasingly utilized in clinical laboratories because it has advantages in terms of specificity and sensitivity over other analytical technologies. These advantages come with additional responsibilities and challenges given that many assays and platforms are not provided to laboratories as a single kit or device. The skills, staff, and assays used in LC-MS/MS are internally developed by the laboratory, with relatively few exceptions. Hence, a laboratory that deploys LC-MS/MS assays must be conscientious of the practices and procedures adopted to overcome the challenges associated with the technology. This review discusses the post-development landscape of LC-MS/MS assays, including validation, quality assurance, operations, and troubleshooting. The content knowledge of LC-MS/MS users is quite broad and deep and spans multiple scientific fields, including biology, clinical chemistry, chromatography, engineering, and MS. However, there are no formal academic programs or specific literature to train laboratory staff on the fundamentals of LC-MS/MS beyond the reports on method development. Therefore, depending on their experience level, some readers may be familiar with aspects of the laboratory practices described herein, while others may be not. This review endeavors to assemble aspects of LC-MS/MS operations in the clinical laboratory to provide a framework for the thoughtful development and execution of LC-MS/MS applications.

Continuous vital sign monitoring using a wearable patch sensor in obese patients: a validation study in a clinical setting

Our aim was to determine the agreement of heart rate (HR) and respiratory rate (RR) measurements by the Philips Biosensor with a reference monitor (General Electric Carescape B650) in severely obese patients during and after bariatric surgery. Additionally, sensor reliability was assessed. Ninety-four severely obese patients were monitored with both the Biosensor and reference monitor during and after bariatric surgery. Agreement was defined as the mean absolute difference between both monitoring devices. Bland Altman plots and Clarke Error Grid analysis (CEG) were used to visualise differences. Sensor reliability was reflected by the amount, duration and causes of data loss. The mean absolute difference for HR was 1.26 beats per minute (bpm) (SD 0.84) during surgery and 1.84 bpm (SD 1.22) during recovery, and never exceeded the 8 bpm limit of agreement. The mean absolute difference for RR was 1.78 breaths per minute (brpm) (SD 1.90) during surgery and 4.24 brpm (SD 2.75) during recovery. The Biosensor's RR measurements exceeded the 2 brpm limit of agreement in 58% of the compared measurements. Averaging 15 min of measurements for both devices improved agreement. CEG showed that 99% of averaged RR measurements resulted in adequate treatment. Data loss was limited to 4.5% of the total duration of measurements for RR. No clear causes for data loss were found. The Biosensor is suitable for remote monitoring of HR, but not RR in morbidly obese patients. Future research should focus on improving RR measurements, the interpretation of continuous data, and development of smart alarm systems.

Identification of nutritional risk in the acute care setting: progress towards a practice and evidence informed systems level approach

Background

To improve nutritional assessment and care pathways in the acute care setting, it is important to understand the indicators that may predict nutritional risk. Informed by a review of systematic reviews, this project engaged stakeholders to prioritise and reach consensus on a list of evidence based and clinically contextualised indicators for identifying malnutrition risk in the acute care setting.

Methods

A modified Delphi approach was employed which consisted of four rounds of consultation with 54 stakeholders and 10 experts to reach consensus and refine a list of 57 risk indicators identified from a review of systematic reviews. Weighted mean and variance scores for each indicator were evaluated. Consistency was tested with intra class correlation coefficient. Cronbach's alpha was used to determine the reliability of the indicators. The final list of indicators was subject to Cronbach’s alpha and exploratory principal component analysis.

Results

Fifteen indicators were considered to be the most important in identifying nutritional risk. These included difficulty self-feeding, polypharmacy, surgery and impaired gastro-intestinal function. There was 82% agreement for the final 15 indicators that they collectively would predict malnutrition risk in hospital inpatients.

Conclusion

The 15 indicators identified are supported by evidence and are clinically informed. This represents an opportunity for translation into a novel and automated systems level approach for identifying malnutrition risk in the acute care setting.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12913-021-07299-y.

Nursing and precision predictive analytics monitoring in the acute and intensive care setting: An emerging role for responding to COVID-19 and beyond

As the global response to COVID-19 continues, nurses will be tasked with appropriately triaging patients, responding to events of clinical deterioration, and developing family-centered plans of care within a healthcare system exceeding capacity. Predictive analytics monitoring, an artificial intelligence (AI)-based tool that translates streaming clinical data into a real-time visual estimation of patient risks, allows for evolving acuity assessments and detection of clinical deterioration while the patient is in pre-symptomatic states. While nurses are on the frontline for the COVID-19 pandemic, the use of AI-based predictive analytics monitoring may help cognitively complex clinical decision-making tasks and pave a pathway for early detection of patients at risk for decompensation. We must develop strategies and techniques to study the impact of AI-based technologies on patient care outcomes and the clinical workflow. This paper outlines key concepts for the intersection of nursing and precision predictive analytics monitoring.

Predicting Patient Deterioration: A Review of Tools in the Digital Hospital Setting

BACKGROUND

Early warning tools identify patients at risk of deterioration in hospitals. Electronic medical records in hospitals offer real-time data and the opportunity to automate early warning tools and provide real-time, dynamic risk estimates.

OBJECTIVE

This review describes published studies on the development, validation, and implementation of tools for predicting patient deterioration in general wards in hospitals.

METHODS

An electronic database search of peer reviewed journal papers from 2008-2020 identified studies reporting the use of tools and algorithms for predicting patient deterioration, defined by unplanned transfer to the intensive care unit, cardiac arrest, or death. Studies conducted solely in intensive care units, emergency departments, or single diagnosis patient groups were excluded.

RESULTS

A total of 46 publications were eligible for inclusion. These publications were heterogeneous in design, setting, and outcome measures. Most studies were retrospective studies using cohort data to develop, validate, or statistically evaluate prediction tools. The tools consisted of early warning, screening, or scoring systems based on physiologic data, as well as more complex algorithms developed to better represent real-time data, deal with complexities of longitudinal data, and warn of deterioration risk earlier. Only a few studies detailed the results of the implementation of deterioration warning tools.

CONCLUSIONS

Despite relative progress in the development of algorithms to predict patient deterioration, the literature has not shown that the deployment or implementation of such algorithms is reproducibly associated with improvements in patient outcomes. Further work is needed to realize the potential of automated predictions and update dynamic risk estimates as part of an operational early warning system for inpatient deterioration.

Predictive Monitoring-Impact in Acute Care Cardiology Trial (PM-IMPACCT): Protocol for a Randomized Controlled Trial

BACKGROUND

Patients in acute care wards who deteriorate and are emergently transferred to intensive care units (ICUs) have poor outcomes. Early identification of patients who are decompensating might allow for earlier clinical intervention and reduced morbidity and mortality. Advances in bedside continuous predictive analytics monitoring (ie, artificial intelligence [AI]-based risk prediction) have made complex data easily available to health care providers and have provided early warning of potentially catastrophic clinical events. We present a dynamic, visual, predictive analytics monitoring tool that integrates real-time bedside telemetric physiologic data into robust clinical models to estimate and communicate risk of imminent events. This tool, Continuous Monitoring of Event Trajectories (CoMET), has been shown in retrospective observational studies to predict clinical decompensation on the acute care ward. There is a need to more definitively study this advanced predictive analytics or AI monitoring system in a prospective, randomized controlled, clinical trial.

OBJECTIVE

The goal of this trial is to determine the impact of an AI-based visual risk analytic, CoMET, on improving patient outcomes related to clinical deterioration, response time to proactive clinical action, and costs to the health care system.

METHODS

We propose a cluster randomized controlled trial to test the impact of using the CoMET display in an acute care cardiology and cardiothoracic surgery hospital floor. The number of admissions to a room undergoing cluster randomization was estimated to be 10,424 over the 20-month study period. Cluster randomization based on bed number will occur every 2 months. The intervention cluster will have the CoMET score displayed (along with standard of care), while the usual care group will receive standard of care only.

RESULTS

The primary outcome will be hours free from events of clinical deterioration. Hours of acute clinical events are defined as time when one or more of the following occur: emergent ICU transfer, emergent surgery prior to ICU transfer, cardiac arrest prior to ICU transfer, emergent intubation, or death. The clinical trial began randomization in January 2021.

CONCLUSIONS

Very few AI-based health analytics have been translated from algorithm to real-world use. This study will use robust, prospective, randomized controlled, clinical trial methodology to assess the effectiveness of an advanced AI predictive analytics monitoring system in incorporating real-time telemetric data for identifying clinical deterioration on acute care wards. This analysis will strengthen the ability of health care organizations to evolve as learning health systems, in which bioinformatics data are applied to improve patient outcomes by incorporating AI into knowledge tools that are successfully integrated into clinical practice by health care providers.

TRIAL REGISTRATION

ClinicalTrials.gov NCT04359641; https://clinicaltrials.gov/ct2/show/NCT04359641.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

DERR1-10.2196/29631.

Accuracy and Monitoring of Pediatric Early Warning Score (PEWS) Scores Prior to Emergent Pediatric Intensive Care Unit (ICU) Transfer: Retrospective Analysis

BACKGROUND

Current approaches to early detection of clinical deterioration in children have relied on intermittent track-and-trigger warning scores such as the Pediatric Early Warning Score (PEWS) that rely on periodic assessment and vital sign entry. There are limited data on the utility of these scores prior to events of decompensation leading to pediatric intensive care unit (PICU) transfer.

OBJECTIVE

The purpose of our study was to determine the accuracy of recorded PEWS scores, assess clinical reasons for transfer, and describe the monitoring practices prior to PICU transfer involving acute decompensation.

METHODS

We conducted a retrospective cohort study of patients ≤21 years of age transferred emergently from the acute care pediatric floor to the PICU due to clinical deterioration over an 8-year period. Clinical charts were abstracted to (1) determine the clinical reason for transfer, (2) quantify the frequency of physiological monitoring prior to transfer, and (3) assess the timing and accuracy of the PEWS scores 24 hours prior to transfer.

RESULTS

During the 8-year period, 72 children and adolescents had an emergent PICU transfer due to clinical deterioration, most often due to acute respiratory distress. Only 35% (25/72) of the sample was on continuous telemetry or pulse oximetry monitoring prior to the transfer event, and 47% (34/72) had at least one incorrectly documented PEWS score in the 24 hours prior to the event, with a score underreporting the actual severity of illness.

CONCLUSIONS

This analysis provides support for the routine assessment of clinical deterioration and advocates for more research focused on the use and utility of continuous cardiorespiratory monitoring for patients at risk for emergent transfer.

Development and validation of a sample entropy-based method to identify complex patient-ventilator interactions during mechanical ventilation

Patient-ventilator asynchronies can be detected by close monitoring of ventilator screens by clinicians or through automated algorithms. However, detecting complex patient-ventilator interactions (CP-VI), consisting of changes in the respiratory rate and/or clusters of asynchronies, is a challenge. Sample Entropy (SE) of airway flow (SE-Flow) and airway pressure (SE-Paw) waveforms obtained from 27 critically ill patients was used to develop and validate an automated algorithm for detecting CP-VI. The algorithm's performance was compared versus the gold standard (the ventilator's waveform recordings for CP-VI were scored visually by three experts; Fleiss' kappa = 0.90 (0.87-0.93)). A repeated holdout cross-validation procedure using the Matthews correlation coefficient (MCC) as a measure of effectiveness was used for optimization of different combinations of SE settings (embedding dimension, m, and tolerance value, r), derived SE features (mean and maximum values), and the thresholds of change (Th) from patient's own baseline SE value. The most accurate results were obtained using the maximum values of SE-Flow (m = 2, r = 0.2, Th = 25%) and SE-Paw (m = 4, r = 0.2, Th = 30%) which report MCCs of 0.85 (0.78-0.86) and 0.78 (0.78-0.85), and accuracies of 0.93 (0.89-0.93) and 0.89 (0.89-0.93), respectively. This approach promises an improvement in the accurate detection of CP-VI, and future study of their clinical implications.