Prospective, multi-site study of patient outcomes after implementation of the TREWS machine learning-based early warning system for sepsis

Racial differences in laboratory testing as a potential mechanism for bias in AI: A matched cohort analysis in emergency department visits

AI models are often trained using available laboratory test results. Racial differences in laboratory testing may bias AI models for clinical decision support, amplifying existing inequities. This study aims to measure the extent of racial differences in laboratory testing in adult emergency department (ED) visits. We conducted a retrospective 1:1 exact-matched cohort study of Black and White adult patients seen in the ED, matching on age, biological sex, chief complaint, and ED triage score, using ED visits at two U.S. teaching hospitals: Michigan Medicine, Ann Arbor, MI (U-M, 2015-2022), and Beth Israel Deaconess Medical Center, Boston, MA (BIDMC, 2011-2019). Post-matching, White patients had significantly higher testing rates than Black patients for complete blood count (BIDMC difference: 1.7%, 95% CI: 1.1% to 2.4%, U-M difference: 2.0%, 95% CI: 1.6% to 2.5%), metabolic panel (BIDMC: 1.5%, 95% CI: 0.9% to 2.1%, U-M: 1.9%, 95% CI: 1.4% to 2.4%), and blood culture (BIDMC: 0.9%, 95% CI: 0.5% to 1.2%, U-M: 0.7%, 95% CI: 0.4% to 1.1%). Black patients had significantly higher testing rates for troponin than White patients (BIDMC: -2.1%, 95% CI: -2.6% to -1.6%, U-M: -2.2%, 95% CI: -2.7% to -1.8%). The observed racial testing differences may impact AI models trained using available laboratory results. The findings also motivate further study of how such differences arise and how to mitigate potential impacts on AI models.

Design and development of a machine-learning-driven opioid overdose risk prediction tool integrated in electronic health records in primary care settings

BACKGROUND

Integrating advanced machine-learning (ML) algorithms into clinical practice is challenging and requires interdisciplinary collaboration to develop transparent, interpretable, and ethically sound clinical decision support (CDS) tools. We aimed to design a ML-driven CDS tool to predict opioid overdose risk and gather feedback for its integration into the University of Florida Health (UFHealth) electronic health record (EHR) system.

METHODS

We used user-centered design methods to integrate the ML algorithm into the EHR system. The backend and UI design sub-teams collaborated closely, both informed by user feedback sessions. We conducted seven user feedback sessions with five UF Health primary care physicians (PCPs) to explore aspects of CDS tools, including workflow, risk display, and risk mitigation strategies. After customizing the tool based on PCPs' feedback, we held two rounds of one-on-one usability testing sessions with 8 additional PCPs to gather feedback on prototype alerts. These sessions informed iterative UI design and backend processes, including alert frequency and reappearance circumstances.

RESULTS

The backend process development identified needs and requirements from our team, information technology, UFHealth, and PCPs. Thirteen PCPs (male = 62%, White = 85%) participated across 7 user feedback sessions and 8 usability testing sessions. During the user feedback sessions, PCPs (n = 5) identified flaws such as the term "high risk" of overdose potentially leading to unintended consequences (e.g., immediate addiction services referrals), offered suggestions, and expressed trust in the tool. In the first usability testing session, PCPs (n = 4) emphasized the need for natural risk presentation (e.g., 1 in 200) and suggested displaying the alert multiple times yearly for at-risk patients. Another 4 PCPs in the second usability testing session valued the UFHealth-specific alert for managing new or unfamiliar patients, expressed concerns about PCPs' workload when prescribing to high-risk patients, and recommended incorporating the details page into training sessions to enhance usability.

CONCLUSIONS

The final backend process for our CDS alert aligns with PCP needs and UFHealth standards. Integrating feedback from PCPs in the early development phase of our ML-driven CDS tool helped identify barriers and facilitators in the CDS integration process. This collaborative approach yielded a refined prototype aimed at minimizing unintended consequences and enhancing usability.

Views and Uses of Sepsis Digital Alerts in National Health Service Trusts in England: Qualitative Study With Health Care Professionals

BACKGROUND

Sepsis is a common cause of serious illness and death. Sepsis management remains challenging and suboptimal. To support rapid sepsis diagnosis and treatment, screening tools have been embedded into hospital digital systems to appear as digital alerts. The implementation of digital alerts to improve the management of sepsis and deterioration is a complex intervention that has to fit with team workflow and the views and practices of hospital staff. Despite the importance of human decision-making and behavior in optimal implementation, there are limited qualitative studies that explore the views and experiences of health care professionals regarding digital alerts as sepsis or deterioration computerized clinician decision support systems (CCDSSs).

OBJECTIVE

This study aims to explore the views and experiences of health care professionals on the use of sepsis or deterioration CCDSSs and to identify barriers and facilitators to their implementation and use in National Health Service (NHS) hospitals.

METHODS

We conducted a qualitative, multisite study with unstructured observations and semistructured interviews with health care professionals from emergency departments, outreach teams, and intensive or acute units in 3 NHS hospital trusts in England. Data from both interviews and observations were analyzed together inductively using thematic analysis.

RESULTS

A total of 22 health care professionals were interviewed, and 12 observation sessions were undertaken. A total of four themes regarding digital alerts were identified: (1) support decision-making as nested in electronic health records, but never substitute professionals' knowledge and experience; (2) remind to take action according to the context, such as the hospital unit and the job role; (3) improve the alerts and their introduction, by making them more accessible, easy to use, not intrusive, more accurate, as well as integrated across the whole health care system; and (4) contextual factors affecting views and use of alerts in the NHS trusts. Digital alerts are more optimally used in general hospital units with a lower senior decision maker:patient ratio and by health care professionals with experience of a similar technology. Better use of the alerts was associated with quality improvement initiatives and continuous sepsis training. The trusts' features, such as the presence of a 24/7 emergency outreach team, good technological resources, and staffing and teamwork, favored a more optimal use.

CONCLUSIONS

Trust implementation of sepsis or deterioration CCDSSs requires support on multiple levels and at all phases of the intervention, starting from a prego-live analysis addressing organizational needs and readiness. Advancements toward minimally disruptive and smart digital alerts as sepsis or deterioration CCDSSs, which are more accurate and specific but at the same time scalable and accessible, require policy changes and investments in multidisciplinary research.

Machine learning interpretability methods to characterize the importance of hematologic biomarkers in prognosticating patients with suspected infection

OBJECTIVE

To evaluate the effectiveness of Monocyte Distribution Width (MDW) in predicting sepsis outcomes in emergency department (ED) patients compared to other hematologic parameters and vital signs, and to determine whether routine parameters could substitute MDW in machine learning models.

METHODS

We conducted a retrospective analysis of data from 10,229 ED patients admitted to a large regional safety-net hospital in Cleveland, Ohio who had suspected infections and developed sepsis-associated poor outcomes. We developed a new analytical framework consisting of seven data models and an ensemble of high accuracy machine learning (ML) algorithms (accuracy values ranging from 0.83 to 0.90) to predict sepsis-associated poor outcomes (3-day intensive care unit stay or death). Local Interpretable Model-Agnostic Explanation (LIME) and Shapley Additive Value (SHAP) interpretability methods were utilized to assess the contributions of individual hematologic parameters.

RESULTS

The ML interpretability analysis indicated that the predictive value of MDW is significantly reduced when other hematological parameters and vital signs are considered. The results suggest that complete blood count with differential (CBD-DIFF) alongside vital signs can effectively replace MDW in high accuracy machine learning algorithms for screening poor outcome associated with sepsis.

CONCLUSION

MDW, although a newly approved biomarker for sepsis, does not significantly enhance prediction models when combined with routinely available parameters and vital signs. Hospitals, especially those with resource constraints, can rely on existing parameters with high accuracy machine learning models to predict sepsis outcomes effectively, thereby reducing the need for specialized tests like MDW.

The impact of laboratory data missingness on sepsis diagnosis timeliness

Objective

To investigate the impact of missing laboratory measurements on sepsis diagnostic delays.

Materials and Methods

In adult patients admitted to 2 University of California San Diego (UCSD) hospitals from January 1, 2021 to June 30, 2024, we evaluated the relative time of organ failure (T OF) and time of clinical suspicion of sepsis (T suspicion) in patients with sepsis according to the Centers for Medicare & Medicaid Services (CMS) definition.

Results

Of the patients studied, 48.7% (n = 2017) in the emergency department (ED), 30.8% (n = 209) in the wards, and 14.4% (n = 167) in the intensive care unit (ICU) had T OF after T suspicion. Patients with T OF after T suspicion had significantly higher data missingness of 1 or more of the 5 laboratory components used to determine organ failure. The mean number of missing labs was 4.23 vs 2.83 in the ED, 4.04 vs 3.38 in the wards, and 3.98 vs 3.19 in the ICU.

Discussion

Our study identified many sepsis patients with missing laboratory results vital for the identification of organ failure and the diagnosis of sepsis at or before the time of clinical suspicion of sepsis. Addressing data missingness via more timely laboratory assessment could precipitate an earlier recognition of organ failure and potentially earlier diagnosis of and treatment initiation for sepsis.

Conclusions

More prompt laboratory assessment might improve the timeliness of sepsis recognition and treatment.

Investigating computational models for diagnosis and prognosis of sepsis based on clinical parameters: Opportunities, challenges, and future research directions

This study investigates the use of computational frameworks for sepsis. We consider two dimensions for investigation - early diagnosis of sepsis (EDS) and mortality prediction rate for sepsis patients (MPS). We concentrate on the clinical parameters on which sepsis diagnosis and prognosis are currently done, including customized treatment plans based on historical data of the patient. We identify the most notable literature that uses computational models to address EDS and MPS based on those clinical parameters. In addition to the review of the computational models built upon the clinical parameters, we also provide details regarding the popular publicly available data sources. We provide brief reviews for each model in terms of prior art and present an analysis of their results, as claimed by the respective authors. With respect to the use of machine learning models, we have provided avenues for model analysis in terms of model selection, model validation, model interpretation, and model comparison. We further present the challenges and limitations of the use of computational models, providing future research directions. This study intends to serve as a benchmark for first-hand impressions on the use of computational models for EDS and MPS of sepsis, along with the details regarding which model has been the most promising to date. We have provided details regarding all the ML models that have been used to date for EDS and MPS of sepsis.

Improving Triage Accuracy in Prehospital Emergency Telemedicine: Scoping Review of Machine Learning-Enhanced Approaches

BACKGROUND

Prehospital telemedicine triage systems combined with machine learning (ML) methods have the potential to improve triage accuracy and safely redirect low-acuity patients from attending the emergency department. However, research in prehospital settings is limited but needed; emergency department overcrowding and adverse patient outcomes are increasingly common.

OBJECTIVE

In this scoping review, we sought to characterize the existing methods for ML-enhanced telemedicine emergency triage. In order to support future research, we aimed to delineate what data sources, predictors, labels, ML models, and performance metrics were used, and in which telemedicine triage systems these methods were applied.

METHODS

A scoping review was conducted, querying multiple databases (MEDLINE, PubMed, Scopus, and IEEE Xplore) through February 24, 2023, to identify potential ML-enhanced methods, and for those eligible, relevant study characteristics were extracted, including prehospital triage setting, types of predictors, ground truth labeling method, ML models used, and performance metrics. Inclusion criteria were restricted to the triage of emergency telemedicine services using ML methods on an undifferentiated (disease nonspecific) population. Only primary research studies in English were considered. Furthermore, only those studies using data collected remotely (as opposed to derived from physical assessments) were included. In order to limit bias, we exclusively included articles identified through our predefined search criteria and had 3 researchers (DR, JS, and KS) independently screen the resulting studies. We conducted a narrative synthesis of findings to establish a knowledge base in this domain and identify potential gaps to be addressed in forthcoming ML-enhanced methods.

RESULTS

A total of 165 unique records were screened for eligibility and 15 were included in the review. Most studies applied ML methods during emergency medical dispatch (7/15, 47%) or used chatbot applications (5/15, 33%). Patient demographics and health status variables were the most common predictors, with a notable absence of social variables. Frequently used ML models included support vector machines and tree-based methods. ML-enhanced models typically outperformed conventional triage algorithms, and we found a wide range of methods used to establish ground truth labels.

CONCLUSIONS

This scoping review observed heterogeneity in dataset size, predictors, clinical setting (triage process), and reported performance metrics. Standard structured predictors, including age, sex, and comorbidities, across articles suggest the importance of these inputs; however, there was a notable absence of other potentially useful data, including medications, social variables, and health system exposure. Ground truth labeling practices should be reported in a standard fashion as the true model performance hinges on these labels. This review calls for future work to form a standardized framework, thereby supporting consistent reporting and performance comparisons across ML-enhanced prehospital triage systems.

Susceptibility to childhood sepsis, contemporary management, and future directions

Sepsis disproportionally affects children across all health-care settings and is one of the leading causes of morbidity and mortality in neonatal and paediatric age groups. As shown in the first paper in this Series, the age-specific incidence of sepsis is highest during the first years of life, before approaching adult incidence rates during adolescence. In the second paper in this Series, we focus on the unique susceptibility of paediatric patients to sepsis and how the underlying dysregulated host response relates to developmental aspects of children's immune system, genetic, perinatal, and environmental factors, and comorbidities and socioeconomic determinants of health, which often differ between children and adults. State-of-the-art clinical management of paediatric sepsis is organised around three treatment pillars-diagnosis, early resuscitation, and titration of advanced care-and we examine available treatment guidelines and the limitations of their supporting evidence. Serious evidence gaps remain in key areas of paediatric sepsis care, especially surrounding recognition, common interventions, and survivor support, and to this end we offer a research roadmap for the next decade that could accelerate targeted diagnostics and personalised use of immunomodulation. However, improving outcomes for children with sepsis relies fundamentally on systematic quality improvement in both recognition and treatment, which is the theme of the third paper in this Series. Digital health, as shown in the fourth and final paper of this Series, holds promising potential in breaking down the barriers that hinder progress in paediatric sepsis care and, ultimately, global child health.

Platelet Metabolites as Candidate Biomarkers in Sepsis Diagnosis and Management Using the Proposed Explainable Artificial Intelligence Approach

Background: Sepsis is characterized by an atypical immune response to infection and is a dangerous health problem leading to significant mortality. Current diagnostic methods exhibit insufficient sensitivity and specificity and require the discovery of precise biomarkers for the early diagnosis and treatment of sepsis. Platelets, known for their hemostatic abilities, also play an important role in immunological responses. This study aims to develop a model integrating machine learning and explainable artificial intelligence (XAI) to identify novel platelet metabolomics markers of sepsis. Methods: A total of 39 participants, 25 diagnosed with sepsis and 14 control subjects, were included in the study. The profiles of platelet metabolites were analyzed using quantitative 1H-nuclear magnetic resonance (NMR) technology. Data were processed using the synthetic minority oversampling method (SMOTE)-Tomek to address the issue of class imbalance. In addition, missing data were filled using a technique based on random forests. Three machine learning models, namely extreme gradient boosting (XGBoost), light gradient boosting machine (LightGBM), and kernel tree boosting (KTBoost), were used for sepsis prediction. The models were validated using cross-validation. Clinical annotations of the optimal sepsis prediction model were analyzed using SHapley Additive exPlanations (SHAP), an XAI technique. Results: The results showed that the KTBoost model (0.900 accuracy and 0.943 AUC) achieved better performance than the other models in sepsis diagnosis. SHAP results revealed that metabolites such as carnitine, glutamate, and myo-inositol are important biomarkers in sepsis prediction and intuitively explained the prediction decisions of the model. Conclusion: Platelet metabolites identified by the KTBoost model and XAI have significant potential for the early diagnosis and monitoring of sepsis and improving patient outcomes.

Artificial intelligence and prescription of antibiotic therapy: present and future

INTRODUCTION

In the past few years, the use of artificial intelligence in healthcare has grown exponentially. Prescription of antibiotics is not exempt from its rapid diffusion, and various machine learning (ML) techniques, from logistic regression to deep neural networks and large language models, have been explored in the literature to support decisions regarding antibiotic prescription.

AREAS COVERED

In this narrative review, we discuss promises and challenges of the application of ML-based clinical decision support systems (ML-CDSSs) for antibiotic prescription. A search was conducted in PubMed up to April 2024.

EXPERT OPINION

Prescribing antibiotics is a complex process involving various dynamic phases. In each of these phases, the support of ML-CDSSs has shown the potential, and also the actual ability in some studies, to favorably impacting relevant clinical outcomes. Nonetheless, before widely exploiting this massive potential, there are still crucial challenges ahead that are being intensively investigated, pertaining to the transparency of training data, the definition of the sufficient degree of prediction explanations when predictions are obtained through black box models, and the legal and ethical framework for decision responsibility whenever an antibiotic prescription is supported by ML-CDSSs.

Toward Realizing the Promise of AI in Precision Health Across the Spectrum of Care

Significant progress has been made in augmenting clinical decision-making using artificial intelligence (AI) in the context of secondary and tertiary care at large academic medical centers. For such innovations to have an impact across the spectrum of care, additional challenges must be addressed, including inconsistent use of preventative care and gaps in chronic care management. The integration of additional data, including genomics and data from wearables, could prove critical in addressing these gaps, but technical, legal, and ethical challenges arise. On the technical side, approaches for integrating complex and messy data are needed. Data and design imperfections like selection bias, missing data, and confounding must be addressed. In terms of legal and ethical challenges, while AI has the potential to aid in leveraging patient data to make clinical care decisions, we also risk exacerbating existing disparities. Organizations implementing AI solutions must carefully consider how they can improve care for all and reduce inequities.

Early Diagnosis of Sepsis: The Role of Biomarkers and Rapid Microbiological Tests

Sepsis is a medical emergency resulting from a dysregulated response to an infection, causing preventable deaths and a high burden of morbidity. Protocolized and accurate interventions in sepsis are time-critical. Therefore, earlier recognition of cases allows for preventive interventions, early treatment, and improved outcomes. Clinical diagnosis of sepsis by clinical scores cannot be considered an early diagnosis, given that underlying molecular pathophysiological mechanisms have been activated in the preceding hour or days. There is a lack of a widely available tool enhancing preclinical diagnosis of sepsis. Sophisticated technologies for sepsis prediction have several limitations, including high costs. Novel technologies for fast molecular and microbiological diagnosis are focusing on bedside point-of-care combined testing to reach most settings where sepsis represents a challenge.

Improving sepsis prediction in intensive care with SepsisAI: A clinical decision support system with a focus on minimizing false alarms

Prediction of sepsis using machine-learning approaches has recently gained traction. However, the lack of translation of these algorithms into clinical routine remains a major issue. Existing early sepsis detection methods are either based on the older definition of sepsis or do not accurately detect sepsis leading to the high frequency of false-positive alarms. This results in a well-known issue of clinicians' "alarm fatigue", leading to decreased responsiveness and identification, ultimately resulting in delayed clinical intervention. Hence, there is a fundamental, unmet need for a clinical decision system capable of accurate and timely sepsis diagnosis, running at the point of need. In this work, SepsisAI-a deep-learning algorithm based on long short-term memory (LSTM) networks was developed to predict the early onset of hospital-acquired sepsis in real-time for patients admitted to the ICU. The models are trained and validated with data from the PhysioNet Challenge, consisting of 40,336 patient data files from two healthcare systems: Beth Israel Deaconess Medical Center and Emory University Hospital. In the short term, the algorithm tracks frequently measured vital signs, sparsely available lab parameters, demographic features, and certain derived features for making predictions. A real-time alert system, which monitors the trajectory of the predictions, is developed on top of the deep-learning framework to minimize false alarms. On a balanced test dataset, the model achieves an AUROC, AUPRC, sensitivity, and specificity of 0.95, 0.96, 88.19%, and 96.75%, respectively at the patient level. In terms of lookahead time, the model issues a warning at a median of 6 hours (IQR 6 to 20 hours) and raises an alert at a median of 4 hours (IQR 2 to 5 hours) ahead of sepsis onset. Most importantly, the model achieves a false-alarm ratio of 3.18% for alerts, which is significantly less than other sepsis alarm systems. Additionally, on a disease prevalence-based test set, the algorithm reported similar outcomes with AUROC and AUPRC of 0.94 and 0.87, respectively, with sensitivity, and specificity of 97.05%, and 96.75%, respectively. The proposed algorithm might serve as a clinical decision support system to assist clinicians in the accurate and timely diagnosis of sepsis. With exceptionally high specificity and low false-alarm rate, this algorithm also helps mitigate the well-known issue of clinician alert fatigue arising from currently proposed sepsis alarm systems. Consequently, the algorithm partially addresses the challenges of successfully integrating machine-learning algorithms into routine clinical care.

Real-time machine learning-assisted sepsis alert enhances the timeliness of antibiotic administration and diagnostic accuracy in emergency department patients with sepsis: a cluster-randomized trial

Machine learning (ML) has been applied in sepsis recognition across different healthcare settings with outstanding diagnostic accuracy. However, the advantage of ML-assisted sepsis alert in expediting clinical decisions leading to enhanced quality for emergency department (ED) patients remains unclear. A cluster-randomized trial was conducted in a tertiary-care hospital. Adult patient data were subjected to an ML model for sepsis alert. Patient visits were assigned into one of two groups. In the intervention cluster, staff received alerts on a display screen if patients met the ML threshold for sepsis diagnosis, while patients in the control cluster followed the regular alert process. The study compared triage-to-antibiotic (TTA) time, length of stay, and mortality rate between the two groups. Additionally, the diagnostic performance of the ML model was assessed. A total of 256 (intervention) and 318 (control) sepsis patients were analyzed. The proportions of patients who received antibiotics within 1 and 3 h were higher in the intervention group than in the control group (in 1 h; 68.4 vs. 60.1%, respectively; P = 0.04, in 3 h; 94.5 vs. 89.0%, respectively; P = 0.02). The median TTA times were marginally shorter in the intervention group (46 vs. 50 min). The area under the receiver operating characteristic curve (AUROC) of ML in early sepsis identification was significantly higher than qSOFA, SIRS, and MEWS. The ML-assisted sepsis alert system may help sepsis ED patients receive antibiotics more rapidly than with the conventional, human-dedicated alert process. The diagnostic performance of ML in prompt sepsis detection was superior to that of the rule-based system.Trial registration Thai Clinical Trials Registry TCTR20230120001. Registered 16 January 2023-Retrospectively registered, https://www.thaiclinicaltrials.org/show/TCTR20230120001 .

Compartmentalization of the inflammatory response during bacterial sepsis and severe COVID-19

Acute infections cause local and systemic disorders which can lead in the most severe forms to multi-organ failure and eventually to death. The host response to infection encompasses a large spectrum of reactions with a concomitant activation of the so-called inflammatory response aimed at fighting the infectious agent and removing damaged tissues or cells, and the anti-inflammatory response aimed at controlling inflammation and initiating the healing process. Fine-tuning at the local and systemic levels is key to preventing local and remote injury due to immune system activation. Thus, during bacterial sepsis and Coronavirus disease 2019 (COVID-19), concomitant systemic and compartmentalized pro-inflammatory and compensatory anti-inflammatory responses are occurring. Immune cells (e.g., macrophages, neutrophils, natural killer cells, and T-lymphocytes), as well as endothelial cells, differ from one compartment to another and contribute to specific organ responses to sterile and microbial insult. Furthermore, tissue-specific microbiota influences the local and systemic response. A better understanding of the tissue-specific immune status, the organ immunity crosstalk, and the role of specific mediators during sepsis and COVID-19 can foster the development of more accurate biomarkers for better diagnosis and prognosis and help to define appropriate host-targeted treatments and vaccines in the context of precision medicine.

Early Warning Systems for Critical Illness Outside the Intensive Care Unit

Early warning systems (EWSs) are designed and deployed to create a rapid assessment and response for patients with clinical deterioration outside the intensive care unit (ICU). These models incorporate patient-level data such as vital signs and laboratory values to detect or prevent adverse clinical events, such as vital signs and laboratories to allow detection and prevention of adverse clinical events such as cardiac arrest, intensive care transfer, or sepsis. The applicability, development, clinical utility, and general perception of EWS in clinical practice vary widely. Here, we review the field as it has grown from early vital sign-based scoring systems to contemporary multidimensional algorithms and predictive technologies for clinical decompensation outside the ICU.

Effect of Digital Early Warning Scores on Hospital Vital Sign Observation Protocol Adherence: Stepped-Wedge Evaluation

BACKGROUND

Early warning scores (EWS) are routinely used in hospitals to assess a patient's risk of deterioration. EWS are traditionally recorded on paper observation charts but are increasingly recorded digitally. In either case, evidence for the clinical effectiveness of such scores is mixed, and previous studies have not considered whether EWS leads to changes in how deteriorating patients are managed.

OBJECTIVE

This study aims to examine whether the introduction of a digital EWS system was associated with more frequent observation of patients with abnormal vital signs, a precursor to earlier clinical intervention.

METHODS

We conducted a 2-armed stepped-wedge study from February 2015 to December 2016, over 4 hospitals in 1 UK hospital trust. In the control arm, vital signs were recorded using paper observation charts. In the intervention arm, a digital EWS system was used. The primary outcome measure was time to next observation (TTNO), defined as the time between a patient's first elevated EWS (EWS ≥3) and subsequent observations set. Secondary outcomes were time to death in the hospital, length of stay, and time to unplanned intensive care unit admission. Differences between the 2 arms were analyzed using a mixed-effects Cox model. The usability of the system was assessed using the system usability score survey.

RESULTS

We included 12,802 admissions, 1084 in the paper (control) arm and 11,718 in the digital EWS (intervention) arm. The system usability score was 77.6, indicating good usability. The median TTNO in the control and intervention arms were 128 (IQR 73-218) minutes and 131 (IQR 73-223) minutes, respectively. The corresponding hazard ratio for TTNO was 0.99 (95% CI 0.91-1.07; P=.73).

CONCLUSIONS

We demonstrated strong clinical engagement with the system. We found no difference in any of the predefined patient outcomes, suggesting that the introduction of a highly usable electronic system can be achieved without impacting clinical care. Our findings contrast with previous claims that digital EWS systems are associated with improvement in clinical outcomes. Future research should investigate how digital EWS systems can be integrated with new clinical pathways adjusting staff behaviors to improve patient outcomes.

Strengthening the use of artificial intelligence within healthcare delivery organizations: balancing regulatory compliance and patient safety

OBJECTIVES

Surface the urgent dilemma that healthcare delivery organizations (HDOs) face navigating the US Food and Drug Administration (FDA) final guidance on the use of clinical decision support (CDS) software.

MATERIALS AND METHODS

We use sepsis as a case study to highlight the patient safety and regulatory compliance tradeoffs that 6129 hospitals in the United States must navigate.

RESULTS

Sepsis CDS remains in broad, routine use. There is no commercially available sepsis CDS system that is FDA cleared as a medical device. There is no public disclosure of an HDO turning off sepsis CDS due to regulatory compliance concerns. And there is no public disclosure of FDA enforcement action against an HDO for using sepsis CDS that is not cleared as a medical device.

DISCUSSION AND CONCLUSION

We present multiple policy interventions that would relieve the current tension to enable HDOs to utilize artificial intelligence to improve patient care while also addressing FDA concerns about product safety, efficacy, and equity.

Prospective validation of clinical deterioration predictive models prior to intensive care unit transfer among patients admitted to acute care cardiology wards

Objective. Very few predictive models have been externally validated in a prospective cohort following the implementation of an artificial intelligence analytic system. This type of real-world validation is critically important due to the risk of data drift, or changes in data definitions or clinical practices over time, that could impact model performance in contemporaneous real-world cohorts. In this work, we report the model performance of a predictive analytics tool developed before COVID-19 and demonstrate model performance during the COVID-19 pandemic.Approach. The analytic system (CoMETⓇ, Nihon Kohden Digital Health Solutions LLC, Irvine, CA) was implemented in a randomized controlled trial that enrolled 10 422 patient visits in a 1:1 display-on display-off design. The CoMET scores were calculated for all patients but only displayed in the display-on arm. Only the control/display-off group is reported here because the scores could not alter care patterns.Main results.Of the 5184 visits in the display-off arm, 311 experienced clinical deterioration and care escalation, resulting in transfer to the intensive care unit, primarily due to respiratory distress. The model performance of CoMET was assessed based on areas under the receiver operating characteristic curve, which ranged from 0.725 to 0.737.Significance.The models were well-calibrated, and there were dynamic increases in the model scores in the hours preceding the clinical deterioration events. A hypothetical alerting strategy based on a rise in score and duration of the rise would have had good performance, with a positive predictive value more than 10-fold the event rate. We conclude that predictive statistical models developed five years before study initiation had good model performance despite the passage of time and the impact of the COVID-19 pandemic.

Multisite Pragmatic Cluster-Randomized Controlled Trial of the CONCERN Early Warning System

Importance

Late predictions of hospitalized patient deterioration, resulting from early warning systems (EWS) with limited data sources and/or a care team's lack of shared situational awareness, contribute to delays in clinical interventions. The COmmunicating Narrative Concerns Entered by RNs (CONCERN) Early Warning System (EWS) uses real-time nursing surveillance documentation patterns in its machine learning algorithm to identify patients' deterioration risk up to 42 hours earlier than other EWSs.

Objective

To test our a priori hypothesis that patients with care teams informed by the CONCERN EWS intervention have a lower mortality rate and shorter length of stay (LOS) than the patients with teams not informed by CONCERN EWS.

Design

One-year multisite, pragmatic controlled clinical trial with cluster-randomization of acute and intensive care units to intervention or usual-care groups.

Setting

Two large U.S. health systems.

Participants

Adult patients admitted to acute and intensive care units, excluding those on hospice/palliative/comfort care, or with Do Not Resuscitate/Do Not Intubate orders.

Intervention

The CONCERN EWS intervention calculates patient deterioration risk based on nurses' concern levels measured by surveillance documentation patterns, and it displays the categorical risk score (low, increased, high) in the electronic health record (EHR) for care team members.

Main Outcomes and Measures

Primary outcomes: in-hospital mortality, LOS; survival analysis was used. Secondary outcomes: cardiopulmonary arrest, sepsis, unanticipated ICU transfers, 30-day hospital readmission.

Results

A total of 60 893 hospital encounters (33 024 intervention and 27 869 usual-care) were included. Both groups had similar patient age, race, ethnicity, and illness severity distributions. Patients in the intervention group had a 35.6% decreased risk of death (adjusted hazard ratio [HR], 0.644; 95% confidence interval [CI], 0.532-0.778; P<.0001), 11.2% decreased LOS (adjusted incidence rate ratio, 0.914; 95% CI, 0.902-0.926; P<.0001), 7.5% decreased risk of sepsis (adjusted HR, 0.925; 95% CI, 0.861-0.993; P=.0317), and 24.9% increased risk of unanticipated ICU transfer (adjusted HR, 1.249; 95% CI, 1.093-1.426; P=.0011) compared with patients in the usual-care group.

Conclusions and Relevance

A hospital-wide EWS based on nursing surveillance patterns decreased in-hospital mortality, sepsis, and LOS when integrated into the care team's EHR workflow.

Trial Registration

ClinicalTrials.gov Identifier: NCT03911687.

[Data-driven intensive care: a lack of comprehensive datasets]

Intensive care units provide a data-rich environment with the potential to generate datasets in the realm of big data, which could be utilized to train powerful machine learning (ML) models. However, the currently available datasets are too small and exhibit too little diversity due to their limitation to individual hospitals. This lack of extensive and varied datasets is a primary reason for the limited generalizability and resulting low clinical utility of current ML models. Often, these models are based on data from single centers and suffer from poor external validity. There is an urgent need for the development of large-scale, multicentric, and multinational datasets. Ensuring data protection and minimizing re-identification risks pose central challenges in this process. The "Amsterdam University Medical Center database (AmsterdamUMCdb)" and the "Salzburg Intensive Care database (SICdb)" demonstrate that open access datasets are possible in Europe while complying with the data protection regulations of the General Data Protection Regulation (GDPR). Another challenge in building intensive care datasets is the absence of semantic definitions in the source data and the heterogeneity of data formats. Establishing binding industry standards for the semantic definition is crucial to ensure seamless semantic interoperability between datasets.

Hospital Acquired Sepsis, Disease Prevalence, and Recent Advances in Sepsis Mitigation

Sepsis is a life-threatening organ dysfunction caused by a dysregulated host response to infection, commonly associated with nosocomial transmission. Gram-negative bacterial species are particularly problematic due to the release of the lipopolysaccharide toxins upon cell death. The lipopolysaccharide toxin of E. coli has a greater immunogenic potential than that of other Gram-negative bacteria. The resultant dysregulation of the immune system is associated with organ failure and mortality, with pregnant women, ICU patients, and neonates being particularly vulnerable. Additionally, sepsis recovery patients have an increased risk of re-hospitalisation, chronic illness, co-morbidities, organ damage/failure, and a reduced life expectancy. The emergence and increasing prevalence of antimicrobial resistance in bacterial and fungal species has impacted the treatment of sepsis patients, leading to increasing mortality rates. Multidrug resistant pathogens including vancomycin-resistant Enterococcus, beta lactam-resistant Klebsiella, and carbapenem-resistant Acinetobacter species are associated with an increased risk of mortality. To improve the prognosis of sepsis patients, predominantly high-risk neonates, advances must be made in the early diagnosis, triage, and control of sepsis. The identification of suitable biomarkers and biomarker combinations, coupled with machine learning and artificial intelligence, show promise in early detection protocols. Rapid diagnosis of sepsis in patients is essential to inform on clinical treatment, especially with resistant infectious agents. This timely review aims to discuss sepsis prevalence, aetiology, and recent advances towards disease mitigation and control.

Objectivizing issues in the diagnosis of complex rare diseases: lessons learned from testing existing diagnosis support systems on ciliopathies

BACKGROUND

There are approximately 8,000 different rare diseases that affect roughly 400 million people worldwide. Many of them suffer from delayed diagnosis. Ciliopathies are rare monogenic disorders characterized by a significant phenotypic and genetic heterogeneity that raises an important challenge for clinical diagnosis. Diagnosis support systems (DSS) applied to electronic health record (EHR) data may help identify undiagnosed patients, which is of paramount importance to improve patients' care. Our objective was to evaluate three online-accessible rare disease DSSs using phenotypes derived from EHRs for the diagnosis of ciliopathies.

METHODS

Two datasets of ciliopathy cases, either proven or suspected, and two datasets of controls were used to evaluate the DSSs. Patient phenotypes were automatically extracted from their EHRs and converted to Human Phenotype Ontology terms. We tested the ability of the DSSs to diagnose cases in contrast to controls based on Orphanet ontology.

RESULTS

A total of 79 cases and 38 controls were selected. Performances of the DSSs on ciliopathy real world data (best DSS with area under the ROC curve = 0.72) were not as good as published performances on the test set used in the DSS development phase. None of these systems obtained results which could be described as "expert-level". Patients with multisystemic symptoms were generally easier to diagnose than patients with isolated symptoms. Diseases easily confused with ciliopathy generally affected multiple organs and had overlapping phenotypes. Four challenges need to be considered to improve the performances: to make the DSSs interoperable with EHR systems, to validate the performances in real-life settings, to deal with data quality, and to leverage methods and resources for rare and complex diseases.

CONCLUSION

Our study provides insights into the complexities of diagnosing highly heterogenous rare diseases and offers lessons derived from evaluation existing DSSs in real-world settings. These insights are not only beneficial for ciliopathy diagnosis but also hold relevance for the enhancement of DSS for various complex rare disorders, by guiding the development of more clinically relevant rare disease DSSs, that could support early diagnosis and finally make more patients eligible for treatment.

Ensuring useful adoption of generative artificial intelligence in healthcare

OBJECTIVES

This article aims to examine how generative artificial intelligence (AI) can be adopted with the most value in health systems, in response to the Executive Order on AI.

MATERIALS AND METHODS

We reviewed how technology has historically been deployed in healthcare, and evaluated recent examples of deployments of both traditional AI and generative AI (GenAI) with a lens on value.

RESULTS

Traditional AI and GenAI are different technologies in terms of their capability and modes of current deployment, which have implications on value in health systems.

DISCUSSION

Traditional AI when applied with a framework top-down can realize value in healthcare. GenAI in the short term when applied top-down has unclear value, but encouraging more bottom-up adoption has the potential to provide more benefit to health systems and patients.

CONCLUSION

GenAI in healthcare can provide the most value for patients when health systems adapt culturally to grow with this new technology and its adoption patterns.

Development and preliminary testing of Health Equity Across the AI Lifecycle (HEAAL): A framework for healthcare delivery organizations to mitigate the risk of AI solutions worsening health inequities

The use of data-driven technologies such as Artificial Intelligence (AI) and Machine Learning (ML) is growing in healthcare. However, the proliferation of healthcare AI tools has outpaced regulatory frameworks, accountability measures, and governance standards to ensure safe, effective, and equitable use. To address these gaps and tackle a common challenge faced by healthcare delivery organizations, a case-based workshop was organized, and a framework was developed to evaluate the potential impact of implementing an AI solution on health equity. The Health Equity Across the AI Lifecycle (HEAAL) is co-designed with extensive engagement of clinical, operational, technical, and regulatory leaders across healthcare delivery organizations and ecosystem partners in the US. It assesses 5 equity assessment domains-accountability, fairness, fitness for purpose, reliability and validity, and transparency-across the span of eight key decision points in the AI adoption lifecycle. It is a process-oriented framework containing 37 step-by-step procedures for evaluating an existing AI solution and 34 procedures for evaluating a new AI solution in total. Within each procedure, it identifies relevant key stakeholders and data sources used to conduct the procedure. HEAAL guides how healthcare delivery organizations may mitigate the potential risk of AI solutions worsening health inequities. It also informs how much resources and support are required to assess the potential impact of AI solutions on health inequities.

Can Machine Learning Personalize Cardiovascular Therapy in Sepsis?

Large randomized trials in sepsis have generally failed to find effective novel treatments. This is increasingly attributed to patient heterogeneity, including heterogeneous cardiovascular changes in septic shock. We discuss the potential for machine learning systems to personalize cardiovascular resuscitation in sepsis. While the literature is replete with proofs of concept, the technological readiness of current systems is low, with a paucity of clinical trials and proven patient benefit. Systems may be vulnerable to confounding and poor generalization to new patient populations or contemporary patterns of care. Typical electronic health records do not capture rich enough data, at sufficient temporal resolution, to produce systems that make actionable treatment suggestions. To resolve these issues, we recommend a simultaneous focus on technical challenges and removing barriers to translation. This will involve improving data quality, adopting causally grounded models, prioritizing safety assessment and integration into healthcare workflows, conducting randomized clinical trials and aligning with regulatory requirements.

Experts vs. machine - comparison of machine learning to expert-informed prediction of outcome after major liver surgery

BACKGROUND

Machine learning (ML) has been successfully implemented for classification tasks (e.g., cancer diagnosis). ML performance for more challenging predictions is largely unexplored. This study's objective was to compare machine learning vs. expert-informed predictions for surgical outcome in patients undergoing major liver surgery.

METHODS

Single tertiary center data on preoperative parameters and postoperative complications for elective hepatic surgery patients were included (2008-2021). Expert-informed prediction models were established on 14 parameters identified by two expert liver surgeons to impact on postoperative outcome. ML models used all available preoperative patient variables (n = 62). Model performance was compared for predicting 3-month postoperative overall morbidity. Temporal validation and additional analysis in major liver resection patients were conducted.

RESULTS

889 patients included. Expert-informed models showed low average bias (2-5 CCI points) with high over/underprediction. ML models performed similarly: average prediction 5-10 points higher than observed CCI values with high variability (95% CI -30 to 50). No performance improvement for major liver surgery patients.

CONCLUSION

No clinical relevance in the application of ML for predicting postoperative overall morbidity was found. Despite being a novel hype, ML has the potential for application in clinical practice. However, at this stage it does not replace established approaches of prediction modelling.

Measuring algorithmic bias to analyze the reliability of AI tools that predict depression risk using smartphone sensed-behavioral data

AI tools intend to transform mental healthcare by providing remote estimates of depression risk using behavioral data collected by sensors embedded in smartphones. While these tools accurately predict elevated depression symptoms in small, homogenous populations, recent studies show that these tools are less accurate in larger, more diverse populations. In this work, we show that accuracy is reduced because sensed-behaviors are unreliable predictors of depression across individuals: sensed-behaviors that predict depression risk are inconsistent across demographic and socioeconomic subgroups. We first identified subgroups where a developed AI tool underperformed by measuring algorithmic bias, where subgroups with depression were incorrectly predicted to be at lower risk than healthier subgroups. We then found inconsistencies between sensed-behaviors predictive of depression across these subgroups. Our findings suggest that researchers developing AI tools predicting mental health from sensed-behaviors should think critically about the generalizability of these tools, and consider tailored solutions for targeted populations.

Measuring algorithmic bias to analyze the reliability of AI tools that predict depression risk using smartphone sensed-behavioral data

AI tools intend to transform mental healthcare by providing remote estimates of depression risk using behavioral data collected by sensors embedded in smartphones. While these tools accurately predict elevated symptoms in small, homogenous populations, recent studies show that these tools are less accurate in larger, more diverse populations. In this work, we show that accuracy is reduced because sensed-behaviors are unreliable predictors of depression across individuals; specifically the sensed-behaviors that predict depression risk are inconsistent across demographic and socioeconomic subgroups. We first identified subgroups where a developed AI tool underperformed by measuring algorithmic bias, where subgroups with depression were incorrectly predicted to be at lower risk than healthier subgroups. We then found inconsistencies between sensed-behaviors predictive of depression across these subgroups. Our findings suggest that researchers developing AI tools predicting mental health from behavior should think critically about the generalizability of these tools, and consider tailored solutions for targeted populations.

Use of artificial intelligence in critical care: opportunities and obstacles

BACKGROUND

Perhaps nowhere else in the healthcare system than in the intensive care unit environment are the challenges to create useful models with direct time-critical clinical applications more relevant and the obstacles to achieving those goals more massive. Machine learning-based artificial intelligence (AI) techniques to define states and predict future events are commonplace activities of modern life. However, their penetration into acute care medicine has been slow, stuttering and uneven. Major obstacles to widespread effective application of AI approaches to the real-time care of the critically ill patient exist and need to be addressed.

MAIN BODY

Clinical decision support systems (CDSSs) in acute and critical care environments support clinicians, not replace them at the bedside. As will be discussed in this review, the reasons are many and include the immaturity of AI-based systems to have situational awareness, the fundamental bias in many large databases that do not reflect the target population of patient being treated making fairness an important issue to address and technical barriers to the timely access to valid data and its display in a fashion useful for clinical workflow. The inherent "black-box" nature of many predictive algorithms and CDSS makes trustworthiness and acceptance by the medical community difficult. Logistically, collating and curating in real-time multidimensional data streams of various sources needed to inform the algorithms and ultimately display relevant clinical decisions support format that adapt to individual patient responses and signatures represent the efferent limb of these systems and is often ignored during initial validation efforts. Similarly, legal and commercial barriers to the access to many existing clinical databases limit studies to address fairness and generalizability of predictive models and management tools.

CONCLUSIONS

AI-based CDSS are evolving and are here to stay. It is our obligation to be good shepherds of their use and further development.

Immune Dysfunction-Associated Elevated RDW, APACHE-II, and SOFA Scores Were a Possible Cause of 28-Day Mortality in Sepsis Patients

Objective

To explore the early predictors and their predicting value of 28-day mortality in sepsis patients and to investigate the possible causes of death.

Methods

127 sepsis patients were included, including 79 cases in the survival group and 48 cases in the death group. The results of all patients on admission were recorded. After screening the risk factors of 28-day mortality, the receiver operating characteristic curve (ROC) was used to determine their predictive value for the 28-day mortality rate on admission, and the Kaplan-Meier curve was drawn to compare the 28-day mortality rate between groups. Finally, patients with cytokine and lymphocyte subsets results were included for investigating the possible causes of death through correlation analysis.

Results

APACHE II (acute physiology and chronic health evaluation II), SOFA (Sequential Organ Failure Assessment) and red blood cell distribution width (RDW) were the risk factors for 28-day mortality in sepsis patients (OR: 1.130 vs.1.160 vs.1.530, P < 0.05). The area under the curve (AUC), sensitivity and specificity of APACHE II, SOFA and RDW in predicting the mortality rate at 28 days after admission in sepsis patients were 0.763 vs 0.806 vs 0.723, 79.2% vs 68.8% vs 75.0%, 65.8% vs 89.9% vs 68.4%. The combined predicted AUC was 0.873, the sensitivity was 89.6%, and the specificity was 82.3%. The Kaplan-Meier survival curve showed that the 28-day mortality rates of sepsis patients with APACHE II≥18.5, SOFA≥11.5 and RDW≥13.8 were 58.5%, 80.5% and 59.0%, respectively. In the death group, APACHE II was positively correlated with SOFA, IL-2, and IL-10, and RDW was positively correlated with PLT, TNF-α, CD3+ lymphocyte count, and CD8+ lymphocyte count.

Conclusion

Sepsis patients with high APACHE II, SOFA and RDW levels at admission have an increased 28-day mortality rate. The elevation of these indicators in dead patients are related to immune dysfunction.

The Impact of Digital Hospitals on Patient and Clinician Experience: Systematic Review and Qualitative Evidence Synthesis

BACKGROUND

The digital transformation of health care is advancing rapidly. A well-accepted framework for health care improvement is the Quadruple Aim: improved clinician experience, improved patient experience, improved population health, and reduced health care costs. Hospitals are attempting to improve care by using digital technologies, but the effectiveness of these technologies is often only measured against cost and quality indicators, and less is known about the clinician and patient experience.

OBJECTIVE

This study aims to conduct a systematic review and qualitative evidence synthesis to assess the clinician and patient experience of digital hospitals.

METHODS

The PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) and ENTREQ (Enhancing the Transparency in Reporting the Synthesis of Qualitative Research) guidelines were followed. The PubMed, Embase, Scopus, CINAHL, and PsycINFO databases were searched from January 2010 to June 2022. Studies that explored multidisciplinary clinician or adult inpatient experiences of digital hospitals (with a full electronic medical record) were included. Study quality was assessed using the Mixed Methods Appraisal Tool. Data synthesis was performed narratively for quantitative studies. Qualitative evidence synthesis was performed via (1) automated machine learning text analytics using Leximancer (Leximancer Pty Ltd) and (2) researcher-led inductive synthesis to generate themes.

RESULTS

A total of 61 studies (n=39, 64% quantitative; n=15, 25% qualitative; and n=7, 11% mixed methods) were included. Most studies (55/61, 90%) investigated clinician experiences, whereas few (10/61, 16%) investigated patient experiences. The study populations ranged from 8 to 3610 clinicians, 11 to 34,425 patients, and 5 to 2836 hospitals. Quantitative outcomes indicated that clinicians had a positive overall satisfaction (17/24, 71% of the studies) with digital hospitals, and most studies (11/19, 58%) reported a positive sentiment toward usability. Data accessibility was reported positively, whereas adaptation, clinician-patient interaction, and workload burnout were reported negatively. The effects of digital hospitals on patient safety and clinicians' ability to deliver patient care were mixed. The qualitative evidence synthesis of clinician experience studies (18/61, 30%) generated 7 themes: inefficient digital documentation, inconsistent data quality, disruptions to conventional health care relationships, acceptance, safety versus risk, reliance on hybrid (digital and paper) workflows, and patient data privacy. There was weak evidence of a positive association between digital hospitals and patient satisfaction scores.

CONCLUSIONS

Clinicians' experience of digital hospitals appears positive according to high-level indicators (eg, overall satisfaction and data accessibility), but the qualitative evidence synthesis revealed substantive tensions. There is insufficient evidence to draw a definitive conclusion on the patient experience within digital hospitals, but indications appear positive or agnostic. Future research must prioritize equitable investigation and definition of the digital clinician and patient experience to achieve the Quadruple Aim of health care.

Applying Machine Learning to Blood Count Data Predicts Sepsis with ICU Admission

BACKGROUND

Timely diagnosis is crucial for sepsis treatment. Current machine learning (ML) models suffer from high complexity and limited applicability. We therefore created an ML model using only complete blood count (CBC) diagnostics.

METHODS

We collected non-intensive care unit (non-ICU) data from a German tertiary care centre (January 2014 to December 2021). Using patient age, sex, and CBC parameters (haemoglobin, platelets, mean corpuscular volume, white and red blood cells), we trained a boosted random forest, which predicts sepsis with ICU admission. Two external validations were conducted using data from another German tertiary care centre and the Medical Information Mart for Intensive Care IV database (MIMIC-IV). Using the subset of laboratory orders also including procalcitonin (PCT), an analogous model was trained with PCT as an additional feature.

RESULTS

After exclusion, 1 381 358 laboratory requests (2016 from sepsis cases) were available. The CBC model shows an area under the receiver operating characteristic (AUROC) of 0.872 (95% CI, 0.857-0.887). External validations show AUROCs of 0.805 (95% CI, 0.787-0.824) for University Medicine Greifswald and 0.845 (95% CI, 0.837-0.852) for MIMIC-IV. The model including PCT revealed a significantly higher AUROC (0.857; 95% CI, 0.836-0.877) than PCT alone (0.790; 95% CI, 0.759-0.821; P < 0.001).

CONCLUSIONS

Our results demonstrate that routine CBC results could significantly improve diagnosis of sepsis when combined with ML. The CBC model can facilitate early sepsis prediction in non-ICU patients with high robustness in external validations. Its implementation in clinical decision support systems has strong potential to provide an essential time advantage and increase patient safety.

Development of a machine learning model to predict risk of development of COVID-19-associated mucormycosis

Aim: The study aimed to identify quantitative parameters that increase the risk of rhino-orbito-cerebral mucormycosis, and subsequently developed a machine learning model that can anticipate susceptibility to developing this condition. Methods: Clinicopathological data from 124 patients were used to quantify their association with COVID-19-associated mucormycosis (CAM) and subsequently develop a machine learning model to predict its likelihood. Results: Diabetes mellitus, noninvasive ventilation and hypertension were found to have statistically significant associations with radiologically confirmed CAM cases. Conclusion: Machine learning models can be used to accurately predict the likelihood of development of CAM, and this methodology can be used in creating prediction algorithms of a wide variety of infections and complications.

Maximizing Readmission Reduction in Colon Cancer Patients

INTRODUCTION

Multiple studies have identified risk factors for readmission in colon cancer patients. We need to determine which risk factors, when modified, produce the greatest decrease in readmission for patients so that limited resources can be used most effectively by implementing targeted evidence-based performance improvements. We determined the potential impact of various modifiable risk factors on reducing 30-d readmission in colon cancer patients.

METHODS

We used a cohort design with the 2012-2020 American College of Surgeons' National Surgical Quality Improvement Program data to track colon cancer patients for 30 d following surgery. Colon cancer patients who received colectomies and were discharged alive were included. Readmission (to the same or another hospital) for any reason within 30 d of the resection was the outcome measure. Modifiable risk factors were the use of minimally invasive surgery (MIS) versus open colectomy, mechanical bowel preparation, preoperative antibiotic use, functional status, smoking, complications (deep vein thrombosis, pulmonary embolism, myocardial infarction, stroke, infections, anastomotic leakage, prolonged postoperative ileus, extensive blood loss, and sepsis), serum albumin, and hematocrit.

RESULTS

111,691 patients with colon cancer were included in the analysis. About half of the patients were male, most were aged 75 or older, and were discharged home. Overall, 11,138 patients (10.0%) were readmitted within 30 d of surgery. In adjusted analysis, the reduction in readmission would be largest by preventing both prolonged ileus and by switching open colectomies to MIS (28.0% relative reduction) followed by preventing anastomotic leaks (6.2% relative reduction). Improving other modifiable risk factors would have a more limited impact.

CONCLUSIONS

The focus of readmission reduction should be on preventing prolonged ileus, increasing the use of MIS, and preventing anastomotic leaks.

Unlocking the potential of senescence-related gene signature as a diagnostic and prognostic biomarker in sepsis: insights from meta-analyses, single-cell RNA sequencing, and in vitro experiments

Cellular senescence is closely associated with the pathogenesis of sepsis. However, the diagnostic and prognostic value of senescence-related genes remain unclear. In this study, 866 senescence-related genes were collected from CellAge. The training cohort, GSE65682, which included 42 control and 760 sepsis samples, was obtained from the Gene Expression Omnibus (GEO). Feature selection was performed using gene expression difference detection, LASSO analysis, random forest, and Cox regression. TGFBI and MAD1L1 were ultimately selected for inclusion in the multivariate Cox regression model. Clustering based on the expressions of TGFBI and MAD1L1 was significantly associated with sepsis characteristics and prognoses (all P < 0.05). The risk signature served as a reliable prognostic predictor across the GSE65682, GSE95233, and GSE4607 cohorts (pooled hazard ratio = 4.27; 95% confidence interval [CI] = 1.63-11.17). Furthermore, it also served as a robust classifier to distinguish sepsis samples from control cases across 14 cohorts (pooled odds ratio = 5.88; 95% CI = 3.54-9.77). Single-cell RNA sequencing analyses from five healthy controls and four sepsis subjects indicated that the risk signature could reflect the senescence statuses of monocytes and B cells; this finding was then experimentally validated in THP-1 and IM-9 cells in vitro (both P < 0.05). In all, a senescence-related gene signature was developed as a prognostic and diagnostic biomarker for sepsis, providing cut-in points to uncover underlying mechanisms and a promising clinical tool to support precision medicine.

Machine Learning Interpretability Methods to Characterize the Importance of Hematologic Biomarkers in Prognosticating Patients with Suspected Infection

Early detection of sepsis in patients admitted to the emergency department (ED) is an important clinical objective as early identification and treatment can help reduce morbidity and mortality rate of 20% or higher. Hematologic changes during sepsis-associated organ dysfunction are well established and a new biomarker called Monocyte Distribution Width (MDW) has been recently approved by the US Food and Drug Administration for sepsis. However, MDW, which quantifies monocyte activation in sepsis patients, is not a routinely reported parameter and it requires specialized proprietary laboratory equipment. Further, the relative importance of MDW as compared to other routinely available hematologic parameters and vital signs has not been studied, which makes it difficult for resource constrained hospital systems to make informed decisions in this regard. To address this issue, we analyzed data from a cohort of ED patients (n=10,229) admitted to a large regional safety-net hospital in Cleveland, Ohio with suspected infection who later developed poor outcomes associated with sepsis. We developed a new analytical framework consisting of seven data models and an ensemble of high accuracy machine learning (ML) algorithms (accuracy values ranging from 0.83 to 0.90) for the prediction of outcomes more common in sepsis than uncomplicated infection (3-day intensive care unit stay or death). To characterize the contributions of individual hematologic parameters, we applied the Local Interpretable Model-Agnostic Explanation (LIME) and Shapley Additive Value (SHAP) interpretability methods to the high accuracy ML algorithms. The ML interpretability results were consistent in their findings that the value of MDW is grossly attenuated in the presence of other routinely reported hematologic parameters and vital signs data. Further, this study for the first time shows that complete blood count with differential (CBC-DIFF) together with vital signs data can be used as a substitute for MDW in high accuracy ML algorithms to screen for poor outcomes associated with sepsis.

Academic machine learning researchers' ethical perspectives on algorithm development for health care: a qualitative study

OBJECTIVES

We set out to describe academic machine learning (ML) researchers' ethical considerations regarding the development of ML tools intended for use in clinical care.

MATERIALS AND METHODS

We conducted in-depth, semistructured interviews with a sample of ML researchers in medicine (N = 10) as part of a larger study investigating stakeholders' ethical considerations in the translation of ML tools in medicine. We used a qualitative descriptive design, applying conventional qualitative content analysis in order to allow participant perspectives to emerge directly from the data.

RESULTS

Every participant viewed their algorithm development work as holding ethical significance. While participants shared positive attitudes toward continued ML innovation, they described concerns related to data sampling and labeling (eg, limitations to mitigating bias; ensuring the validity and integrity of data), and algorithm training and testing (eg, selecting quantitative targets; assessing reproducibility). Participants perceived a need to increase interdisciplinary training across stakeholders and to envision more coordinated and embedded approaches to addressing ethics issues.

DISCUSSION AND CONCLUSION

Participants described key areas where increased support for ethics may be needed; technical challenges affecting clinical acceptability; and standards related to scientific integrity, beneficence, and justice that may be higher in medicine compared to other industries engaged in ML innovation. Our results help shed light on the perspectives of ML researchers in medicine regarding the range of ethical issues they encounter or anticipate in their work, including areas where more attention may be needed to support the successful development and integration of medical ML tools.

Effect of Automated Real-Time Feedback on Early-Sepsis Care: A Pragmatic Clinical Trial

OBJECTIVES

To determine if a real-time monitoring system with automated clinician alerts improves 3-hour sepsis bundle adherence.

DESIGN

Prospective, pragmatic clinical trial. Allocation alternated every 7 days.

SETTING

Quaternary hospital from December 1, 2020 to November 30, 2021.

PATIENTS

Adult emergency department or inpatients meeting objective sepsis criteria triggered an electronic medical record (EMR)-embedded best practice advisory. Enrollment occurred when clinicians acknowledged the advisory indicating they felt sepsis was likely.

INTERVENTION

Real-time automated EMR monitoring identified suspected sepsis patients with incomplete bundle measures within 1-hour of completion deadlines and generated reminder pages. Clinicians responsible for intervention group patients received reminder pages; no pages were sent for controls. The primary analysis cohort was the subset of enrolled patients at risk of bundle nonadherent care that had reminder pages generated.

MEASUREMENTS AND MAIN RESULTS

The primary outcome was orders for all 3-hour bundle elements within guideline time limits. Secondary outcomes included guideline-adherent delivery of all 3-hour bundle elements, 28-day mortality, antibiotic discontinuation within 48-hours, and pathogen recovery from any culture within 7 days of time-zero. Among 3,269 enrolled patients, 1,377 had reminder pages generated and were included in the primary analysis. There were 670 (48.7%) at-risk patients randomized to paging alerts and 707 (51.3%) to control. Bundle-adherent orders were placed for 198 intervention patients (29.6%) versus 149 (21.1%) controls (difference: 8.5%; 95% CI, 3.9-13.1%; p = 0.0003). Bundle-adherent care was delivered for 152 (22.7%) intervention versus 121 (17.1%) control patients (difference: 5.6%; 95% CI, 1.4-9.8%; p = 0.0095). Mortality was similar between groups (8.4% vs 8.3%), as were early antibiotic discontinuation (35.1% vs 33.4%) and pan-culture negativity (69.0% vs 68.2%).

CONCLUSIONS

Real-time monitoring and paging alerts significantly increased orders for and delivery of guideline-adherent care for suspected sepsis patients at risk of 3-hour bundle nonadherence. The trial was underpowered to determine whether adherence affected mortality. Despite enrolling patients with clinically suspected sepsis, early antibiotic discontinuation and pan-culture negativity were common, highlighting challenges in identifying appropriate patients for sepsis bundle application.

Impact of a deep learning sepsis prediction model on quality of care and survival

Sepsis remains a major cause of mortality and morbidity worldwide. Algorithms that assist with the early recognition of sepsis may improve outcomes, but relatively few studies have examined their impact on real-world patient outcomes. Our objective was to assess the impact of a deep-learning model (COMPOSER) for the early prediction of sepsis on patient outcomes. We completed a before-and-after quasi-experimental study at two distinct Emergency Departments (EDs) within the UC San Diego Health System. We included 6217 adult septic patients from 1/1/2021 through 4/30/2023. The exposure tested was a nurse-facing Best Practice Advisory (BPA) triggered by COMPOSER. In-hospital mortality, sepsis bundle compliance, 72-h change in sequential organ failure assessment (SOFA) score following sepsis onset, ICU-free days, and the number of ICU encounters were evaluated in the pre-intervention period (705 days) and the post-intervention period (145 days). The causal impact analysis was performed using a Bayesian structural time-series approach with confounder adjustments to assess the significance of the exposure at the 95% confidence level. The deployment of COMPOSER was significantly associated with a 1.9% absolute reduction (17% relative decrease) in in-hospital sepsis mortality (95% CI, 0.3%-3.5%), a 5.0% absolute increase (10% relative increase) in sepsis bundle compliance (95% CI, 2.4%-8.0%), and a 4% (95% CI, 1.1%-7.1%) reduction in 72-h SOFA change after sepsis onset in causal inference analysis. This study suggests that the deployment of COMPOSER for early prediction of sepsis was associated with a significant reduction in mortality and a significant increase in sepsis bundle compliance.

Grand rounds in methodology: key considerations for implementing machine learning solutions in quality improvement initiatives

Machine learning (ML) solutions are increasingly entering healthcare. They are complex, sociotechnical systems that include data inputs, ML models, technical infrastructure and human interactions. They have promise for improving care across a wide range of clinical applications but if poorly implemented, they may disrupt clinical workflows, exacerbate inequities in care and harm patients. Many aspects of ML solutions are similar to other digital technologies, which have well-established approaches to implementation. However, ML applications present distinct implementation challenges, given that their predictions are often complex and difficult to understand, they can be influenced by biases in the data sets used to develop them, and their impacts on human behaviour are poorly understood. This manuscript summarises the current state of knowledge about implementing ML solutions in clinical care and offers practical guidance for implementation. We propose three overarching questions for potential users to consider when deploying ML solutions in clinical care: (1) Is a clinical or operational problem likely to be addressed by an ML solution? (2) How can an ML solution be evaluated to determine its readiness for deployment? (3) How can an ML solution be deployed and maintained optimally? The Quality Improvement community has an essential role to play in ensuring that ML solutions are translated into clinical practice safely, effectively, and ethically.

A Nationwide Network of Health AI Assurance Laboratories

Importance

Given the importance of rigorous development and evaluation standards needed of artificial intelligence (AI) models used in health care, nationwide accepted procedures to provide assurance that the use of AI is fair, appropriate, valid, effective, and safe are urgently needed.

Observations

While there are several efforts to develop standards and best practices to evaluate AI, there is a gap between having such guidance and the application of such guidance to both existing and new AI models being developed. As of now, there is no publicly available, nationwide mechanism that enables objective evaluation and ongoing assessment of the consequences of using health AI models in clinical care settings.

Conclusion and Relevance

The need to create a public-private partnership to support a nationwide health AI assurance labs network is outlined here. In this network, community best practices could be applied for testing health AI models to produce reports on their performance that can be widely shared for managing the lifecycle of AI models over time and across populations and sites where these models are deployed.

Delay Between Actual Occurrence of Patient Vital Sign and the Nominal Appearance in the Electronic Health Record: Single-Center, Retrospective Study of PICU Data, 2014-2018

OBJECTIVES

Patient vital sign data charted in the electronic health record (EHR) are used for time-sensitive decisions, yet little is known about when these data become nominally available compared with when the vital sign was actually measured. The objective of this study was to determine the magnitude of any delay between when a vital sign was actually measured in a patient and when it nominally appears in the EHR.

DESIGN

We performed a single-center retrospective cohort study.

SETTING

Tertiary academic children's hospital.

PATIENTS

A total of 5,458 patients were admitted to a PICU from January 2014 to December 2018.

INTERVENTIONS

None.

MEASUREMENTS AND MAIN RESULTS

We analyzed entry and display times of all vital signs entered in the EHR. The primary outcome measurement was time between vital sign occurrence and nominal timing of the vital sign in the EHR. An additional outcome measurement was the frequency of batch charting. A total of 9,818,901 vital sign recordings occurred during the study period. Across the entire cohort the median (interquartile range [IQR]) difference between time of occurrence and nominal time in the EHR was in hours:minutes:seconds, 00:41:58 (IQR 00:13:42-01:44:10). Lag in the first 24 hours of PICU admission was 00:47:34 (IQR 00:15:23-02:19:00), lag in the last 24 hours was 00:38:49 (IQR 00:13:09-01:29:22; p < 0.001). There were 1,892,143 occurrences of batch charting.

CONCLUSIONS

This retrospective study shows a lag between vital sign occurrence and its appearance in the EHR, as well as a frequent practice of batch charting. The magnitude of the delay-median ~40 minutes-suggests that vital signs available in the EHR for clinical review and incorporation into clinical alerts may be outdated by the time they are available.

National Multi-Institutional Validation of a Surgical Transfusion Risk Prediction Model

BACKGROUND

Accurate estimation of surgical transfusion risk is important for many aspects of surgical planning, yet few methods for estimating are available for estimating such risk. There is a need for reliable validated methods for transfusion risk stratification to support effective perioperative planning and resource stewardship.

STUDY DESIGN

This study was conducted using the American College of Surgeons NSQIP datafile from 2019. S-PATH performance was evaluated at each contributing hospital, with and without hospital-specific model tuning. Linear regression was used to assess the relationship between hospital characteristics and area under the receiver operating characteristic (AUROC) curve.

RESULTS

A total of 1,000,927 surgical cases from 414 hospitals were evaluated. Aggregate AUROC was 0.910 (95% CI 0.904 to 0.916) without model tuning and 0.925 (95% CI 0.919 to 0.931) with model tuning. AUROC varied across individual hospitals (median 0.900, interquartile range 0.849 to 0.944), but no statistically significant relationships were found between hospital-level characteristics studied and model AUROC.

CONCLUSIONS

S-PATH demonstrated excellent discriminative performance, although there was variation across hospitals that was not well-explained by hospital-level characteristics. These results highlight the S-PATH's viability as a generalizable surgical transfusion risk prediction tool.

Revolutionizing Patient Monitoring in Age-Related Macular Degeneration: A Comparative Study on the Necessity and Efficiency of the AMD VIEWER

(1) Background: Age-related Macular Degeneration (AMD) is a critical condition leading to blindness, necessitating lifelong clinic visits for management, albeit with existing challenges in monitoring its long-term progression. This study introduced and assessed an innovative tool, the AMD long-term Information Viewer (AMD VIEWER), designed to offer a comprehensive display of crucial medical data-including visual acuity, central retinal thickness, macular volume, vitreous injection treatment history, and Optical Coherent Tomography (OCT) images-across an individual eye's entire treatment course. (2) Methods: By analyzing visit frequencies of patients with a history of invasive AMD treatment, a comparative examination between a Dropout group and an Active group underscored the clinical importance of regular visits, particularly highlighting better treatment outcomes and maintained visual acuity in the Active group. (3) Results: The efficiency of AMD VIEWER was proven by comparing it to manual data input by optometrists, showing significantly faster data display with no errors, unlike the time-consuming and error-prone manual entries. Furthermore, an elicited Net Promoter Score (NPS) of 70 from 10 ophthalmologists strongly endorsed AMD VIEWER's practical utility. (4) Conclusions: This study underscores the importance of regular clinic visits for AMD patients. It suggests the AMD VIEWER as an effective tool for improving treatment data management and display.

Multimodal Data Integration Advances Longitudinal Prediction of the Naturalistic Course of Depression and Reveals a Multimodal Signature of Remission During 2-Year Follow-up

BACKGROUND

The ability to predict the disease course of individuals with major depressive disorder (MDD) is essential for optimal treatment planning. Here, we used a data-driven machine learning approach to assess the predictive value of different sets of biological data (whole-blood proteomics, lipid metabolomics, transcriptomics, genetics), both separately and added to clinical baseline variables, for the longitudinal prediction of 2-year remission status in MDD at the individual-subject level.

METHODS

Prediction models were trained and cross-validated in a sample of 643 patients with current MDD (2-year remission n = 325) and subsequently tested for performance in 161 individuals with MDD (2-year remission n = 82).

RESULTS

Proteomics data showed the best unimodal data predictions (area under the receiver operating characteristic curve = 0.68). Adding proteomic to clinical data at baseline significantly improved 2-year MDD remission predictions (area under the receiver operating characteristic curve = 0.63 vs. 0.78, p = .013), while the addition of other omics data to clinical data did not yield significantly improved model performance. Feature importance and enrichment analysis revealed that proteomic analytes were involved in inflammatory response and lipid metabolism, with fibrinogen levels showing the highest variable importance, followed by symptom severity. Machine learning models outperformed psychiatrists' ability to predict 2-year remission status (balanced accuracy = 71% vs. 55%).

CONCLUSIONS

This study showed the added predictive value of combining proteomic data, but not other omics data, with clinical data for the prediction of 2-year remission status in MDD. Our results reveal a novel multimodal signature of 2-year MDD remission status that shows clinical potential for individual MDD disease course predictions from baseline measurements.