1
|
Levin MA, Kia A, Timsina P, Cheng FY, Nguyen KAN, Kohli-Seth R, Lin HM, Ouyang Y, Freeman R, Reich DL. Real-Time Machine Learning Alerts to Prevent Escalation of Care: A Nonrandomized Clustered Pragmatic Clinical Trial. Crit Care Med 2024; 52:1007-1020. [PMID: 38380992 DOI: 10.1097/ccm.0000000000006243] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/22/2024]
Abstract
OBJECTIVES Machine learning algorithms can outperform older methods in predicting clinical deterioration, but rigorous prospective data on their real-world efficacy are limited. We hypothesized that real-time machine learning generated alerts sent directly to front-line providers would reduce escalations. DESIGN Single-center prospective pragmatic nonrandomized clustered clinical trial. SETTING Academic tertiary care medical center. PATIENTS Adult patients admitted to four medical-surgical units. Assignment to intervention or control arms was determined by initial unit admission. INTERVENTIONS Real-time alerts stratified according to predicted likelihood of deterioration sent either to the primary team or directly to the rapid response team (RRT). Clinical care and interventions were at the providers' discretion. For the control units, alerts were generated but not sent, and standard RRT activation criteria were used. MEASUREMENTS AND MAIN RESULTS The primary outcome was the rate of escalation per 1000 patient bed days. Secondary outcomes included the frequency of orders for fluids, medications, and diagnostic tests, and combined in-hospital and 30-day mortality. Propensity score modeling with stabilized inverse probability of treatment weight (IPTW) was used to account for differences between groups. Data from 2740 patients enrolled between July 2019 and March 2020 were analyzed (1488 intervention, 1252 control). Average age was 66.3 years and 1428 participants (52%) were female. The rate of escalation was 12.3 vs. 11.3 per 1000 patient bed days (difference, 1.0; 95% CI, -2.8 to 4.7) and IPTW adjusted incidence rate ratio 1.43 (95% CI, 1.16-1.78; p < 0.001). Patients in the intervention group were more likely to receive cardiovascular medication orders (16.1% vs. 11.3%; 4.7%; 95% CI, 2.1-7.4%) and IPTW adjusted relative risk (RR) (1.74; 95% CI, 1.39-2.18; p < 0.001). Combined in-hospital and 30-day-mortality was lower in the intervention group (7% vs. 9.3%; -2.4%; 95% CI, -4.5% to -0.2%) and IPTW adjusted RR (0.76; 95% CI, 0.58-0.99; p = 0.045). CONCLUSIONS Real-time machine learning alerts do not reduce the rate of escalation but may reduce mortality.
Collapse
Affiliation(s)
- Matthew A Levin
- Department of Anesthesiology, Perioperative, and Pain Medicine, Icahn School of Medicine at Mount Sinai, New York, NY
- Department of Genetics and Genomic Sciences, Icahn School of Medicine at Mount Sinai, New York, NY
- Windreich Department of Artificial Intelligence and Human Health, Icahn School of Medicine at Mount Sinai, New York, NY
- Institute for Health Care Delivery Science, Icahn School of Medicine at Mount Sinai, New York, NY
- Institute for Critical Care Medicine, Icahn School of Medicine at Mount Sinai, New York, NY
- Department of Anesthesiology and Yale Center for Analytical Sciences, Yale School of Medicine, New Haven, CT
| | - Arash Kia
- Department of Anesthesiology, Perioperative, and Pain Medicine, Icahn School of Medicine at Mount Sinai, New York, NY
- Institute for Health Care Delivery Science, Icahn School of Medicine at Mount Sinai, New York, NY
| | - Prem Timsina
- Institute for Health Care Delivery Science, Icahn School of Medicine at Mount Sinai, New York, NY
| | - Fu-Yuan Cheng
- Institute for Health Care Delivery Science, Icahn School of Medicine at Mount Sinai, New York, NY
| | - Kim-Anh-Nhi Nguyen
- Institute for Health Care Delivery Science, Icahn School of Medicine at Mount Sinai, New York, NY
| | - Roopa Kohli-Seth
- Institute for Critical Care Medicine, Icahn School of Medicine at Mount Sinai, New York, NY
| | - Hung-Mo Lin
- Department of Anesthesiology and Yale Center for Analytical Sciences, Yale School of Medicine, New Haven, CT
| | - Yuxia Ouyang
- Department of Anesthesiology, Perioperative, and Pain Medicine, Icahn School of Medicine at Mount Sinai, New York, NY
| | - Robert Freeman
- Institute for Health Care Delivery Science, Icahn School of Medicine at Mount Sinai, New York, NY
| | - David L Reich
- Department of Anesthesiology, Perioperative, and Pain Medicine, Icahn School of Medicine at Mount Sinai, New York, NY
| |
Collapse
|
2
|
Greenberg JK, Otun A, Ghogawala Z, Yen PY, Molina CA, Limbrick DD, Foraker RE, Kelly MP, Ray WZ. Translating Data Analytics Into Improved Spine Surgery Outcomes: A Roadmap for Biomedical Informatics Research in 2021. Global Spine J 2022; 12:952-963. [PMID: 33973491 PMCID: PMC9344511 DOI: 10.1177/21925682211008424] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 01/29/2023] Open
Abstract
STUDY DESIGN Narrative review. OBJECTIVES There is growing interest in the use of biomedical informatics and data analytics tools in spine surgery. Yet despite the rapid growth in research on these topics, few analytic tools have been implemented in routine spine practice. The purpose of this review is to provide a health information technology (HIT) roadmap to help translate data assets and analytics tools into measurable advances in spine surgical care. METHODS We conducted a narrative review of PubMed and Google Scholar to identify publications discussing data assets, analytical approaches, and implementation strategies relevant to spine surgery practice. RESULTS A variety of data assets are available for spine research, ranging from commonly used datasets, such as administrative billing data, to emerging resources, such as mobile health and biobanks. Both regression and machine learning techniques are valuable for analyzing these assets, and researchers should recognize the particular strengths and weaknesses of each approach. Few studies have focused on the implementation of HIT, and a variety of methods exist to help translate analytic tools into clinically useful interventions. Finally, a number of HIT-related challenges must be recognized and addressed, including stakeholder acceptance, regulatory oversight, and ethical considerations. CONCLUSIONS Biomedical informatics has the potential to support the development of new HIT that can improve spine surgery quality and outcomes. By understanding the development life-cycle that includes identifying an appropriate data asset, selecting an analytic approach, and leveraging an effective implementation strategy, spine researchers can translate this potential into measurable advances in patient care.
Collapse
Affiliation(s)
- Jacob K. Greenberg
- Department of Neurological Surgery, Washington University School of Medicine,
St. Louis, MO, USA,Jacob K. Greenberg, Department of
Neurosurgery, Washington University School of Medicine, 660S. Euclid Ave., Box
8057, St. Louis, MO 63 110, USA.
| | - Ayodamola Otun
- Department of Neurological Surgery, Washington University School of Medicine,
St. Louis, MO, USA
| | - Zoher Ghogawala
- Department of Neurosurgery, Lahey Hospital and Medical Center, Burlington, MA, USA
| | - Po-Yin Yen
- Institute for Informatics, Washington University School of Medicine,
St. Louis, MO, USA
| | - Camilo A. Molina
- Department of Neurological Surgery, Washington University School of Medicine,
St. Louis, MO, USA
| | - David D. Limbrick
- Department of Neurological Surgery, Washington University School of Medicine,
St. Louis, MO, USA
| | - Randi E Foraker
- Institute for Informatics, Washington University School of Medicine,
St. Louis, MO, USA
| | - Michael P. Kelly
- Department of Orthopaedic Surgery, Washington University School of Medicine,
St. Louis, MO, USA
| | - Wilson Z. Ray
- Department of Neurological Surgery, Washington University School of Medicine,
St. Louis, MO, USA
| |
Collapse
|
3
|
Nielsen PB, Langkjær CS, Schultz M, Kodal AM, Pedersen NE, Petersen JA, Lange T, Arvig MD, Meyhoff CS, Bestle MH, Hølge-Hazelton B, Bunkenborg G, Lippert A, Andersen O, Rasmussen LS, Iversen KK. Clinical assessment as a part of an early warning score—a Danish cluster-randomised, multicentre study of an individual early warning score. THE LANCET DIGITAL HEALTH 2022; 4:e497-e506. [DOI: 10.1016/s2589-7500(22)00067-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/04/2021] [Revised: 03/02/2022] [Accepted: 03/29/2022] [Indexed: 10/18/2022]
|
4
|
Langkjaer CS, Bove DG, Nielsen PB, Iversen KK, Bestle MH, Bunkenborg G. Nurses' Experiences and Perceptions of two Early Warning Score systems to Identify Patient Deterioration-A Focus Group Study. Nurs Open 2021; 8:1788-1796. [PMID: 33638617 PMCID: PMC8186715 DOI: 10.1002/nop2.821] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/03/2020] [Revised: 12/29/2020] [Accepted: 01/31/2021] [Indexed: 11/07/2022] Open
Abstract
AIMS To explore Registered Nurses' experiences and perceptions with National Early Warning Score and Individual Early Warning Score to identify patient deterioration. DESIGN A qualitative exploratory design. METHODS Six focus groups were conducted at six Danish hospitals from February to June 2019. Registered Nurses from both medical, surgical and emergency departments participated. The focus groups were analysed using content analysis. RESULTS One theme and four categories were identified. Theme: Meaningful in identifying patient deterioration but causing frustration due to lack of flexibility. Categories: (a) Inter-professional collaboration strengthened through the use of Early Warning Score systems, (b) Enhanced professional development and communication among nurses when using Early Warning Score systems, (c) Detecting patient deterioration by integrating nurses' clinical gaze with Early Warning Score systems and (d) Modification and fear of making mistakes when using Early Warning Score systems.
Collapse
Affiliation(s)
- Caroline S. Langkjaer
- Department of Emergency MedicineNordsjaellands HospitalUniversity of CopenhagenHilleroedDenmark
| | - Dorthe G. Bove
- Department of Emergency MedicineNordsjaellands HospitalUniversity of CopenhagenHilleroedDenmark
| | - Pernille B. Nielsen
- Department of CardiologyHerlev and Gentofte HospitalUniversity of CopenhagenHerlevDenmark
- Department of Emergency MedicineHerlev and Gentofte HospitalUniversity of CopenhagenHerlevDenmark
| | - Kasper K. Iversen
- Department of CardiologyHerlev and Gentofte HospitalUniversity of CopenhagenHerlevDenmark
- Department of Emergency MedicineHerlev and Gentofte HospitalUniversity of CopenhagenHerlevDenmark
- Department of Clinical MedicineUniversity of CopenhagenCopenhagenDenmark
| | - Morten H. Bestle
- Department of Clinical MedicineUniversity of CopenhagenCopenhagenDenmark
- Department of Anaesthesiology and Intensive careNordsjaellands HospitalUniversity of CopenhagenHilleroedDenmark
| | - Gitte Bunkenborg
- Department of AnesthesiologyHolbaek HospitalHolbaekDenmark
- Department of Regional Health ResearchUniversity of Southern DenmarkOdenseDenmark
| |
Collapse
|
5
|
Naito T, Hayashi K, Hsu HC, Aoki K, Nagata K, Arai M, Nakada TA, Suzaki S, Hayashi Y, Fujitani S. Validation of National Early Warning Score for predicting 30-day mortality after rapid response system activation in Japan. Acute Med Surg 2021; 8:e666. [PMID: 34026233 PMCID: PMC8122242 DOI: 10.1002/ams2.666] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/31/2021] [Revised: 03/27/2021] [Accepted: 04/22/2021] [Indexed: 11/24/2022] Open
Abstract
Aim Although rapid response systems (RRS) are used to prevent adverse events, Japan reportedly has low activation rates and high mortality rates. The National Early Warning Score (NEWS) could provide a solution, but it has not been validated in Japan. We aimed to validate NEWS for Japanese patients. Methods This retrospective observational study included data of 2,255 adult patients from 33 facilities registered in the In‐Hospital Emergency Registry in Japan between January 2014 and March 2018. The primary evaluated outcome was mortality rate 30 days after RRS activation. Accuracy of NEWS was analyzed with the correlation coefficient and area under the receiver operating characteristic curve. Prediction weights of NEWS parameters were then analyzed using multiple logistic regression and a machine learning method, classification and regression trees. Results The correlation coefficient of NEWS for 30‐day mortality rate was 0.95 (95% confidence interval [CI], 0.88–0.98) and the area under the receiver operating characteristic curve was 0.668 (95% CI, 0.642–0.693). Sensitivity and specificity values with a cut‐off score of 7 were 89.8% and 45.1%, respectively. Regarding prediction values of each parameter, oxygen saturation showed the highest odds ratio of 1.36 (95% CI, 1.25–1.48), followed by altered mental status 1.23 (95% CI, 1.14–1.32), heart rate 1.21 (95% CI, 1.09–1.34), systolic blood pressure 1.12 (95% CI, 1.04–1.22), and respiratory rate 1.03 (95% CI, 1.05–1.26). Body temperature and oxygen supplementation were not significantly associated. Classification and regression trees showed oxygen saturation as the most heavily weighted parameter, followed by altered mental status and respiratory rate. Conclusions National Early Warning Score could stratify 30‐day mortality risk following RRS activation in Japanese patients.
Collapse
Affiliation(s)
- Takaki Naito
- Department of Emergency and Critical Care Medicine St. Marianna University School of Medicine Kanagawa Japan
| | - Kuniyoshi Hayashi
- Graduate School of Public Health St. Luke's International University Tokyo Japan
| | - Hsiang-Chin Hsu
- Department of Emergency Medicine National Cheng Kung University Tainan City Taiwan
| | - Kazuhiro Aoki
- Department of Anesthesiology and Intensive Care Medicine St. Luke's International Hospital Tokyo Japan
| | - Kazuma Nagata
- Department of Respiratory Medicine Kobe City Medical Center General Hospital Hyogo Japan
| | - Masayasu Arai
- Department of Anesthesiology Kitasato University School of Medicine Kanagawa Japan
| | - Taka-Aki Nakada
- Department of Emergency and Critical Care Medicine Chiba University Graduate School of Medicine Chiba Japan
| | - Shinichiro Suzaki
- Department of Emergency and Critical Care Medicine Japanese Red Cross Musashino Hospital Tokyo Japan
| | - Yoshiro Hayashi
- Department of Intensive Care Medicine Kameda Medical Center Chiba Japan
| | - Shigeki Fujitani
- Department of Emergency and Critical Care Medicine St. Marianna University School of Medicine Kanagawa Japan
| | | |
Collapse
|
6
|
Peng L, Luo Z, Liang L, Liu M, Meng L, Tan J, Song L, Zhang Y, Wu L. Comparison of the Performance of 24 Early Warning Scores with the Updated National Early Warning Score (NEWS2) for Predicting Unplanned Intensive Care Unit (ICU) Admission in Postoperative Brain Tumor Patients: A Retrospective Study at a Single Center. Med Sci Monit 2021; 27:e929168. [PMID: 33649288 PMCID: PMC7938866 DOI: 10.12659/msm.929168] [Citation(s) in RCA: 6] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/23/2022] Open
Abstract
Background There have been few studies to evaluate early warning score (EWS) systems, or track and trigger systems (TTS), to identify early clinical deterioration in patients following brain tumor surgery who are admitted to the Intensive Care Unit (ICU). The National Early Warning Score (NEWS2) is an established method used in the U.K. National Health Service to improve care for in-hospital patients. This retrospective study from a single center aimed to compare the performance of NEWS2 with 24 other types of EWS to evaluate unplanned ICU admissions within 72 h after brain tumor surgery. Material/Methods A total of 326 patients with brain tumors were included in the study. Patients who experienced unplanned ICU transfer after surgery (69 cases) were diagnostically matched with patients who did not require intensive care (257 controls). We collected the physiological variables to calculate the area under the receiver operator characteristic curve (AUROC), sensitivity, specificity, Youden index values, cutoff values, positive predictive values, and negative predictive values. Results The NEWS2 identified postoperative brain tumor patients with AUROC (0.860, p=0.000). The Patient-At-Risk (PAR) score was higher than NEWS2 in terms of AUROC value (0.870, P=0.000), Youden index (0.589 vs 0.542). Conclusions The findings showed that although the NEWS 2 performed well when used to evaluate unplanned ICU admissions within 72 h of postoperative brain tumor patients, the PAR score was also an accurate EWS.
Collapse
Affiliation(s)
- Lingli Peng
- Xiangya School of Public Health, Central South University, Changsha, Hunan, China (mainland).,Orthopedics Department, Xiangya Hospital, Central South University, Changsha, Hunan, China (mainland)
| | - Zhen Luo
- Xiangya Nursing School, Central South University, Changsha, Hunan, China (mainland)
| | - Lingling Liang
- Orthopedics Department, Sixth Affiliated Hospital of Xinjiang Medical University, Urumqi, Xinjiang, China (mainland)
| | - Mingming Liu
- Xiangya Nursing School, Central South University, Changsha, Hunan, China (mainland)
| | - Lingyao Meng
- Xiangya Nursing School, Central South University, Changsha, Hunan, China (mainland)
| | - Jianwen Tan
- Department of Organ Transplantation, Xiangya Hospital, Central South University, Changsha, Hunan, China (mainland)
| | - Lili Song
- Xiangya Nursing School, Central South University, Changsha, Hunan, China (mainland)
| | - Yan Zhang
- Xiangya Nursing School, Central South University, Changsha, Hunan, China (mainland)
| | - Lixiang Wu
- Xiangya School of Public Health, Central South University, Changsha, Hunan, China (mainland)
| |
Collapse
|