1
|
Shung DL, Chan CE, You K, Nakamura S, Saarinen T, Zheng NS, Simonov M, Li DK, Tsay C, Kawamura Y, Shen M, Hsiao A, Sekhon JS, Laine L. Validation of an Electronic Health Record-Based Machine Learning Model Compared With Clinical Risk Scores for Gastrointestinal Bleeding. Gastroenterology 2024; 167:1198-1212. [PMID: 38971198 PMCID: PMC11493512 DOI: 10.1053/j.gastro.2024.06.030] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 02/13/2024] [Revised: 06/25/2024] [Accepted: 06/26/2024] [Indexed: 07/08/2024]
Abstract
BACKGROUND & AIMS Guidelines recommend use of risk stratification scores for patients presenting with gastrointestinal bleeding (GIB) to identify very-low-risk patients eligible for discharge from emergency departments. Machine learning models may outperform existing scores and can be integrated within the electronic health record (EHR) to provide real-time risk assessment without manual data entry. We present the first EHR-based machine learning model for GIB. METHODS The training cohort comprised 2546 patients and internal validation of 850 patients presenting with overt GIB (ie, hematemesis, melena, and hematochezia) to emergency departments of 2 hospitals from 2014 to 2019. External validation was performed on 926 patients presenting to a different hospital with the same EHR from 2014 to 2019. The primary outcome was a composite of red blood cell transfusion, hemostatic intervention (ie, endoscopic, interventional radiologic, or surgical), and 30-day all-cause mortality. We used structured data fields in the EHR, available within 4 hours of presentation, and compared the performance of machine learning models with current guideline-recommended risk scores, Glasgow-Blatchford Score, and Oakland Score. Primary analysis was area under the receiver operating characteristic curve. Secondary analysis was specificity at 99% sensitivity to assess the proportion of patients correctly identified as very low risk. RESULTS The machine learning model outperformed the Glasgow-Blatchford Score (area under the receiver operating characteristic curve, 0.92 vs 0.89; P < .001) and Oakland Score (area under the receiver operating characteristic curve, 0.92 vs 0.89; P < .001). At the very-low-risk threshold of 99% sensitivity, the machine learning model identified more very-low-risk patients: 37.9% vs 18.5% for Glasgow-Blatchford Score and 11.7% for Oakland Score (P < .001 for both comparisons). CONCLUSIONS An EHR-based machine learning model performs better than currently recommended clinical risk scores and identifies more very-low-risk patients eligible for discharge from the emergency department.
Collapse
Affiliation(s)
- Dennis L Shung
- Section of Digestive Diseases, Department of Medicine, Yale School of Medicine, New Haven, Connecticut; Clinical and Translational Research Accelerator, Department of Medicine, Yale School of Medicine, New Haven, Connecticut; Department of Biomedical Informatics and Data Science, Department of Medicine, Yale School of Medicine, New Haven, Connecticut.
| | - Colleen E Chan
- Department of Statistics and Data Science, Yale University, New Haven, Connecticut
| | - Kisung You
- Department of Mathematics, City University of New York, Baruch College, New York, New York
| | - Shinpei Nakamura
- Department of Statistics and Data Science, Yale University, New Haven, Connecticut
| | - Theo Saarinen
- Department of Statistics, University of Berkeley, Berkeley, California
| | - Neil S Zheng
- Section of Digestive Diseases, Department of Medicine, Yale School of Medicine, New Haven, Connecticut
| | | | - Darrick K Li
- Section of Digestive Diseases, Department of Medicine, Yale School of Medicine, New Haven, Connecticut
| | - Cynthia Tsay
- Department of Gastroenterology, Johns Hopkins School of Medicine, Baltimore, Maryland
| | - Yuki Kawamura
- University of Cambridge School of Clinical Medicine, Cambridge, United Kingdom
| | - Matthew Shen
- Department of Statistics, University of Berkeley, Berkeley, California
| | - Allen Hsiao
- Section of Digestive Diseases, Department of Medicine, Yale School of Medicine, New Haven, Connecticut
| | - Jasjeet S Sekhon
- Clinical and Translational Research Accelerator, Department of Medicine, Yale School of Medicine, New Haven, Connecticut; Department of Political Science, Yale University, New Haven, Connecticut
| | - Loren Laine
- Section of Digestive Diseases, Department of Medicine, Yale School of Medicine, New Haven, Connecticut; West Haven Veterans Affairs Medical Center, West Haven, Connecticut
| |
Collapse
|
2
|
Ridha B, Hey N, Ritchie L, Toews R, Turcotte Z, Jamison B. Predicting the need for urgent endoscopic intervention in lower gastrointestinal bleeding: a retrospective review. BMC Emerg Med 2024; 24:71. [PMID: 38654175 DOI: 10.1186/s12873-024-00990-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/27/2023] [Accepted: 04/18/2024] [Indexed: 04/25/2024] Open
Abstract
BACKGROUND Lower gastrointestinal bleeding (LGIB) is a common reason for emergency department visits and subsequent hospitalizations. Recent data suggests that low-risk patients may be safely evaluated as an outpatient. Recommendations for healthcare systems to identify low-risk patients who can be safely discharged with timely outpatient follow-up have yet to be established. The primary objective of this study was to determine the role of patient predictors for the patients with LGIB to receive urgent endoscopic intervention. METHODS A retrospective chart review was performed on 142 patients. Data was collected on patient demographics, clinical features, comorbidities, medications, hemodynamic parameters, laboratory values, and diagnostic imaging. Logistic regression analysis, independent samples t-testing, Mann Whitney U testing for non-parametric data, and univariate analysis of categorical variables by Chi square test was performed to determine relationships within the data. RESULTS On logistic regression analysis, A hemoglobin drop of > 20 g/L was the only variable that predicted endoscopic intervention (p = 0.030). Tachycardia, hypotension, or presence of anticoagulation were not significantly associated with endoscopic intervention (p > 0.05). CONCLUSIONS A hemoglobin drop of > 20 g/L was the only patient parameter that predicted the need for urgent endoscopic intervention in the emergency department.
Collapse
Affiliation(s)
- Barzany Ridha
- Department of Emergency Medicine, College of Medicine, University of Saskatchewan, Royal University Hospital, 104 Hospital Drive, Saskatoon, SK, S7N 0W8, Canada.
| | - Nigel Hey
- Department of Emergency Medicine, College of Medicine, University of Saskatchewan, Royal University Hospital, 104 Hospital Drive, Saskatoon, SK, S7N 0W8, Canada
| | - Lauren Ritchie
- Department of Emergency Medicine, College of Medicine, University of Saskatchewan, Royal University Hospital, 104 Hospital Drive, Saskatoon, SK, S7N 0W8, Canada
| | - Ryan Toews
- Department of Emergency Medicine, College of Medicine, University of Saskatchewan, Royal University Hospital, 104 Hospital Drive, Saskatoon, SK, S7N 0W8, Canada
| | - Zachary Turcotte
- Department of Emergency Medicine, College of Medicine, University of Saskatchewan, Royal University Hospital, 104 Hospital Drive, Saskatoon, SK, S7N 0W8, Canada
| | - Brad Jamison
- Department of Emergency Medicine, College of Medicine, University of Saskatchewan, Royal University Hospital, 104 Hospital Drive, Saskatoon, SK, S7N 0W8, Canada
| |
Collapse
|
3
|
Orpen-Palmer J, Stanley AJ. A Review of Risk Scores within Upper Gastrointestinal Bleeding. J Clin Med 2023; 12:jcm12113678. [PMID: 37297873 DOI: 10.3390/jcm12113678] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/19/2023] [Revised: 05/19/2023] [Accepted: 05/23/2023] [Indexed: 06/12/2023] Open
Abstract
Upper gastrointestinal bleeding is a common medical emergency. Thorough initial assessment and appropriate resuscitation are essential to stabilise the patient. Risk scores provide an important tool to discriminate between lower- and higher-risk patients. Very low-risk patients can be safely discharged for out-patient management, while higher-risk patients can receive appropriate in-patient care. The Glasgow Blatchford Score, with a score of 0-1, performs best in the identification of very low-risk patients who will not require hospital based intervention or die, and is recommended by most guidelines to facilitate safe out-patient management. The performance of risk scores in the identification of specific adverse events to define high-risk patients is less accurate, with no individual score performing consistently well. Ongoing developments in the use of machine learning models and artificial intelligence in predicting poor outcomes in UGIB appear promising and will likely form the basis of dynamic risk assessment in the future.
Collapse
Affiliation(s)
- Josh Orpen-Palmer
- Department of Gastroenterology, Glasgow Royal Infirmary, Glasgow G4 0SF, UK
| | - Adrian J Stanley
- Department of Gastroenterology, Glasgow Royal Infirmary, Glasgow G4 0SF, UK
| |
Collapse
|
4
|
Marmo R, Soncini M, Bucci C, Occhipinti V, Pellegrini L, Zullo A. Derivation and validation of Re.Co.De death score risk in patients with acute nonvariceal upper GI bleeding. Gastrointest Endosc 2022; 96:36-43.e8. [PMID: 35150665 DOI: 10.1016/j.gie.2022.01.024] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 09/10/2021] [Accepted: 01/31/2022] [Indexed: 02/08/2023]
Abstract
BACKGROUND AND AIMS Scores in upper GI bleeding (UGIB) are used to stratify death risk and need for hospitalization at admission, but a tool that incorporates dynamic changes during the hospital stay is lacking. We aimed to develop a death risk score that considers changes in clinical status during hospitalization and compare its performance with existing ones. METHODS A multicenter cohort study enrolling patients with UGIB in 50 Italian hospitals from January 2014 to December 2015 was conducted. Data were collected and used to develop a risk score using logistic regression analyses. Performance curves (area under the receiver-operating characteristic [AUROC] curves), sensitivities, specificities, positive and negative predictive values, and outcomes classified as low, intermediate, and high death risk were calculated. The score's performance was externally validated and then compared with other scores. RESULTS We included 1852 patients with nonvariceal UGIB in the development cohort and 912 in the validation cohorts. The new score, which we named the Re.Co.De (rebleeding-comorbidities-deteriorating) score, included 10 variables depicting the changes in clinical conditions while in the hospital. The mortality AUROC curves were .93 (95% confidence interval, .91-.96) in the derivation cohort and .94 (95% confidence interval, .91-.98) in validation cohort. In a comparison of AUROC curves with other scores, the new score showed a significant performance compared with pre- and postendoscopy scores. Patients with low and high scores had 30-day mortality rates of .001% and 48.2%, respectively. CONCLUSIONS The Re.Co.De score has a higher performance for predicting mortality in patients with UGIB compared with other scores, correctly identifying patients at low and high death risk while in the hospital through a dynamic re-evaluation of clinical status.
Collapse
Affiliation(s)
- Riccardo Marmo
- Gastroenterology and Endoscopy Unit, "L. Curto" Hospital, Salerno, Italy
| | - Marco Soncini
- Department of Internal Medicine, "A. Manzoni" Hospital, Lecco, Italy
| | - Cristina Bucci
- Endoscopy Unit, AORN Santobono-Pausillipon Napoli, Naples, Italy
| | | | | | - Angelo Zullo
- Gastroenterology Unit, "Nuovo Regina Margherita" Hospital, Rome, Italy
| | | |
Collapse
|
5
|
Marmo R, Soncini M, Bucci C, Zullo A. Comparison of assessment tools in acute upper gastrointestinal bleeding: which one for which decision. Scand J Gastroenterol 2022; 57:1-7. [PMID: 34534036 DOI: 10.1080/00365521.2021.1976268] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 02/04/2023]
Abstract
BACKGROUND Upper GI bleeding (UGIB) remains a common emergency with significant mortality. Scores help triage patients, but it is still unclear which score should be used in the different decision-making moments to identify patients at high or low death risk. We aimed to compare the overall performances of the most validated scores and their cut-off performance to identify patients at low and high death risk. The secondary outcome was to compare the scores' performance for predicting therapeutic endoscopy, the need for transfusion(s), rebleeding, and surgery/interventional radiology. METHODS We conducted a prospective multicenter cohort study, including consecutive UGIB patients admitted to 50 Italian hospitals. We collected information to calculate the Rockall, the Progetto Nazionale Endoscopia Digestiva (PNED), the AIMS65, the Glasgow-Blatchford (GBS), and the Age, Blood tests, Comorbidities (ABC) scores, together with demographic figures, clinical data, and outcomes. RESULTS We obtained complete data of 2307 outpatients, including 1887 non-variceal and 420 variceal bleeders. Our cohort's mean age was 67.5 years, with a prevalence of male gender (69%). The GBS has the best overall performance (ROC 0.74) compared to the other scores in identifying low-risk patients (p < .001). At the cut-off 0-1, both GBS and ABC scores provide the highest PPV (100%) for low-risk patients. ABC and PNED scores are the most useful ones (for AUC >80) to assess the high-risk patients for mortality. CONCLUSIONS At admission, GBS and ABC scores identify low-risk patients suitable for outpatient management, while PNED and ABC scores identify high-risk patients. During hospitalization, the PNED score should be used to re-assess the mortality risk if a modification of clinical status occurs.
Collapse
Affiliation(s)
- Riccardo Marmo
- Gastroenterology and Endoscopy Unit, "L. Curto" Hospital, Polla, Italy
| | - Marco Soncini
- Department of Internal Medicine, "A. Manzoni" Hospital, Lecco, Italy
| | - Cristina Bucci
- Gastroenterology and Endoscopy Unit, "L. Curto" Hospital, Polla, Italy
| | - Angelo Zullo
- Gastroenterology Unit, Nuovo Regina Margherita Hospital, Rome, Italy
| | | |
Collapse
|
6
|
Horibe M, Iwasaki E, Bazerbachi F, Kaneko T, Matsuzaki J, Minami K, Masaoka T, Hosoe N, Ogura Y, Namiki S, Hosoda Y, Ogata H, Chan AT, Kanai T. Horibe GI bleeding prediction score: a simple score for triage decision-making in patients with suspected upper GI bleeding. Gastrointest Endosc 2020; 92:578-588.e4. [PMID: 32240682 DOI: 10.1016/j.gie.2020.03.3846] [Citation(s) in RCA: 10] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 11/05/2019] [Accepted: 03/19/2020] [Indexed: 12/11/2022]
Abstract
BACKGROUND AND AIMS Although upper GI bleeding (UGIB) is a significant cause of inpatient admissions, no scoring method has proven to be accurate and simple as a standard for triage purposes. Therefore, we compared a previously described 3-variable score (1 point each for absence of daily proton pump inhibitor use in the week before the index presentation, shock index [heart rate/systolic blood pressure] ≥1, and blood urea nitrogen/creatinine ≥30 [urea/creatinine≥140]), the Horibe gAstRointestinal BleedING scoRe (HARBINGER), with the 8-variable Glasgow-Blatchford Score (GBS) and 5-variable AIMS65 to evaluate and validate the accuracy in predicting high-risk features that warrant admission and urgent endoscopy. METHODS Consecutive patients presenting with suspected UGIB between 2012 and 2015 were prospectively enrolled in 3 acute care Japanese hospitals. On presentation to the emergency setting, an endoscopy was performed in a timely fashion. The primary outcome was the prediction of high-risk endoscopic stigmata. RESULTS Of 1486 enrolled patients, 637 (43%) harbored high-risk endoscopic stigmata according to international consensus statements. The area under the receiver operating characteristic curve (AUC) for the HARBINGER was .76 (95% confidence interval [CI], .72-.79), which was significantly superior to both the GBS (AUC, .68; 95% CI, .64-.71; P < .001) and the AIMS65 (AUC, .54; 95% CI, .50-.58; P < .001). When the HARBINGER cutoff value was set at 1 to rule out patients who needed admission and urgent endoscopy, its sensitivity and specificity was 98.8% (95% CI, 97.9-99.6) and 15.5% (95% CI, 13.1-18.0), respectively. CONCLUSIONS The HARBINGER, a simple 3-variable score, provides a more accurate method for triage of patients with suspected UGIB than both the GBS and AIMS65.
Collapse
Affiliation(s)
- Masayasu Horibe
- Division of Gastroenterology and Hepatology, Department of Internal Medicine, Keio University School of Medicine, Tokyo, Japan; Division of Gastroenterology and Hepatology, Mayo Clinic, Rochester, Minnesota, USA
| | - Eisuke Iwasaki
- Division of Gastroenterology and Hepatology, Department of Internal Medicine, Keio University School of Medicine, Tokyo, Japan
| | - Fateh Bazerbachi
- Division of Gastroenterology and Hepatology, Mayo Clinic, Rochester, Minnesota, USA; Division of Gastroenterology, Department of Medicine, Massachusetts General Hospital, Boston, Massachusetts, USA
| | - Tetsuji Kaneko
- Department of Clinical Trial, Tokyo Metropolitan Children's Medical Center, Tokyo, Japan; Teikyo Academic Research Center, Teikyo University, Tokyo, Japan
| | - Juntaro Matsuzaki
- Division of Gastroenterology and Hepatology, Department of Internal Medicine, Keio University School of Medicine, Tokyo, Japan
| | - Kazuhiro Minami
- Division of Gastroenterology and Hepatology, Department of Internal Medicine, Keio University School of Medicine, Tokyo, Japan
| | - Tatsuhiro Masaoka
- Division of Gastroenterology and Hepatology, Department of Internal Medicine, Keio University School of Medicine, Tokyo, Japan
| | - Naoki Hosoe
- Center for Diagnostic and Therapeutic Endoscopy, Keio University Hospital, Tokyo, Japan
| | - Yuki Ogura
- Department of Gastroenterology and Hepatology, Tokyo Metropolitan Tama Medical Center, Tokyo, Japan
| | - Shin Namiki
- Department of Gastroenterology and Hepatology, Tokyo Metropolitan Tama Medical Center, Tokyo, Japan
| | - Yasuo Hosoda
- Division of Gastroenterology, Department of Internal Medicine, National Hospital Organization Saitama National Hospital, Saitama, Japan
| | - Haruhiko Ogata
- Center for Diagnostic and Therapeutic Endoscopy, Keio University Hospital, Tokyo, Japan
| | - Andrew T Chan
- Division of Gastroenterology, Department of Medicine, Massachusetts General Hospital, Boston, Massachusetts, USA
| | - Takanori Kanai
- Division of Gastroenterology and Hepatology, Department of Internal Medicine, Keio University School of Medicine, Tokyo, Japan
| |
Collapse
|