1
|
Liu J, Chen J, Dong Y, Lou Y, Tian Y, Sun H, Jin Y, Li J, Qiu Y. Clinical Timing-Sequence Warning Models for Serious Bacterial Infections in Adults Based on Machine Learning: Retrospective Study. J Med Internet Res 2023; 25:e45515. [PMID: 38109177 PMCID: PMC10758945 DOI: 10.2196/45515] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/04/2023] [Revised: 03/22/2023] [Accepted: 11/22/2023] [Indexed: 12/19/2023] Open
Abstract
BACKGROUND Serious bacterial infections (SBIs) are linked to unplanned hospital admissions and a high mortality rate. The early identification of SBIs is crucial in clinical practice. OBJECTIVE This study aims to establish and validate clinically applicable models designed to identify SBIs in patients with infective fever. METHODS Clinical data from 945 patients with infective fever, encompassing demographic and laboratory indicators, were retrospectively collected from a 2200-bed teaching hospital between January 2013 and December 2020. The data were randomly divided into training and test sets at a ratio of 7:3. Various machine learning (ML) algorithms, including Boruta, Lasso (least absolute shrinkage and selection operator), and recursive feature elimination, were utilized for feature filtering. The selected features were subsequently used to construct models predicting SBIs using logistic regression (LR), random forest (RF), and extreme gradient boosting (XGBoost) with 5-fold cross-validation. Performance metrics, including the receiver operating characteristic (ROC) curve and area under the ROC curve (AUC), accuracy, sensitivity, and other relevant parameters, were used to assess model performance. Considering both model performance and clinical needs, 2 clinical timing-sequence warning models were ultimately confirmed using LR analysis. The corresponding predictive nomograms were then plotted for clinical use. Moreover, a physician, blinded to the study, collected additional data from the same center involving 164 patients during 2021. The nomograms developed in the study were then applied in clinical practice to further validate their clinical utility. RESULTS In total, 69.9% (661/945) of the patients developed SBIs. Age, hemoglobin, neutrophil-to-lymphocyte ratio, fibrinogen, and C-reactive protein levels were identified as important features by at least two ML algorithms. Considering the collection sequence of these indicators and clinical demands, 2 timing-sequence models predicting the SBI risk were constructed accordingly: the early admission model (model 1) and the model within 24 hours of admission (model 2). LR demonstrated better stability than RF and XGBoost in both models and performed the best in model 2, with an AUC, accuracy, and sensitivity of 0.780 (95% CI 0.720-841), 0.754 (95% CI 0.698-804), and 0.776 (95% CI 0.711-832), respectively. XGBoost had an advantage over LR in AUC (0.708, 95% CI 0.641-775 vs 0.686, 95% CI 0.617-754), while RF achieved better accuracy (0.729, 95% CI 0.673-780) and sensitivity (0.790, 95% CI 0.728-844) than the other 2 approaches in model 1. Two SBI-risk prediction nomograms were developed for clinical use based on LR, and they exhibited good performance with an accuracy of 0.707 and 0.750 and a sensitivity of 0.729 and 0.927 in clinical application. CONCLUSIONS The clinical timing-sequence warning models demonstrated efficacy in predicting SBIs in patients suspected of having infective fever and in clinical application, suggesting good potential in clinical decision-making. Nevertheless, additional prospective and multicenter studies are necessary to further confirm their clinical utility.
Collapse
Affiliation(s)
- Jian Liu
- Department of Intensive Care Unit, the First Affiliated Hospital, College of Medicine, Zhejiang University, Hangzhou, China
| | - Jia Chen
- Research Center for Healthcare Data Science, Zhejiang Laboratory, Hangzhou, China
| | - Yongquan Dong
- Department of Respiratory Disease, Yinzhou Second Hospital, Ningbo, China
| | - Yan Lou
- Zhejiang Provincial Key Laboratory for Drug Clinical Research and Evaluation,Department of Clinical Pharmacy, the First Affiliated Hospital, College of Medicine, Zhejiang University, Hangzhou, China
| | - Yu Tian
- Engineering Research Center of Electronic Medical Record and Intelligent Expert System, Ministry of Education, College of Biomedical Engineering and Instrument Science, Zhejiang University, Hangzhou, China
| | - Huiyao Sun
- Research Center for Healthcare Data Science, Zhejiang Laboratory, Hangzhou, China
| | - Yuqing Jin
- Research Center for Healthcare Data Science, Zhejiang Laboratory, Hangzhou, China
| | - Jingsong Li
- Research Center for Healthcare Data Science, Zhejiang Laboratory, Engineering Research Center of Electronic Medical Record and Intelligent Expert System, Ministry of Education, College of Biomedical Engineering and Instrument Science, Zhejiang University, Hangzhou, China
| | - Yunqing Qiu
- State Key Laboratory for Diagnosis and Treatment of Infectious Diseases, Zhejiang Provincial Key Laboratory for Drug Clinical Research and Evaluation, the First Affiliated Hospital, College of Medicine, Zhejiang University, Hangzhou, China
| |
Collapse
|
2
|
Chong SL, Niu C, Ong GYK, Piragasam R, Khoo ZX, Koh ZX, Guo D, Lee JH, Ong MEH, Liu N. Febrile infants risk score at triage (FIRST) for the early identification of serious bacterial infections. Sci Rep 2023; 13:15845. [PMID: 37740004 PMCID: PMC10516995 DOI: 10.1038/s41598-023-42854-z] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/18/2023] [Accepted: 09/15/2023] [Indexed: 09/24/2023] Open
Abstract
We aimed to derive the Febrile Infants Risk Score at Triage (FIRST) to quantify risk for serious bacterial infections (SBIs), defined as bacteremia, meningitis and urinary tract infections. We performed a prospective observational study on febrile infants < 3 months old at a tertiary hospital in Singapore between 2018 and 2021. We utilized machine learning and logistic regression to derive 2 models: FIRST, based on patient demographics, vital signs and history, and FIRST + , adding laboratory results to the same variables. SBIs were diagnosed in 224/1002 (22.4%) infants. Among 994 children with complete data, age (adjusted odds ratio [aOR] 1.01 95%CI 1.01-1.02, p < 0.001), high temperature (aOR 2.22 95%CI 1.69-2.91, p < 0.001), male sex (aOR 2.62 95%CI 1.86-3.70, p < 0.001) and fever of ≥ 2 days (aOR 1.79 95%CI 1.18-2.74, p = 0.007) were independently associated with SBIs. For FIRST + , abnormal urine leukocyte esterase (aOR 16.46 95%CI 10.00-27.11, p < 0.001) and procalcitonin (aOR 1.05 95%CI 1.01-1.09, p = 0.009) were further identified. A FIRST + threshold of ≥ 15% predicted risk had a sensitivity of 81.8% (95%CI 70.5-91.0%) and specificity of 65.6% (95%CI 57.8-72.7%). In the testing dataset, FIRST + had an area under receiver operating characteristic curve of 0.87 (95%CI 0.81-0.94). These scores can potentially guide triage and prioritization of febrile infants.
Collapse
Affiliation(s)
- Shu-Ling Chong
- Department of Emergency Medicine, KK Women's and Children's Hospital, 100 Bukit Timah Road, Singapore, 229899, Singapore.
- Paediatrics Academic Clinical Programme, Duke-NUS Medical School, 8 College Road, Singapore, 169857, Singapore.
- Emergency Medicine Academic Clinical Programme, Duke-NUS Medical School, 8 College Road, Singapore, 169857, Singapore.
| | - Chenglin Niu
- Centre for Quantitative Medicine, Duke-NUS Medical School, 8 College Road, Singapore, 169857, Singapore
| | - Gene Yong-Kwang Ong
- Department of Emergency Medicine, KK Women's and Children's Hospital, 100 Bukit Timah Road, Singapore, 229899, Singapore
- Paediatrics Academic Clinical Programme, Duke-NUS Medical School, 8 College Road, Singapore, 169857, Singapore
- Emergency Medicine Academic Clinical Programme, Duke-NUS Medical School, 8 College Road, Singapore, 169857, Singapore
| | - Rupini Piragasam
- KK Research Centre, KK Women's and Children's Hospital, 100 Bukit Timah Road, Singapore, 229899, Singapore
| | - Zi Xean Khoo
- Paediatrics Academic Clinical Programme, Duke-NUS Medical School, 8 College Road, Singapore, 169857, Singapore
- Department of Paediatrics, KK Women's and Children's Hospital, 100 Bukit Timah Road, Singapore, 229899, Singapore
| | - Zhi Xiong Koh
- Department of Emergency Medicine, Singapore General Hospital, 1 Hospital Crescent, Outram Road, Singapore, 169608, Singapore
| | - Dagang Guo
- Centre for Quantitative Medicine, Duke-NUS Medical School, 8 College Road, Singapore, 169857, Singapore
- Department of Emergency Medicine, Singapore General Hospital, 1 Hospital Crescent, Outram Road, Singapore, 169608, Singapore
| | - Jan Hau Lee
- Paediatrics Academic Clinical Programme, Duke-NUS Medical School, 8 College Road, Singapore, 169857, Singapore
- Children's Intensive Care Unit, KK Women's and Children's Hospital, 100 Bukit Timah Road, Singapore, 229899, Singapore
| | - Marcus Eng Hock Ong
- Emergency Medicine Academic Clinical Programme, Duke-NUS Medical School, 8 College Road, Singapore, 169857, Singapore
- Department of Emergency Medicine, Singapore General Hospital, 1 Hospital Crescent, Outram Road, Singapore, 169608, Singapore
- Health Services Research Centre, Singapore Health Services, 8 College Road, Singapore, 169857, Singapore
| | - Nan Liu
- Centre for Quantitative Medicine, Duke-NUS Medical School, 8 College Road, Singapore, 169857, Singapore
- Department of Emergency Medicine, Singapore General Hospital, 1 Hospital Crescent, Outram Road, Singapore, 169608, Singapore
- Health Services Research Centre, Singapore Health Services, 8 College Road, Singapore, 169857, Singapore
| |
Collapse
|
3
|
Yang Y, Wang YM, Lin CHR, Cheng CY, Tsai CM, Huang YH, Chen TY, Chiu IM. Explainable deep learning model to predict invasive bacterial infection in febrile young infants: A retrospective study. Int J Med Inform 2023; 172:105007. [PMID: 36731394 DOI: 10.1016/j.ijmedinf.2023.105007] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/03/2022] [Revised: 12/09/2022] [Accepted: 01/24/2023] [Indexed: 01/29/2023]
Abstract
BACKGROUND Machine learning models have demonstrated superior performance in predicting invasive bacterial infection (IBI) in febrile infants compared to commonly used risk stratification criteria in recent studies. However, the black-box nature of these models can make them difficult to apply in clinical practice. In this study, we developed and validated an explainable deep learning model that can predict IBI in febrile infants ≤ 60 days of age visiting the emergency department. METHODS We conducted a retrospective study of febrile infants aged ≤ 60 days who presented to the pediatric emergency department of a medical center in Taiwan between January 1, 2011 and December 31, 2019. Patients with uncertain test results and complex chronic health conditions were excluded. IBI was defined as the growth of a pathogen in the blood or cerebrospinal fluid. We used a deep neural network to develop a predictive model for IBI and compared its performance to the IBI score and step-by-step approach. The SHapley Additive Explanations (SHAP) technique was used to explain the model's predictions at different levels. RESULTS Our study included 1847 patients, 53 (2.7%) of whom had IBI. The deep learning model performed similarly to the IBI score and step-by-step approach in terms of sensitivity and negative predictive value, but provided better specificity (54%), positive predictive value (5%), and area under the receiver-operating characteristic curve (0.87). SHapley Additive exPlanations identified five influential predictive variables (absolute neutrophil count, body temperature, heart rate, age, and C-reactive protein). CONCLUSION We have developed an explainable deep learning model that can predict IBI in febrile infants aged 0-60 days. The model not only performs better than previous scoring systems, but also provides insight into how it arrives at its predictions through individual features and cases.
Collapse
Affiliation(s)
- Ying Yang
- Department of Emergency Medicine, Kaohsiung Chang Gung Memorial Hospital, Kaohsiung, Taiwan
| | - Yi-Min Wang
- Department of Emergency Medicine, Kaohsiung Chang Gung Memorial Hospital, Kaohsiung, Taiwan
| | - Chun-Hung Richard Lin
- Department of Computer Science and Engineering, National Sun Yat-sen University, Kaohsiung, Taiwan
| | - Chi-Yung Cheng
- Department of Emergency Medicine, Kaohsiung Chang Gung Memorial Hospital, Kaohsiung, Taiwan; Department of Computer Science and Engineering, National Sun Yat-sen University, Kaohsiung, Taiwan
| | - Chi-Ming Tsai
- Department of Emergency Medicine, Kaohsiung Chang Gung Memorial Hospital, Kaohsiung, Taiwan; Department of Computer Science and Engineering, National Sun Yat-sen University, Kaohsiung, Taiwan
| | - Ying-Hsien Huang
- Department of Pediatrics, Kaohsiung Chang Gung Memorial Hospital, Kaohsiung, Taiwan
| | - Tien-Yu Chen
- Division of Cardiology, Department of Internal Medicine, Kaohsiung Chang Gung Memorial Hospital, Kaohsiung, Taiwan
| | - I-Min Chiu
- Department of Emergency Medicine, Kaohsiung Chang Gung Memorial Hospital, Kaohsiung, Taiwan; Department of Computer Science and Engineering, National Sun Yat-sen University, Kaohsiung, Taiwan.
| |
Collapse
|
4
|
Nomura O, Ihara T, Morikawa Y, Sakakibara H, Horikoshi Y, Inoue N. Predictor of Early Administration of Antibiotics and a Volume Resuscitation for Young Infants with Septic Shock. Antibiotics (Basel) 2021; 10:1414. [PMID: 34827352 PMCID: PMC8615069 DOI: 10.3390/antibiotics10111414] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/08/2021] [Revised: 11/16/2021] [Accepted: 11/17/2021] [Indexed: 12/02/2022] Open
Abstract
(1) Background: It is critical to administer antibiotics and fluid bolus within 1 h of recognizing sepsis in pediatric patients. This study aimed to identify the predictor of the successful completion of a 1-h sepsis bundle for infants with suspected sepsis. (2) Methods: This is an observational study using a prospective registry including febrile young infants (aged < 90 days) who visited a pediatric emergency department with a core body temperature of 38.0 °C or higher and 36.0 °C or lower. Univariate and logistic regression analyses were conducted to determine the predictor (s) of successful sepsis bundle completion. (3) Results: Of the 323 registered patients, 118 patients with suspected sepsis were analyzed, and 38 patients (32.2%) received a bundle-compliant treatment. Among potential variables, such as age, sex, and vital sign parameters, the logistic regression analysis showed that heart rate (odds ratio: OR 1.02; 95% confidence interval: 1.00-1.04) is a significant predictor of the completion of a 1-h sepsis bundle. (4) Conclusions: We found that tachycardia facilitated the sepsis recognition and promoted the successful completion of a 1-h sepsis bundle for young infants with suspected septic shock and a possible indicator for improving the quality of the team-based sepsis management.
Collapse
Affiliation(s)
- Osamu Nomura
- Department of Emergency and Disaster Medicine, Hirosaki University, Hirosaki 036-8562, Japan
- Division of Pediatric Emergency Medicine, Tokyo Metropolitan Children’s Medical Center, Tokyo 183-8561, Japan;
| | - Takateru Ihara
- Division of Pediatric Emergency Medicine, Tokyo Metropolitan Children’s Medical Center, Tokyo 183-8561, Japan;
| | - Yoshihiko Morikawa
- Clinical Research Support Center, Tokyo Metropolitan Children’s Medical Center, Tokyo 183-8561, Japan;
| | - Hiroshi Sakakibara
- Division of General Pediatrics, Department of Pediatrics, Tokyo Metropolitan Children’s Medical Center, Tokyo 183-8561, Japan;
| | - Yuho Horikoshi
- Division of Infectious Diseases, Department of Pediatrics, Tokyo Metropolitan Children’s Medical Center, Tokyo 183-8561, Japan;
| | - Nobuaki Inoue
- Department of Human Resources and Development, National Center for Global Health and Medicine, Tokyo 162-8655, Japan;
| |
Collapse
|