1
|
Weidmann AE, Watson EW. Novel opportunities for clinical pharmacy research: development of a machine learning model to identify medication related causes of delirium in different patient groups. Int J Clin Pharm 2024; 46:992-995. [PMID: 38594470 PMCID: PMC11286716 DOI: 10.1007/s11096-024-01707-z] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/02/2024] [Accepted: 01/22/2024] [Indexed: 04/11/2024]
Abstract
The advent of artificial intelligence (AI) technologies has taken the world of science by storm in 2023. The opportunities of this easy to access technology for clinical pharmacy research are yet to be fully understood. The development of a custom-made large language model (LLM) (DELSTAR) trained on a wide range of internationally recognised scientific publication databases, pharmacovigilance sites and international product characteristics to help identify and summarise medication related information on delirium, as a proof-of-concept model, identified new facilitators and barriers for robust clinical pharmacy practice research. This technology holds great promise for the development of much more comprehensive prescribing guidelines, practice support applications for clinical pharmacy, increased patient and prescribing safety and resultant implications for healthcare costs. The challenge will be to ensure its methodologically robust use and the detailed and transparent verification of its information accuracy.
Collapse
Affiliation(s)
- Anita Elaine Weidmann
- Department of Clinical Pharmacy, Institute of Pharmacy, Innsbruck University, Innrain 80, 6020, Innsbruck, Austria.
| | - Edward William Watson
- Department of Media and Learning Technology, Innsbruck University, Innrain 52, 6020, Innsbruck, Austria
| |
Collapse
|
2
|
Li Q, Li J, Chen J, Zhao X, Zhuang J, Zhong G, Song Y, Lei L. A machine learning-based prediction model for postoperative delirium in cardiac valve surgery using electronic health records. BMC Cardiovasc Disord 2024; 24:56. [PMID: 38238677 PMCID: PMC10795338 DOI: 10.1186/s12872-024-03723-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/01/2023] [Accepted: 01/11/2024] [Indexed: 01/22/2024] Open
Abstract
BACKGROUND Previous models for predicting delirium after cardiac surgery remained inadequate. This study aimed to develop and validate a machine learning-based prediction model for postoperative delirium (POD) in cardiac valve surgery patients. METHODS The electronic medical information of the cardiac surgical intensive care unit (CSICU) was extracted from a tertiary and major referral hospital in southern China over 1 year, from June 2019 to June 2020. A total of 507 patients admitted to the CSICU after cardiac valve surgery were included in this study. Seven classical machine learning algorithms (Random Forest Classifier, Logistic Regression, Support Vector Machine Classifier, K-nearest Neighbors Classifier, Gaussian Naive Bayes, Gradient Boosting Decision Tree, and Perceptron.) were used to develop delirium prediction models under full (q = 31) and selected (q = 19) feature sets, respectively. RESULT The Random Forest classifier performs exceptionally well in both feature datasets, with an Area Under the Curve (AUC) of 0.92 for the full feature dataset and an AUC of 0.86 for the selected feature dataset. Additionally, it achieves a relatively lower Expected Calibration Error (ECE) and the highest Average Precision (AP), with an AP of 0.80 for the full feature dataset and an AP of 0.73 for the selected feature dataset. To further evaluate the best-performing Random Forest classifier, SHAP (Shapley Additive Explanations) was used, and the importance matrix plot, scatter plots, and summary plots were generated. CONCLUSIONS We established machine learning-based prediction models to predict POD in patients undergoing cardiac valve surgery. The random forest model has the best predictive performance in prediction and can help improve the prognosis of patients with POD.
Collapse
Affiliation(s)
- Qiuying Li
- Department of Cardiac Surgical Intensive Care Unit, Guangdong Cardiovascular Institute, Guangdong Provincial People's Hospital, Guangdong Academy of Medical Sciences, Southern Medical University, Guangzhou, 510080, China
- Shantou University Medical College (SUMC), Shantou, 515041, China
| | - Jiaxin Li
- Department of Cardiac Surgical Intensive Care Unit, Guangdong Cardiovascular Institute, Guangdong Provincial People's Hospital, Guangdong Academy of Medical Sciences, Southern Medical University, Guangzhou, 510080, China
| | - Jiansong Chen
- Department of Cardiovascular Surgery, Guangdong General Hospital's Nanhai Hospital, The Second People's Hospital of Nanhai District, Foshan, Guangdong, 528251, China
| | - Xu Zhao
- Institute of Clinical Pharmacology, Guangdong Provincial Key Laboratory of New Drug Design and Evaluation, School of Pharmaceutical Sciences, Sun Yat-Sen University, Guangzhou, Guangdong, China
| | - Jian Zhuang
- Department of Cardiovascular Surgery, Guangdong Cardiovascular Institute, Guangdong Provincial People's Hospital, Guangdong Academy of Medical Sciences, Southern Medical University, Guangzhou, 510080, China
- Guangdong Provincial Key Laboratory of South China Structural Heart Disease, Guangdong Cardiovascular Institute, Guangdong Provincial People's Hospital, Guangdong Academy of Medical Sciences, Southern Medical University, Guangzhou, 510080, China
| | - Guoping Zhong
- Institute of Clinical Pharmacology, Guangdong Provincial Key Laboratory of New Drug Design and Evaluation, School of Pharmaceutical Sciences, Sun Yat-Sen University, Guangzhou, Guangdong, China.
| | - Yamin Song
- Department of Cardiac Surgical Intensive Care Unit, Guangdong Cardiovascular Institute, Guangdong Provincial People's Hospital, Guangdong Academy of Medical Sciences, Southern Medical University, Guangzhou, 510080, China.
| | - Liming Lei
- Department of Cardiac Surgical Intensive Care Unit, Guangdong Cardiovascular Institute, Guangdong Provincial People's Hospital, Guangdong Academy of Medical Sciences, Southern Medical University, Guangzhou, 510080, China.
- Guangdong Provincial Key Laboratory of South China Structural Heart Disease, Guangdong Cardiovascular Institute, Guangdong Provincial People's Hospital, Guangdong Academy of Medical Sciences, Southern Medical University, Guangzhou, 510080, China.
- Shantou University Medical College (SUMC), Shantou, 515041, China.
| |
Collapse
|
3
|
Bednorz A, Mak JKL, Jylhävä J, Religa D. Use of Electronic Medical Records (EMR) in Gerontology: Benefits, Considerations and a Promising Future. Clin Interv Aging 2023; 18:2171-2183. [PMID: 38152074 PMCID: PMC10752027 DOI: 10.2147/cia.s400887] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/29/2023] [Accepted: 11/05/2023] [Indexed: 12/29/2023] Open
Abstract
Electronic medical records (EMRs) have many benefits in clinical research in gerontology, enabling data analysis, development of prognostic tools and disease risk prediction. EMRs also offer a range of advantages in clinical practice, such as comprehensive medical records, streamlined communication with healthcare providers, remote data access, and rapid retrieval of test results, ultimately leading to increased efficiency, enhanced patient safety, and improved quality of care in gerontology, which includes benefits like reduced medication use and better patient history taking and physical examination assessments. The use of artificial intelligence (AI) and machine learning (ML) approaches on EMRs can further improve disease diagnosis, symptom classification, and support clinical decision-making. However, there are also challenges related to data quality, data entry errors, as well as the ethics and safety of using AI in healthcare. This article discusses the future of EMRs in gerontology and the application of AI and ML in clinical research. Ethical and legal issues surrounding data sharing and the need for healthcare professionals to critically evaluate and integrate these technologies are also emphasized. The article concludes by discussing the challenges related to the use of EMRs in research as well as in their primary intended use, the daily clinical practice.
Collapse
Affiliation(s)
- Adam Bednorz
- John Paul II Geriatric Hospital, Katowice, Poland
- Institute of Psychology, Humanitas Academy, Sosnowiec, Poland
| | - Jonathan K L Mak
- Department of Medical Epidemiology and Biostatistics, Karolinska Institutet, Stockholm, Sweden
| | - Juulia Jylhävä
- Department of Medical Epidemiology and Biostatistics, Karolinska Institutet, Stockholm, Sweden
- Faculty of Social Sciences (Health Sciences) and Gerontology Research Center (GEREC), University of Tampere, Tampere, Finland
| | - Dorota Religa
- Division of Clinical Geriatrics, Department of Neurobiology, Care sciences and Society, Karolinska Institutet, Stockholm, Sweden
- Theme Inflammation and Aging, Karolinska University Hospital, Huddinge, Sweden
| |
Collapse
|
4
|
Ariño H, Bae SK, Chaturvedi J, Wang T, Roberts A. Identifying encephalopathy in patients admitted to an intensive care unit: Going beyond structured information using natural language processing. Front Digit Health 2023; 5:1085602. [PMID: 36755566 PMCID: PMC9899891 DOI: 10.3389/fdgth.2023.1085602] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/31/2022] [Accepted: 01/05/2023] [Indexed: 01/24/2023] Open
Abstract
Background Encephalopathy is a severe co-morbid condition in critically ill patients that includes different clinical constellation of neurological symptoms. However, even for the most recognised form, delirium, this medical condition is rarely recorded in structured fields of electronic health records precluding large and unbiased retrospective studies. We aimed to identify patients with encephalopathy using a machine learning-based approach over clinical notes in electronic health records. Methods We used a list of ICD-9 codes and clinical concepts related to encephalopathy to define a cohort of patients from the MIMIC-III dataset. Clinical notes were annotated with MedCAT and vectorized with a bag-of-word approach or word embedding using clinical concepts normalised to standard nomenclatures as features. Machine learning algorithms (support vector machines and random forest) trained with clinical notes from patients who had a diagnosis of encephalopathy (defined by ICD-9 codes) were used to classify patients with clinical concepts related to encephalopathy in their clinical notes but without any ICD-9 relevant code. A random selection of 50 patients were reviewed by a clinical expert for model validation. Results Among 46,520 different patients, 7.5% had encephalopathy related ICD-9 codes in all their admissions (group 1, definite encephalopathy), 45% clinical concepts related to encephalopathy only in their clinical notes (group 2, possible encephalopathy) and 38% did not have encephalopathy related concepts neither in structured nor in clinical notes (group 3, non-encephalopathy). Length of stay, mortality rate or number of co-morbid conditions were higher in groups 1 and 2 compared to group 3. The best model to classify patients from group 2 as patients with encephalopathy (SVM using embeddings) had F1 of 85% and predicted 31% patients from group 2 as having encephalopathy with a probability >90%. Validation on new cases found a precision ranging from 92% to 98% depending on the criteria considered. Conclusions Natural language processing techniques can leverage relevant clinical information that might help to identify patients with under-recognised clinical disorders such as encephalopathy. In the MIMIC dataset, this approach identifies with high probability thousands of patients that did not have a formal diagnosis in the structured information of the EHR.
Collapse
Affiliation(s)
- Helena Ariño
- Institut D’Investigacions Biomèdiques August Pi I Sunyer (IDIBAPS), Barcelona, Spain,Institute of Psychiatry, Psychology and Neuroscience, King’s College London, London, United Kingdom
| | - Soo Kyung Bae
- Dept. of Integrated Medicine, Yonsei University College of Medicine, Seoul, South Korea,Translational AI Laboratory, Yonsei University College of Medicine, Seoul, South Korea
| | - Jaya Chaturvedi
- Institute of Psychiatry, Psychology and Neuroscience, King’s College London, London, United Kingdom
| | - Tao Wang
- Institute of Psychiatry, Psychology and Neuroscience, King’s College London, London, United Kingdom,Correspondence: Tao Wang
| | - Angus Roberts
- Institute of Psychiatry, Psychology and Neuroscience, King’s College London, London, United Kingdom,National Institute for Health Research, Maudsley Biomedical Research Centre, South London and Maudsley National Health Service (NHS) Foundation Trust, London, United Kingdom
| |
Collapse
|
5
|
Jiang Z, Cai Y, Liu S, Ye P, Yang Y, Lin G, Li S, Xu Y, Zheng Y, Bao Z, Nie S, Gu W. Decreased default mode network functional connectivity with visual processing regions as potential biomarkers for delayed neurocognitive recovery: A resting-state fMRI study and machine-learning analysis. Front Aging Neurosci 2023; 14:1109485. [PMID: 36688167 PMCID: PMC9853194 DOI: 10.3389/fnagi.2022.1109485] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/27/2022] [Accepted: 12/16/2022] [Indexed: 01/07/2023] Open
Abstract
Objectives The abnormal functional connectivity (FC) pattern of default mode network (DMN) may be key markers for early identification of various cognitive disorders. However, the whole-brain FC changes of DMN in delayed neurocognitive recovery (DNR) are still unclear. Our study was aimed at exploring the whole-brain FC patterns of all regions in DMN and the potential features as biomarkers for the prediction of DNR using machine-learning algorithms. Methods Resting-state functional magnetic resonance imaging (fMRI) was conducted before surgery on 74 patients undergoing non-cardiac surgery. Seed-based whole-brain FC with 18 core regions located in the DMN was performed, and FC features that were statistically different between the DNR and non-DNR patients after false discovery correction were extracted. Afterward, based on the extracted FC features, machine-learning algorithms such as support vector machine, logistic regression, decision tree, and random forest were established to recognize DNR. The machine learning experiment procedure mainly included three following steps: feature standardization, parameter adjustment, and performance comparison. Finally, independent testing was conducted to validate the established prediction model. The algorithm performance was evaluated by a permutation test. Results We found significantly decreased DMN connectivity with the brain regions involved in visual processing in DNR patients than in non-DNR patients. The best result was obtained from the random forest algorithm based on the 20 decision trees (estimators). The random forest model achieved the accuracy, sensitivity, and specificity of 84.0, 63.1, and 89.5%, respectively. The area under the receiver operating characteristic curve of the classifier reached 86.4%. The feature that contributed the most to the random forest model was the FC between the left retrosplenial cortex/posterior cingulate cortex and left precuneus. Conclusion The decreased FC of DMN with regions involved in visual processing might be effective markers for the prediction of DNR and could provide new insights into the neural mechanisms of DNR. Clinical Trial Registration : Chinese Clinical Trial Registry, ChiCTR-DCD-15006096.
Collapse
Affiliation(s)
- Zhaoshun Jiang
- Department of Anesthesiology, Huadong Hospital Affiliated to Fudan University, Shanghai, China,Shanghai Key Laboratory of Clinical Geriatric Medicine, Huadong Hospital Affiliated to Fudan University, Shanghai, China
| | - Yuxi Cai
- Department of Anesthesiology, Huadong Hospital Affiliated to Fudan University, Shanghai, China,Shanghai Key Laboratory of Clinical Geriatric Medicine, Huadong Hospital Affiliated to Fudan University, Shanghai, China
| | - Songbin Liu
- Department of Anesthesiology, Huadong Hospital Affiliated to Fudan University, Shanghai, China,Shanghai Key Laboratory of Clinical Geriatric Medicine, Huadong Hospital Affiliated to Fudan University, Shanghai, China
| | - Pei Ye
- Department of Anesthesiology, Huadong Hospital Affiliated to Fudan University, Shanghai, China,Shanghai Key Laboratory of Clinical Geriatric Medicine, Huadong Hospital Affiliated to Fudan University, Shanghai, China
| | - Yifeng Yang
- School of Health Science and Engineering, University of Shanghai for Science and Technology, Shanghai, China
| | - Guangwu Lin
- Department of Radiology, Huadong Hospital Affiliated to Fudan University, Shanghai, China
| | - Shihong Li
- Department of Radiology, Huadong Hospital Affiliated to Fudan University, Shanghai, China
| | - Yan Xu
- Department of Anesthesiology, Huadong Hospital Affiliated to Fudan University, Shanghai, China
| | - Yangjing Zheng
- Department of Anesthesiology, Huadong Hospital Affiliated to Fudan University, Shanghai, China
| | - Zhijun Bao
- Shanghai Key Laboratory of Clinical Geriatric Medicine, Huadong Hospital Affiliated to Fudan University, Shanghai, China,Department of Geriatric Medicine, Huadong Hospital Affiliated to Fudan University, Shanghai, China,Research Center on Aging and Medicine, Fudan University, Shanghai, China
| | - Shengdong Nie
- School of Health Science and Engineering, University of Shanghai for Science and Technology, Shanghai, China,Shengdong Nie,
| | - Weidong Gu
- Department of Anesthesiology, Huadong Hospital Affiliated to Fudan University, Shanghai, China,Shanghai Key Laboratory of Clinical Geriatric Medicine, Huadong Hospital Affiliated to Fudan University, Shanghai, China,*Correspondence: Weidong Gu,
| |
Collapse
|
6
|
Wang L, Zhang Y, Chignell M, Shan B, Sheehan KA, Razak F, Verma A. Boosting Delirium Identification Accuracy With Sentiment-Based Natural Language Processing: Mixed Methods Study. JMIR Med Inform 2022; 10:e38161. [PMID: 36538363 PMCID: PMC9812273 DOI: 10.2196/38161] [Citation(s) in RCA: 6] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/21/2022] [Revised: 08/22/2022] [Accepted: 09/19/2022] [Indexed: 01/07/2023] Open
Abstract
BACKGROUND Delirium is an acute neurocognitive disorder that affects up to half of older hospitalized medical patients and can lead to dementia, longer hospital stays, increased health costs, and death. Although delirium can be prevented and treated, it is difficult to identify and predict. OBJECTIVE This study aimed to improve machine learning models that retrospectively identify the presence of delirium during hospital stays (eg, to measure the effectiveness of delirium prevention interventions) by using the natural language processing (NLP) technique of sentiment analysis (in this case a feature that identifies sentiment toward, or away from, a delirium diagnosis). METHODS Using data from the General Medicine Inpatient Initiative, a Canadian hospital data and analytics network, a detailed manual review of medical records was conducted from nearly 4000 admissions at 6 Toronto area hospitals. Furthermore, 25.74% (994/3862) of the eligible hospital admissions were labeled as having delirium. Using the data set collected from this study, we developed machine learning models with, and without, the benefit of NLP methods applied to diagnostic imaging reports, and we asked the question "can NLP improve machine learning identification of delirium?" RESULTS Among the eligible 3862 hospital admissions, 994 (25.74%) admissions were labeled as having delirium. Identification and calibration of the models were satisfactory. The accuracy and area under the receiver operating characteristic curve of the main model with NLP in the independent testing data set were 0.807 and 0.930, respectively. The accuracy and area under the receiver operating characteristic curve of the main model without NLP in the independent testing data set were 0.811 and 0.869, respectively. Model performance was also found to be stable over the 5-year period used in the experiment, with identification for a likely future holdout test set being no worse than identification for retrospective holdout test sets. CONCLUSIONS Our machine learning model that included NLP (ie, sentiment analysis in medical image description text mining) produced valid identification of delirium with the sentiment analysis, providing significant additional benefit over the model without NLP.
Collapse
Affiliation(s)
- Lu Wang
- Department of Mechanical & Industrial Engineering, University of Toronto, Toronto, ON, Canada
- Department of Computer Science, Texas State University, San Marcos, TX, United States
| | - Yilun Zhang
- Department of Mechanical & Industrial Engineering, University of Toronto, Toronto, ON, Canada
| | - Mark Chignell
- Department of Mechanical & Industrial Engineering, University of Toronto, Toronto, ON, Canada
| | - Baizun Shan
- Department of Mechanical & Industrial Engineering, University of Toronto, Toronto, ON, Canada
| | - Kathleen A Sheehan
- GEMINI - The General Medicine Inpatient Initiative, Unity Health Toronto, Toronto, ON, Canada
- Department of Psychiatry, University of Toronto, Toronto, ON, Canada
| | - Fahad Razak
- GEMINI - The General Medicine Inpatient Initiative, Unity Health Toronto, Toronto, ON, Canada
- Faculty of Medicine & Institute of Health Policy, Management and Evaluation, University of Toronto, Toronto, ON, Canada
| | - Amol Verma
- GEMINI - The General Medicine Inpatient Initiative, Unity Health Toronto, Toronto, ON, Canada
- Faculty of Medicine & Institute of Health Policy, Management and Evaluation, University of Toronto, Toronto, ON, Canada
| |
Collapse
|
7
|
Zhang X, Xue Y, Su X, Chen S, Liu K, Chen W, Liu M, Hu Y. A Transfer Learning Approach to Correct the Temporal Performance Drift of Clinical Prediction Models: Retrospective Cohort Study. JMIR Med Inform 2022; 10:e38053. [DOI: 10.2196/38053] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/17/2022] [Revised: 07/31/2022] [Accepted: 10/12/2022] [Indexed: 11/11/2022] Open
Abstract
Background
Clinical prediction models suffer from performance drift as the patient population shifts over time. There is a great need for model updating approaches or modeling frameworks that can effectively use the old and new data.
Objective
Based on the paradigm of transfer learning, we aimed to develop a novel modeling framework that transfers old knowledge to the new environment for prediction tasks, and contributes to performance drift correction.
Methods
The proposed predictive modeling framework maintains a logistic regression–based stacking ensemble of 2 gradient boosting machine (GBM) models representing old and new knowledge learned from old and new data, respectively (referred to as transfer learning gradient boosting machine [TransferGBM]). The ensemble learning procedure can dynamically balance the old and new knowledge. Using 2010-2017 electronic health record data on a retrospective cohort of 141,696 patients, we validated TransferGBM for hospital-acquired acute kidney injury prediction.
Results
The baseline models (ie, transported models) that were trained on 2010 and 2011 data showed significant performance drift in the temporal validation with 2012-2017 data. Refitting these models using updated samples resulted in performance gains in nearly all cases. The proposed TransferGBM model succeeded in achieving uniformly better performance than the refitted models.
Conclusions
Under the scenario of population shift, incorporating new knowledge while preserving old knowledge is essential for maintaining stable performance. Transfer learning combined with stacking ensemble learning can help achieve a balance of old and new knowledge in a flexible and adaptive way, even in the case of insufficient new data.
Collapse
|
8
|
Jung JW, Hwang S, Ko S, Jo C, Park HY, Han HS, Lee MC, Park JE, Ro DH. A machine-learning model to predict postoperative delirium following knee arthroplasty using electronic health records. BMC Psychiatry 2022; 22:436. [PMID: 35761274 PMCID: PMC9235137 DOI: 10.1186/s12888-022-04067-y] [Citation(s) in RCA: 6] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 11/01/2021] [Accepted: 06/08/2022] [Indexed: 11/17/2022] Open
Abstract
BACKGROUND Postoperative delirium is a challenging complication due to its adverse outcome such as long hospital stay. The aims of this study were: 1) to identify preoperative risk factors of postoperative delirium following knee arthroplasty, and 2) to develop a machine-learning prediction model. METHOD A total of 3,980 patients from two hospitals were included in this study. The model was developed and trained with 1,931 patients from one hospital and externally validated with 2,049 patients from another hospital. Twenty preoperative variables were collected using electronic hospital records. Feature selection was conducted using the sequential feature selection (SFS). Extreme Gradient Boosting algorithm (XGBoost) model as a machine-learning classifier was applied to predict delirium. A tenfold-stratified area under the curve (AUC) served as the metric for variable selection and internal validation. RESULTS The incidence rate of delirium was 4.9% (n = 196). The following seven key predictors of postoperative delirium were selected: age, serum albumin, number of hypnotics and sedatives drugs taken preoperatively, total number of drugs (any kinds of oral medication) taken preoperatively, neurologic disorders, depression, and fall-down risk (all p < 0.05). The predictive performance of our model was good for the developmental cohort (AUC: 0.80, 95% CI: 0.77-0.84). It was also good for the external validation cohort (AUC: 0.82, 95% CI: 0.80-0.83). Our model can be accessed at https://safetka.connecteve.com . CONCLUSIONS A web-based predictive model for delirium after knee arthroplasty was developed using a machine-learning algorithm featuring seven preoperative variables. This model can be used only with information that can be obtained from pre-operative electronic hospital records. Thus, this model could be used to predict delirium before surgery and may assist physician's effort on delirium prevention.
Collapse
Affiliation(s)
- Jong Wook Jung
- grid.31501.360000 0004 0470 5905Department of Orthopedic Surgery, Seoul National University College of Medicine, Seoul, South Korea
| | - Sunghyun Hwang
- grid.412484.f0000 0001 0302 820XDepartment of Orthopedic Surgery, Seoul National University Hospital, 101 Daehak-ro, Jongno-gu, Seoul, 110-744 South Korea
| | - Sunho Ko
- grid.31501.360000 0004 0470 5905Department of Orthopedic Surgery, Seoul National University College of Medicine, Seoul, South Korea
| | - Changwung Jo
- grid.31501.360000 0004 0470 5905Department of Orthopedic Surgery, Seoul National University College of Medicine, Seoul, South Korea
| | - Hye Youn Park
- grid.412480.b0000 0004 0647 3378Department of Psychiatry, Seoul National University Bundang Hospital, Seoul, South Korea
| | - Hyuk-Soo Han
- grid.31501.360000 0004 0470 5905Department of Orthopedic Surgery, Seoul National University College of Medicine, Seoul, South Korea ,grid.412484.f0000 0001 0302 820XDepartment of Orthopedic Surgery, Seoul National University Hospital, 101 Daehak-ro, Jongno-gu, Seoul, 110-744 South Korea
| | - Myung Chul Lee
- grid.31501.360000 0004 0470 5905Department of Orthopedic Surgery, Seoul National University College of Medicine, Seoul, South Korea ,grid.412484.f0000 0001 0302 820XDepartment of Orthopedic Surgery, Seoul National University Hospital, 101 Daehak-ro, Jongno-gu, Seoul, 110-744 South Korea
| | - Jee Eun Park
- grid.412484.f0000 0001 0302 820XDepartment of Psychiatry, Seoul National University Hospital, Seoul, South Korea
| | - Du Hyun Ro
- Department of Orthopedic Surgery, Seoul National University College of Medicine, Seoul, South Korea. .,Department of Orthopedic Surgery, Seoul National University Hospital, 101 Daehak-ro, Jongno-gu, Seoul, 110-744, South Korea. .,CONNECTEVE Co., LTD., Seoul, South Korea.
| |
Collapse
|
9
|
Wu C, Zhou T, Tian Y, Wu J, Li J, Liu Z. A method for the early prediction of chronic diseases based on short sequential medical data. Artif Intell Med 2022; 127:102262. [DOI: 10.1016/j.artmed.2022.102262] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/09/2020] [Revised: 02/18/2022] [Accepted: 02/23/2022] [Indexed: 11/30/2022]
|
10
|
Bosma M, Jansen SA, Gawel JH, van Dullemen CEM, Priems MM, Westerhof A, Meijer AR, Ruven HJT. Prediction of the Values of CRP, eGFR, and Hemoglobin in the Follow-Up of Renal Cell Carcinoma Patients after (Cryo)Surgery Using Machine Learning Algorithms. J Appl Lab Med 2022; 7:819-826. [DOI: 10.1093/jalm/jfab177] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/03/2021] [Accepted: 11/10/2021] [Indexed: 01/11/2023]
Abstract
Abstract
Background
Artificial intelligence can support clinical decisions by predictive modeling. Using patient-specific characteristics, models may predict the course of clinical parameters, thus guiding monitoring approaches for the individual patient. Here, we present prediction models for inflammation and for the course of renal function and hemoglobin (Hb) in renal cell carcinoma patients after (cryo)surgery.
Methods
Using random forest machine learning in a longitudinal value-based healthcare data set (n = 86) of renal cell carcinoma patients, prediction models were established and optimized using random and grid searches. Data were split into a training and test set in a 70:30 ratio. Inflammation was predicted for a single timepoint, whereas for renal function estimated glomerular filtration rate (eGFR) and Hb time course prediction was performed.
Results
Whereas the last Hb and eGFR values before (cryo)surgery were the main basis for the course of Hb and renal function, age and several time frame features also contributed significantly. For eGFR, the type of (cryo)surgery was also a main predicting feature, and for Hb, tumor location, and body mass index were important predictors. With regard to prediction of inflammation no feature was markedly prominent. Inflammation prediction was based on a combination of patient characteristics, physiological parameters, and time frame features.
Conclusions
This study provided interesting insights into factors influencing complications and recovery in individual renal cell carcinoma patients. The established prediction models provide the basis for development of clinical decision support tools for selection and timing of laboratory analyses after (cryo)surgery, thus contributing to quality and efficiency of care.
Collapse
Affiliation(s)
- Madeleen Bosma
- Department of Clinical Chemistry, St. Antonius Hospital, Nieuwegein/Utrecht, The Netherlands
| | | | - Job H Gawel
- Data Science Lab, Amsterdam, The Netherlands
| | | | - Margrite M Priems
- Department of Indicators and Value-Based Healthcare, St. Antonius Hospital, Nieuwegein/Utrecht, The Netherlands
| | - Alisa Westerhof
- Department of Business Intelligence, St. Antonius Hospital, Nieuwegein/Utrecht, The Netherlands
| | - Aswin R Meijer
- Department of Urology, St. Antonius Hospital, Nieuwegein/Utrecht, The Netherlands
| | - Henk J T Ruven
- Department of Clinical Chemistry, St. Antonius Hospital, Nieuwegein/Utrecht, The Netherlands
| |
Collapse
|
11
|
Jiang Z, Cai Y, Zhang X, Lv Y, Zhang M, Li S, Lin G, Bao Z, Liu S, Gu W. Predicting Delayed Neurocognitive Recovery After Non-cardiac Surgery Using Resting-State Brain Network Patterns Combined With Machine Learning. Front Aging Neurosci 2021; 13:715517. [PMID: 34867266 PMCID: PMC8633536 DOI: 10.3389/fnagi.2021.715517] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/27/2021] [Accepted: 10/25/2021] [Indexed: 01/14/2023] Open
Abstract
Delayed neurocognitive recovery (DNR) is a common subtype of postoperative neurocognitive disorders. An objective approach for identifying subjects at high risk of DNR is yet lacking. The present study aimed to predict DNR using the machine learning method based on multiple cognitive-related brain network features. A total of 74 elderly patients (≥ 60-years-old) undergoing non-cardiac surgery were subjected to resting-state functional magnetic resonance imaging (rs-fMRI) before the surgery. Seed-based whole-brain functional connectivity (FC) was analyzed with 18 regions of interest (ROIs) located in the default mode network (DMN), limbic network, salience network (SN), and central executive network (CEN). Multiple machine learning models (support vector machine, decision tree, and random forest) were constructed to recognize the DNR based on FC network features. The experiment has three parts, including performance comparison, feature screening, and parameter adjustment. Then, the model with the best predictive efficacy for DNR was identified. Finally, independent testing was conducted to validate the established predictive model. Compared to the non-DNR group, the DNR group exhibited aberrant whole-brain FC in seven ROIs, including the right posterior cingulate cortex, right medial prefrontal cortex, and left lateral parietal cortex in the DMN, the right insula in the SN, the left anterior prefrontal cortex in the CEN, and the left ventral hippocampus and left amygdala in the limbic network. The machine learning experimental results identified a random forest model combined with FC features of DMN and CEN as the best prediction model. The area under the curve was 0.958 (accuracy = 0.935, precision = 0.899, recall = 0.900, F1 = 0.890) on the test set. Thus, the current study indicated that the random forest machine learning model based on rs-FC features of DMN and CEN predicts the DNR following non-cardiac surgery, which could be beneficial to the early prevention of DNR. Clinical Trial Registration: The study was registered at the Chinese Clinical Trial Registry (Identification number: ChiCTR-DCD-15006096).
Collapse
Affiliation(s)
- Zhaoshun Jiang
- Department of Anesthesiology, Huadong Hospital Affiliated to Fudan University, Shanghai, China.,Shanghai Key Laboratory of Clinical Geriatric Medicine, Shanghai, China
| | - Yuxi Cai
- Department of Anesthesiology, Huadong Hospital Affiliated to Fudan University, Shanghai, China.,Shanghai Key Laboratory of Clinical Geriatric Medicine, Shanghai, China
| | - Xixue Zhang
- Department of Anesthesiology, Huadong Hospital Affiliated to Fudan University, Shanghai, China.,Shanghai Key Laboratory of Clinical Geriatric Medicine, Shanghai, China
| | - Yating Lv
- Center for Cognition and Brain Disorders, The Affiliated Hospital of Hangzhou Normal University, Hangzhou, China
| | - Mengting Zhang
- Department of Anesthesiology, Huadong Hospital Affiliated to Fudan University, Shanghai, China.,Shanghai Key Laboratory of Clinical Geriatric Medicine, Shanghai, China
| | - Shihong Li
- Department of Radiology, Huadong Hospital Affiliated to Fudan University, Shanghai, China
| | - Guangwu Lin
- Department of Radiology, Huadong Hospital Affiliated to Fudan University, Shanghai, China
| | - Zhijun Bao
- Shanghai Key Laboratory of Clinical Geriatric Medicine, Shanghai, China.,Department of Geriatric Medicine, Huadong Hospital Affiliated to Fudan University, Shanghai, China.,Research Center on Aging and Medicine, Fudan University, Shanghai, China
| | - Songbin Liu
- Department of Anesthesiology, Huadong Hospital Affiliated to Fudan University, Shanghai, China.,Shanghai Key Laboratory of Clinical Geriatric Medicine, Shanghai, China
| | - Weidong Gu
- Department of Anesthesiology, Huadong Hospital Affiliated to Fudan University, Shanghai, China.,Shanghai Key Laboratory of Clinical Geriatric Medicine, Shanghai, China
| |
Collapse
|
12
|
Son CS, Kang WS, Lee JH, Moon KJ. Machine Learning to Identify Psychomotor Behaviors of Delirium for Patients in Long-Term Care Facility. IEEE J Biomed Health Inform 2021; 26:1802-1814. [PMID: 34596563 DOI: 10.1109/jbhi.2021.3116967] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/06/2022]
Abstract
This study aimed to develop accurate and explainable machine learning models for three psychomotor behaviors of delirium for hospitalized adult patients. A prospective pilot study was conducted with 33 participants admitted to a long-term care facility between August 10 and 25, 2020. During the pilot study, we collected 560 cases that included 33 clinical variables and the survey items from the short confusion assessment method (S-CAM), and developed a mobile-based application. Multiple machine learning algorithms, including four rule-mining algorithms (C4.5, CBA, MCAR, and LEM2) and four other statistical learning algorithms (LR, ANNs, SVMs with three kernel functions, and random forest), were validated by paired Wilcoxon signed-rank tests on both macro-averaged F1 and weighted average F1-measures during the 10-times stratified 2-fold cross-validation. The LEM2 algorithm achieved the best prediction performance (macro-averaged F1-measure of 49.35%; weighted average F1-measure of 96.55%), correctly identifying adult patients at delirium risk. In the pairwise comparison between predictive powers observed from independent models, the LEM2 model showed a medium or large effect size between 0.4925 and 0.8766 when compared with LR, ANN, SVM with RBF, and MCAR models. We have confirmed that acute consciousness in S-CAM assessment is closely associated with different predictors for screening three psychomotor behaviors of delirium: 1) education level, dementia type or its level, sleep disorder, dehydration, and infection in mixed-type delirium; 2) gender, education level, dementia type, dehydration, bedsores, and foley catheter in hyperactive delirium; and 3) pain, sleep disorder, and haloperidol use in hypoactive delirium.
Collapse
|
13
|
Li Q, Zhao Y, Chen Y, Yue J, Xiong Y. Developing a machine learning model to identify delirium risk in geriatric internal medicine inpatients. Eur Geriatr Med 2021; 13:173-183. [PMID: 34553310 DOI: 10.1007/s41999-021-00562-9] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/23/2021] [Accepted: 09/06/2021] [Indexed: 02/08/2023]
Abstract
PURPOSE To develop a machine learning model that predicts delirium risk in geriatric internal medicine inpatients. METHODS A prospective cohort study of internal medicine wards in a tertiary care hospital in China. Blinded observers assessed delirium using the Confusion Assessment Method (CAM). The data set was randomly divided into a training set (70%) and a test set (30%). The model was trained on the training set using the decision tree and the five-fold cross-validation, and then the model performance was evaluated on the test set. Under-sampling was used to address the class imbalance. The discriminatory power of the model was measured by the area under the receiver operating characteristic curve (AUC) and F1 score. The data set comprised 740 patients from March 2016 to January 2017. RESULTS The training set included 518 patients; the median (IQR) age was 84 (79-87) years; 364 (70.3%) were men; 71 (13.7%) with delirium. The test set included 222 patients; the median (IQR) age was 84.5 (79-87) years; 163 (73.4%) were men; 30 (13.5%) with delirium. In total, the data set included 740 hospital admissions with a median (IQR) age of 84 (79-87) years, 527 (71.2%) were men, and 101 (13.6%) with delirium. From 32 potential predictors, we included five variables in the predictive model: depression, cognitive impairment, types of drugs, nutritional status, and activity of daily life (ADL). The mean AUC on the training set was 0.967, the AUC and F1 score on the test set was 0.950 and 0.810, respectively. The model achieved 93.3% sensitivity, 94.3% specificity, 71.8% positive predictive value, 98.9% negative predictive value, and 94.1% accuracy on the test set. CONCLUSION This machine learning model may allow more precise targeting of delirium prevention and could support clinical decision making in geriatric internal medicine wards.
Collapse
Affiliation(s)
- Qinzheng Li
- School of Mechanical Engineering, Sichuan University, Chengdu, China
| | - Yanli Zhao
- Department of Geriatrics, West China Hospital, Sichuan University, Chengdu, China
| | - Yu Chen
- Department of Applied Mechanics, Sichuan University, Chengdu, China.,Medical Big Data Center, Sichuan University, Chengdu, China
| | - Jirong Yue
- Department of Geriatrics, West China Hospital, Sichuan University, Chengdu, China.
| | - Yan Xiong
- School of Mechanical Engineering, Sichuan University, Chengdu, China.
| |
Collapse
|
14
|
Yan C, Gao C, Zhang Z, Chen W, Malin BA, Ely EW, Patel MB, Chen Y. Predicting brain function status changes in critically ill patients via Machine learning. J Am Med Inform Assoc 2021; 28:2412-2422. [PMID: 34402496 DOI: 10.1093/jamia/ocab166] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/06/2021] [Revised: 07/15/2021] [Accepted: 07/21/2021] [Indexed: 11/14/2022] Open
Abstract
OBJECTIVE In intensive care units (ICUs), a patient's brain function status can shift from a state of acute brain dysfunction (ABD) to one that is ABD-free and vice versa, which is challenging to forecast and, in turn, hampers the allocation of hospital resources. We aim to develop a machine learning model to predict next-day brain function status changes. MATERIALS AND METHODS Using multicenter prospective adult cohorts involving medical and surgical ICU patients from 2 civilian and 3 Veteran Affairs hospitals, we trained and externally validated a light gradient boosting machine to predict brain function status changes. We compared the performances of the boosting model against state-of-the-art models-an ABD predictive model and its variants. We applied Shapley additive explanations to identify influential factors to develop a compact model. RESULTS There were 1026 critically ill patients without evidence of prior major dementia, or structural brain diseases, from whom 12 295 daily transitions (ABD: 5847 days; ABD-free: 6448 days) were observed. The boosting model achieved an area under the receiver-operating characteristic curve (AUROC) of 0.824 (95% confidence interval [CI], 0.821-0.827), compared with the state-of-the-art models of 0.697 (95% CI, 0.693-0.701) with P < .001. Using 13 identified top influential factors, the compact model achieved 99.4% of the boosting model on AUROC. The boosting and the compact models demonstrated high generalizability in external validation by achieving an AUROC of 0.812 (95% CI, 0.812-0.813). CONCLUSION The inputs of the compact model are based on several simple questions that clinicians can quickly answer in practice, which demonstrates the model has direct prospective deployment potential into clinical practice, aiding in critical hospital resource allocation.
Collapse
Affiliation(s)
- Chao Yan
- Department of Electrical Engineering and Computer Science, Vanderbilt University, Nashville, Tennessee, USA
| | - Cheng Gao
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, USA
| | - Ziqi Zhang
- Department of Electrical Engineering and Computer Science, Vanderbilt University, Nashville, Tennessee, USA
| | - Wencong Chen
- Department of Biostatistics, School of Medicine, Vanderbilt University, Nashville, Tennessee, USA.,Critical Illness, Brain Dysfunction, and Survivorship Center, Vanderbilt University Medical Center, Nashville, Tennessee, USA
| | - Bradley A Malin
- Department of Electrical Engineering and Computer Science, Vanderbilt University, Nashville, Tennessee, USA.,Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, USA.,Department of Biostatistics, School of Medicine, Vanderbilt University, Nashville, Tennessee, USA
| | - E Wesley Ely
- Critical Illness, Brain Dysfunction, and Survivorship Center, Vanderbilt University Medical Center, Nashville, Tennessee, USA.,Geriatric Research and Education Clinical Center, Tennessee Valley Healthcare System, U.S. Department of Veteran Affairs, Nashville, Tennessee, USA
| | - Mayur B Patel
- Critical Illness, Brain Dysfunction, and Survivorship Center, Vanderbilt University Medical Center, Nashville, Tennessee, USA.,Geriatric Research and Education Clinical Center, Tennessee Valley Healthcare System, U.S. Department of Veteran Affairs, Nashville, Tennessee, USA.,Section of Surgical Sciences, Vanderbilt University Medical Center, Nashville, Tennessee, USA.,Department of Surgery, Vanderbilt University Medical Center, Nashville, Tennessee, USA.,Department of Neurosurgery, Vanderbilt University Medical Center, Nashville, Tennessee, USA.,Department of Hearing & Speech Sciences, Vanderbilt University Medical Center, Nashville, Tennessee, USA.,Division of Trauma, Surgical Critical Care, and Emergency General Surgery, Vanderbilt University Medical Center, Nashville, Tennessee, USA
| | - You Chen
- Department of Electrical Engineering and Computer Science, Vanderbilt University, Nashville, Tennessee, USA.,Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, USA
| |
Collapse
|
15
|
Chua SJ, Wrigley S, Hair C, Sahathevan R. Prediction of delirium using data mining: A systematic review. J Clin Neurosci 2021; 91:288-298. [PMID: 34373042 DOI: 10.1016/j.jocn.2021.07.029] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/15/2021] [Revised: 06/18/2021] [Accepted: 07/18/2021] [Indexed: 12/19/2022]
Abstract
Delirium remains a significant cause of morbidity, mortality and economic burden to society. "Big data" refers to data of significantly large volume, obtained from a variety of resources, which is created and processed at high velocity. We conducted a systematic review and meta-analysis exploring whether big data could predict the incidence of delirium of patients in the inpatient setting. Medline, Embase, the Cochrane Library, Web of Science, CINAHL, clinicaltrials.gov, who.int and IEEE Xplore were searched using MeSH terms "big data", "data mining", "delirium" and "confusion" up to 30th September 2019. We included both randomised and observational studies. The primary outcome of interest was development of delirium and the secondary outcomes of interest were type of statistical methods used, variables included in the mining algorithms and clinically important outcomes such as mortality and length of hospital stay. The quality of studies was graded using the CHARMs checklist. Six retrospective single centre observational studies were included (n = 178,091), of which 17, 574 participants developed delirium. Studies were of generally of low to moderate quality. The most commonly studied method was random forest, followed by support vector machine and artificial neural networks. The model with best performance for delirium prediction was random forest, with area under receiver operating curve (AUROC) ranging from 0.78 to 0.91. Sensitivity ranged from 0.59 to 0.81 and specificity ranged from 0.73 to 0.92. Our systematic review suggests that machine-learning techniques can be utilised to predict delirium.
Collapse
Affiliation(s)
- S J Chua
- Ballarat Health Services, Ballarat, Australia.
| | - S Wrigley
- Ballarat Health Services, Ballarat, Australia
| | - C Hair
- Ballarat Health Services, Ballarat, Australia
| | - R Sahathevan
- Ballarat Health Services, Ballarat, Australia; School of Medicine, Deakin University, Geelong, Australia; Faculty of Medicine, Dentistry and Health Sciences, University of Melbourne, Melbourne, Australia
| |
Collapse
|
16
|
Hur S, Ko RE, Yoo J, Ha J, Cha WC, Chung CR. A Machine Learning-Based Algorithm for the Prediction of Intensive Care Unit Delirium (PRIDE): Retrospective Study. JMIR Med Inform 2021; 9:e23401. [PMID: 34309567 PMCID: PMC8367129 DOI: 10.2196/23401] [Citation(s) in RCA: 11] [Impact Index Per Article: 3.7] [Reference Citation Analysis] [Abstract] [Key Words] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/12/2020] [Revised: 10/10/2020] [Accepted: 06/07/2021] [Indexed: 11/15/2022] Open
Abstract
Background Delirium frequently occurs among patients admitted to the intensive care unit (ICU). There is limited evidence to support interventions to treat or resolve delirium in patients who have already developed delirium. Therefore, the early recognition and prevention of delirium are important in the management of critically ill patients. Objective This study aims to develop and validate a delirium prediction model within 24 hours of admission to the ICU using electronic health record data. The algorithm was named the Prediction of ICU Delirium (PRIDE). Methods This is a retrospective cohort study performed at a tertiary referral hospital with 120 ICU beds. We only included patients who were 18 years or older at the time of admission and who stayed in the medical or surgical ICU. Patients were excluded if they lacked a Confusion Assessment Method for the ICU record from the day of ICU admission or if they had a positive Confusion Assessment Method for the ICU record at the time of ICU admission. The algorithm to predict delirium was developed using patient data from the first 2 years of the study period and validated using patient data from the last 6 months. Random forest (RF), Extreme Gradient Boosting (XGBoost), deep neural network (DNN), and logistic regression (LR) were used. The algorithms were externally validated using MIMIC-III data, and the algorithm with the largest area under the receiver operating characteristics (AUROC) curve in the external data set was named the PRIDE algorithm. Results A total of 37,543 cases were collected. After patient exclusion, 12,409 remained as our study population, of which 3816 (30.8%) patients experienced delirium incidents during the study period. Based on the exclusion criteria, out of the 96,016 ICU admission cases in the MIMIC-III data set, 2061 cases were included, and 272 (13.2%) delirium incidents occurred. The average AUROCs and 95% CIs for internal validation were 0.916 (95% CI 0.916-0.916) for RF, 0.919 (95% CI 0.919-0.919) for XGBoost, 0.881 (95% CI 0.878-0.884) for DNN, and 0.875 (95% CI 0.875-0.875) for LR. Regarding the external validation, the best AUROC were 0.721 (95% CI 0.72-0.721) for RF, 0.697 (95% CI 0.695-0.699) for XGBoost, 0.655 (95% CI 0.654-0.657) for DNN, and 0.631 (95% CI 0.631-0.631) for LR. The Brier score of the RF model is 0.168, indicating that it is well-calibrated. Conclusions A machine learning approach based on electronic health record data can be used to predict delirium within 24 hours of ICU admission. RF, XGBoost, DNN, and LR models were used, and they effectively predicted delirium. However, with the potential to advise ICU physicians and prevent ICU delirium, prospective studies are required to verify the algorithm’s performance.
Collapse
Affiliation(s)
- Sujeong Hur
- Department of Patient Experience Management Part, Samsung Medical Center, Seoul, Republic of Korea.,Department of Digital Health, SAIHST, Sungkyunkwan University, Seoul, Republic of Korea
| | - Ryoung-Eun Ko
- Department of Critical Care Medicine and Medicine, Samsung Medical Center, Sungkyunkwan University School of Medicine, Seoul, Republic of Korea
| | - Junsang Yoo
- Department of Nursing, College of Nursing, Sahmyook University, Seoul, Republic of Korea
| | - Juhyung Ha
- Department of Computer Science, Indiana University, Bloomington, IN, United States
| | - Won Chul Cha
- Department of Digital Health, SAIHST, Sungkyunkwan University, Seoul, Republic of Korea.,Department of Emergency Medicine, Samsung Medical Center, Sungkyunkwan University School of Medicine, Seoul, Republic of Korea.,Digital Innovation Center, Samsung Medical Center, Seoul, Republic of Korea
| | - Chi Ryang Chung
- Department of Critical Care Medicine and Medicine, Samsung Medical Center, Sungkyunkwan University School of Medicine, Seoul, Republic of Korea
| |
Collapse
|
17
|
Ocagli H, Bottigliengo D, Lorenzoni G, Azzolina D, Acar AS, Sorgato S, Stivanello L, Degan M, Gregori D. A Machine Learning Approach for Investigating Delirium as a Multifactorial Syndrome. INTERNATIONAL JOURNAL OF ENVIRONMENTAL RESEARCH AND PUBLIC HEALTH 2021; 18:ijerph18137105. [PMID: 34281037 PMCID: PMC8297073 DOI: 10.3390/ijerph18137105] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 04/26/2021] [Revised: 06/11/2021] [Accepted: 06/14/2021] [Indexed: 12/12/2022]
Abstract
Delirium is a psycho-organic syndrome common in hospitalized patients, especially the elderly, and is associated with poor clinical outcomes. This study aims to identify the predictors that are mostly associated with the risk of delirium episodes using a machine learning technique (MLT). A random forest (RF) algorithm was used to evaluate the association between the subject’s characteristics and the 4AT (the 4 A’s test) score screening tool for delirium. RF algorithm was implemented using information based on demographic characteristics, comorbidities, drugs and procedures. Of the 78 patients enrolled in the study, 49 (63%) were at risk for delirium, 32 (41%) had at least one episode of delirium during the hospitalization (38% in orthopedics and 31% both in internal medicine and in the geriatric ward). The model explained 75.8% of the variability of the 4AT score with a root mean squared error of 3.29. Higher age, the presence of dementia, physical restraint, diabetes and a lower degree are the variables associated with an increase of the 4AT score. Random forest is a valid method for investigating the patients’ characteristics associated with delirium onset also in small case-series. The use of this model may allow for early detection of delirium onset to plan the proper adjustment in healthcare assistance.
Collapse
Affiliation(s)
- Honoria Ocagli
- Unit of Biostatistics, Epidemiology and Public Health, Department of Cardiac, Thoracic, Vascular Sciences and Public Health, University of Padova, Via Loredan 18, 35121 Padova, Italy; (H.O.); (D.B.); (G.L.); (D.A.)
| | - Daniele Bottigliengo
- Unit of Biostatistics, Epidemiology and Public Health, Department of Cardiac, Thoracic, Vascular Sciences and Public Health, University of Padova, Via Loredan 18, 35121 Padova, Italy; (H.O.); (D.B.); (G.L.); (D.A.)
| | - Giulia Lorenzoni
- Unit of Biostatistics, Epidemiology and Public Health, Department of Cardiac, Thoracic, Vascular Sciences and Public Health, University of Padova, Via Loredan 18, 35121 Padova, Italy; (H.O.); (D.B.); (G.L.); (D.A.)
| | - Danila Azzolina
- Unit of Biostatistics, Epidemiology and Public Health, Department of Cardiac, Thoracic, Vascular Sciences and Public Health, University of Padova, Via Loredan 18, 35121 Padova, Italy; (H.O.); (D.B.); (G.L.); (D.A.)
- Department of Medical Science, University of Ferrara, Via Fossato di Mortara 64B, 44121 Ferrara, Italy
| | - Aslihan S. Acar
- Department of Actuarial Sciences, Hacettepe University, Ankara 06800, Turkey;
| | - Silvia Sorgato
- Health Professional Management Service (DPS) of the University Hospital of Padova, 35128 Padova, Italy; (S.S.); (L.S.); (M.D.)
| | - Lucia Stivanello
- Health Professional Management Service (DPS) of the University Hospital of Padova, 35128 Padova, Italy; (S.S.); (L.S.); (M.D.)
| | - Mario Degan
- Health Professional Management Service (DPS) of the University Hospital of Padova, 35128 Padova, Italy; (S.S.); (L.S.); (M.D.)
| | - Dario Gregori
- Unit of Biostatistics, Epidemiology and Public Health, Department of Cardiac, Thoracic, Vascular Sciences and Public Health, University of Padova, Via Loredan 18, 35121 Padova, Italy; (H.O.); (D.B.); (G.L.); (D.A.)
- Correspondence: ; Tel.: +39-049-827-5384
| |
Collapse
|
18
|
Wang J, Zhao D, Lin M, Huang X, Shang X. Post-stroke Anxiety Analysis via Machine Learning Methods. Front Aging Neurosci 2021; 13:657937. [PMID: 34248599 PMCID: PMC8267915 DOI: 10.3389/fnagi.2021.657937] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/24/2021] [Accepted: 05/14/2021] [Indexed: 11/13/2022] Open
Abstract
Post-stroke anxiety (PSA) has caused wide public concern in recent years, and the study on risk factors analysis and prediction is still an open issue. With the deepening of the research, machine learning has been widely applied to various scenarios and make great achievements increasingly, which brings new approaches to this field. In this paper, 395 patients with acute ischemic stroke are collected and evaluated by anxiety scales (i.e., HADS-A, HAMA, and SAS), hence the patients are divided into anxiety group and non-anxiety group. Afterward, the results of demographic data and general laboratory examination between the two groups are compared to identify the risk factors with statistical differences accordingly. Then the factors with statistical differences are incorporated into a multivariate logistic regression to obtain risk factors and protective factors of PSA. Statistical analysis shows great differences in gender, age, serious stroke, hypertension, diabetes mellitus, drinking, and HDL-C level between PSA group and non-anxiety group with HADS-A and HAMA evaluation. Meanwhile, as evaluated by SAS scale, gender, serious stroke, hypertension, diabetes mellitus, drinking, and HDL-C level differ in the PSA group and the non-anxiety group. Multivariate logistic regression analysis of HADS-A, HAMA, and SAS scales suggest that hypertension, diabetes mellitus, drinking, high NIHSS score, and low serum HDL-C level are related to PSA. In other words, gender, age, disability, hypertension, diabetes mellitus, HDL-C, and drinking are closely related to anxiety during the acute stage of ischemic stroke. Hypertension, diabetes mellitus, drinking, and disability increased the risk of PSA, and higher serum HDL-C level decreased the risk of PSA. Several machine learning methods are employed to predict PSA according to HADS-A, HAMA, and SAS scores, respectively. The experimental results indicate that random forest outperforms the competitive methods in PSA prediction, which contributes to early intervention for clinical treatment.
Collapse
Affiliation(s)
- Jirui Wang
- Department of Neurology, The First Affiliated Hospital, China Medical University, Shenyang, China
| | - Defeng Zhao
- The First Clinical Department, China Medical University, Shenyang, China
| | - Meiqing Lin
- Department of Neurology, The First Affiliated Hospital, China Medical University, Shenyang, China
| | - Xinyu Huang
- Software College, Northeastern University, Shenyang, China
| | - Xiuli Shang
- Department of Neurology, The First Affiliated Hospital, China Medical University, Shenyang, China
| |
Collapse
|
19
|
Surodina S, Lam C, Grbich S, Milne-Ives M, van Velthoven M, Meinert E. Machine Learning for Risk Group Identification and User Data Collection in a Herpes Simplex Virus Patient Registry: Algorithm Development and Validation Study. JMIRX MED 2021; 2:e25560. [PMID: 37725536 PMCID: PMC10414389 DOI: 10.2196/25560] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/06/2020] [Revised: 02/04/2021] [Accepted: 03/12/2021] [Indexed: 09/21/2023]
Abstract
BACKGROUND Researching people with herpes simplex virus (HSV) is challenging because of poor data quality, low user engagement, and concerns around stigma and anonymity. OBJECTIVE This project aimed to improve data collection for a real-world HSV registry by identifying predictors of HSV infection and selecting a limited number of relevant questions to ask new registry users to determine their level of HSV infection risk. METHODS The US National Health and Nutrition Examination Survey (NHANES, 2015-2016) database includes the confirmed HSV type 1 and type 2 (HSV-1 and HSV-2, respectively) status of American participants (14-49 years) and a wealth of demographic and health-related data. The questionnaires and data sets from this survey were used to form two data sets: one for HSV-1 and one for HSV-2. These data sets were used to train and test a model that used a random forest algorithm (devised using Python) to minimize the number of anonymous lifestyle-based questions needed to identify risk groups for HSV. RESULTS The model selected a reduced number of questions from the NHANES questionnaire that predicted HSV infection risk with high accuracy scores of 0.91 and 0.96 and high recall scores of 0.88 and 0.98 for the HSV-1 and HSV-2 data sets, respectively. The number of questions was reduced from 150 to an average of 40, depending on age and gender. The model, therefore, provided high predictability of risk of infection with minimal required input. CONCLUSIONS This machine learning algorithm can be used in a real-world evidence registry to collect relevant lifestyle data and identify individuals' levels of risk of HSV infection. A limitation is the absence of real user data and integration with electronic medical records, which would enable model learning and improvement. Future work will explore model adjustments, anonymization options, explicit permissions, and a standardized data schema that meet the General Data Protection Regulation, Health Insurance Portability and Accountability Act, and third-party interface connectivity requirements.
Collapse
Affiliation(s)
- Svitlana Surodina
- Skein Ltd, London, United Kingdom
- Department of Informatics, King's College London, London, United Kingdom
| | - Ching Lam
- Institute of Biomedical Engineering, Department of Engineering Science, University of Oxford, Oxford, United Kingdom
| | | | - Madison Milne-Ives
- Centre for Health Technology, University of Plymouth, Plymouth, United Kingdom
| | - Michelle van Velthoven
- Nuffield Department of Primary Health Sciences, Medical Sciences Division, University of Oxford, Oxford, United Kingdom
| | - Edward Meinert
- Centre for Health Technology, University of Plymouth, Plymouth, United Kingdom
- Department of Primary Care and Public Health, School of Public Health, Imperial College London, London, United Kingdom
| |
Collapse
|
20
|
Jauk S, Kramer D, Großauer B, Rienmüller S, Avian A, Berghold A, Leodolter W, Schulz S. Risk prediction of delirium in hospitalized patients using machine learning: An implementation and prospective evaluation study. J Am Med Inform Assoc 2021; 27:1383-1392. [PMID: 32968811 DOI: 10.1093/jamia/ocaa113] [Citation(s) in RCA: 31] [Impact Index Per Article: 10.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/05/2019] [Revised: 03/11/2020] [Accepted: 05/20/2020] [Indexed: 11/13/2022] Open
Abstract
OBJECTIVE Machine learning models trained on electronic health records have achieved high prognostic accuracy in test datasets, but little is known about their embedding into clinical workflows. We implemented a random forest-based algorithm to identify hospitalized patients at high risk for delirium, and evaluated its performance in a clinical setting. MATERIALS AND METHODS Delirium was predicted at admission and recalculated on the evening of admission. The defined prediction outcome was a delirium coded for the recent hospital stay. During 7 months of prospective evaluation, 5530 predictions were analyzed. In addition, 119 predictions for internal medicine patients were compared with ratings of clinical experts in a blinded and nonblinded setting. RESULTS During clinical application, the algorithm achieved a sensitivity of 74.1% and a specificity of 82.2%. Discrimination on prospective data (area under the receiver-operating characteristic curve = 0.86) was as good as in the test dataset, but calibration was poor. The predictions correlated strongly with delirium risk perceived by experts in the blinded (r = 0.81) and nonblinded (r = 0.62) settings. A major advantage of our setting was the timely prediction without additional data entry. DISCUSSION The implemented machine learning algorithm achieved a stable performance predicting delirium in high agreement with expert ratings, but improvement of calibration is needed. Future research should evaluate the acceptance of implemented machine learning algorithms by health professionals. CONCLUSIONS Our study provides new insights into the implementation process of a machine learning algorithm into a clinical workflow and demonstrates its predictive power for delirium.
Collapse
Affiliation(s)
- Stefanie Jauk
- Department of Information and Process Management, Steiermärkische Krankenanstaltengesellschaft m.b.H. (KAGes), Graz, Austria.,Institute for Medical Informatics, Statistics and Documentation, Medical University of Graz, Graz, Austria
| | - Diether Kramer
- Department of Information and Process Management, Steiermärkische Krankenanstaltengesellschaft m.b.H. (KAGes), Graz, Austria
| | - Birgit Großauer
- Department of Internal Medicine, Steiermärkische Krankenanstaltengesellschaft m.b.H. (KAGes) LKH Graz II, Graz, Austria
| | - Susanne Rienmüller
- Department of Internal Medicine, Steiermärkische Krankenanstaltengesellschaft m.b.H. (KAGes) LKH Graz II, Graz, Austria
| | - Alexander Avian
- Institute for Medical Informatics, Statistics and Documentation, Medical University of Graz, Graz, Austria
| | - Andrea Berghold
- Institute for Medical Informatics, Statistics and Documentation, Medical University of Graz, Graz, Austria
| | - Werner Leodolter
- Department of Information and Process Management, Steiermärkische Krankenanstaltengesellschaft m.b.H. (KAGes), Graz, Austria
| | - Stefan Schulz
- Institute for Medical Informatics, Statistics and Documentation, Medical University of Graz, Graz, Austria
| |
Collapse
|
21
|
Coombes CE, Coombes KR, Fareed N. A novel model to label delirium in an intensive care unit from clinician actions. BMC Med Inform Decis Mak 2021; 21:97. [PMID: 33750375 PMCID: PMC7941123 DOI: 10.1186/s12911-021-01461-6] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/02/2020] [Accepted: 03/02/2021] [Indexed: 11/10/2022] Open
Abstract
BACKGROUND In the intensive care unit (ICU), delirium is a common, acute, confusional state associated with high risk for short- and long-term morbidity and mortality. Machine learning (ML) has promise to address research priorities and improve delirium outcomes. However, due to clinical and billing conventions, delirium is often inconsistently or incompletely labeled in electronic health record (EHR) datasets. Here, we identify clinical actions abstracted from clinical guidelines in electronic health records (EHR) data that indicate risk of delirium among intensive care unit (ICU) patients. We develop a novel prediction model to label patients with delirium based on a large data set and assess model performance. METHODS EHR data on 48,451 admissions from 2001 to 2012, available through Medical Information Mart for Intensive Care-III database (MIMIC-III), was used to identify features to develop our prediction models. Five binary ML classification models (Logistic Regression; Classification and Regression Trees; Random Forests; Naïve Bayes; and Support Vector Machines) were fit and ranked by Area Under the Curve (AUC) scores. We compared our best model with two models previously proposed in the literature for goodness of fit, precision, and through biological validation. RESULTS Our best performing model with threshold reclassification for predicting delirium was based on a multiple logistic regression using the 31 clinical actions (AUC 0.83). Our model out performed other proposed models by biological validation on clinically meaningful, delirium-associated outcomes. CONCLUSIONS Hurdles in identifying accurate labels in large-scale datasets limit clinical applications of ML in delirium. We developed a novel labeling model for delirium in the ICU using a large, public data set. By using guideline-directed clinical actions independent from risk factors, treatments, and outcomes as model predictors, our classifier could be used as a delirium label for future clinically targeted models.
Collapse
Affiliation(s)
- Caitlin E Coombes
- College of Medicine, The Ohio State University, Columbus, OH, 43210, USA
| | - Kevin R Coombes
- Department of Biomedical Informatics, The Ohio State University College of Medicine, 460 Medical Center Dr., 512 Institute of Behavioral Medicine Research, Columbus, OH, 43210, USA
| | - Naleef Fareed
- Department of Biomedical Informatics, The Ohio State University College of Medicine, 460 Medical Center Dr., 512 Institute of Behavioral Medicine Research, Columbus, OH, 43210, USA.
- Center for the Advancement of Team Science, Analytics, and Systems Thinking, College of Medicine, The Ohio State University, Columbus, OH, 43210, USA.
| |
Collapse
|
22
|
Racine AM, Tommet D, D'Aquila ML, Fong TG, Gou Y, Tabloski PA, Metzger ED, Hshieh TT, Schmitt EM, Vasunilashorn SM, Kunze L, Vlassakov K, Abdeen A, Lange J, Earp B, Dickerson BC, Marcantonio ER, Steingrimsson J, Travison TG, Inouye SK, Jones RN. Machine Learning to Develop and Internally Validate a Predictive Model for Post-operative Delirium in a Prospective, Observational Clinical Cohort Study of Older Surgical Patients. J Gen Intern Med 2021; 36:265-273. [PMID: 33078300 PMCID: PMC7878663 DOI: 10.1007/s11606-020-06238-7] [Citation(s) in RCA: 22] [Impact Index Per Article: 7.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 06/26/2019] [Accepted: 09/11/2020] [Indexed: 12/13/2022]
Abstract
BACKGROUND Our objective was to assess the performance of machine learning methods to predict post-operative delirium using a prospective clinical cohort. METHODS We analyzed data from an observational cohort study of 560 older adults (≥ 70 years) without dementia undergoing major elective non-cardiac surgery. Post-operative delirium was determined by the Confusion Assessment Method supplemented by a medical chart review (N = 134, 24%). Five machine learning algorithms and a standard stepwise logistic regression model were developed in a training sample (80% of participants) and evaluated in the remaining hold-out testing sample. We evaluated three overlapping feature sets, restricted to variables that are readily available or minimally burdensome to collect in clinical settings, including interview and medical record data. A large feature set included 71 potential predictors. A smaller set of 18 features was selected by an expert panel using a consensus process, and this smaller feature set was considered with and without a measure of pre-operative mental status. RESULTS The area under the receiver operating characteristic curve (AUC) was higher in the large feature set conditions (range of AUC, 0.62-0.71 across algorithms) versus the selected feature set conditions (AUC range, 0.53-0.57). The restricted feature set with mental status had intermediate AUC values (range, 0.53-0.68). In the full feature set condition, algorithms such as gradient boosting, cross-validated logistic regression, and neural network (AUC = 0.71, 95% CI 0.58-0.83) were comparable with a model developed using traditional stepwise logistic regression (AUC = 0.69, 95% CI 0.57-0.82). Calibration for all models and feature sets was poor. CONCLUSIONS We developed machine learning prediction models for post-operative delirium that performed better than chance and are comparable with traditional stepwise logistic regression. Delirium proved to be a phenotype that was difficult to predict with appreciable accuracy.
Collapse
Affiliation(s)
- Annie M Racine
- Aging Brain Center, Institute for Aging Research, Boston, MA, USA
- Harvard Medical School, Boston, MA, USA
| | - Douglas Tommet
- Department of Psychiatry & Human Behavior, and Neurology, Brown University Warren Alpert Medical School, Providence, RI, USA
| | | | - Tamara G Fong
- Aging Brain Center, Institute for Aging Research, Boston, MA, USA
- Harvard Medical School, Boston, MA, USA
- Department of Neurology, Beth Israel Deaconess Medical Center, Boston, MA, USA
| | - Yun Gou
- Aging Brain Center, Institute for Aging Research, Boston, MA, USA
| | | | - Eran D Metzger
- Harvard Medical School, Boston, MA, USA
- Department of Psychiatry, Beth Israel Deaconess Medical Center, Boston, MA, USA
| | - Tammy T Hshieh
- Harvard Medical School, Boston, MA, USA
- Department of Medicine, Beth Israel Deaconess Medical Center, Boston, MA, USA
| | - Eva M Schmitt
- Aging Brain Center, Institute for Aging Research, Boston, MA, USA
| | - Sarinnapha M Vasunilashorn
- Harvard Medical School, Boston, MA, USA
- Department of Medicine, Beth Israel Deaconess Medical Center, Boston, MA, USA
| | - Lisa Kunze
- Harvard Medical School, Boston, MA, USA
- Department of Anesthesia, Beth Israel Deaconess Medical Center, Boston, MA, USA
| | - Kamen Vlassakov
- Harvard Medical School, Boston, MA, USA
- William F Connell School of Nursing at Boston College, Boston, MA, USA
| | - Ayesha Abdeen
- Harvard Medical School, Boston, MA, USA
- Department of Orthopedic Surgery, Beth Israel Deaconess Medical Center, Boston, MA, USA
| | - Jeffrey Lange
- Harvard Medical School, Boston, MA, USA
- Department of Orthopedic Surgery, Brigham and Women's Hospital, Boston, MA, USA
| | - Brandon Earp
- Harvard Medical School, Boston, MA, USA
- Department of Orthopedics, Brigham and Women's Faulkner Hospital, Boston, MA, USA
| | - Bradford C Dickerson
- Department of Neurology and Massachusetts Alzheimer's Disease Research Center, Massachusetts General Hospital, Boston, MA, USA
| | - Edward R Marcantonio
- Aging Brain Center, Institute for Aging Research, Boston, MA, USA
- Harvard Medical School, Boston, MA, USA
- Department of Medicine, Beth Israel Deaconess Medical Center, Boston, MA, USA
| | | | - Thomas G Travison
- Aging Brain Center, Institute for Aging Research, Boston, MA, USA
- Harvard Medical School, Boston, MA, USA
| | - Sharon K Inouye
- Aging Brain Center, Institute for Aging Research, Boston, MA, USA
- Harvard Medical School, Boston, MA, USA
- Department of Medicine, Beth Israel Deaconess Medical Center, Boston, MA, USA
| | - Richard N Jones
- Department of Psychiatry & Human Behavior, and Neurology, Brown University Warren Alpert Medical School, Providence, RI, USA.
| |
Collapse
|
23
|
Predicting postoperative delirium after microvascular decompression surgery with machine learning. J Clin Anesth 2020; 66:109896. [DOI: 10.1016/j.jclinane.2020.109896] [Citation(s) in RCA: 25] [Impact Index Per Article: 6.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/22/2019] [Revised: 04/20/2020] [Accepted: 05/19/2020] [Indexed: 12/24/2022]
|
24
|
Sufriyana H, Wu YW, Su ECY. Artificial intelligence-assisted prediction of preeclampsia: Development and external validation of a nationwide health insurance dataset of the BPJS Kesehatan in Indonesia. EBioMedicine 2020; 54:102710. [PMID: 32283530 PMCID: PMC7152721 DOI: 10.1016/j.ebiom.2020.102710] [Citation(s) in RCA: 25] [Impact Index Per Article: 6.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/29/2019] [Revised: 02/24/2020] [Accepted: 02/24/2020] [Indexed: 02/06/2023] Open
Abstract
Background We developed and validated an artificial intelligence (AI)-assisted prediction of preeclampsia applied to a nationwide health insurance dataset in Indonesia. Methods The BPJS Kesehatan dataset have been preprocessed using a nested case-control design into preeclampsia/eclampsia (n = 3318) and normotensive pregnant women (n = 19,883) from all women with one pregnancy. The dataset provided 95 features consisting of demographic variables and medical histories started from 24 months to event and ended by delivery as the event. Six algorithms were compared by area under the receiver operating characteristics curve (AUROC) with a subgroup analysis by time to the event. We compared our model to similar prediction models from systematically reviewed studies. In addition, we conducted a text mining analysis based on natural language processing techniques to interpret our modeling results. Findings The best model consisted of 17 predictors extracted by a random forest algorithm. Nine∼12 months to the event was the period that had the best AUROC in external validation by either geographical (0.88, 95% confidence interval (CI) 0.88–0.89) or temporal split (0.86, 95% CI 0.85–0.86). We compared this model to prediction models in seven studies from 869 records in PUBMED, EMBASE, and SCOPUS. This model outperformed the previous models in terms of the precision, sensitivity, and specificity in all validation sets. Interpretation Our low-cost model improved preliminary prediction to decide pregnant women that will be predicted by the models with high specificity and advanced predictors. Funding This work was supported by grant no. MOST108-2221-E-038-018 from the Ministry of Science and Technology of Taiwan.
Collapse
Affiliation(s)
- Herdiantri Sufriyana
- Graduate Institute of Biomedical Informatics, College of Medical Science and Technology, Taipei Medical University, Taipei 11031, Taiwan; Department of Medical Physiology, College of Medicine, University of Nahdlatul Ulama Surabaya, Surabaya 60237, Indonesia.
| | - Yu-Wei Wu
- Department of Medical Physiology, College of Medicine, University of Nahdlatul Ulama Surabaya, Surabaya 60237, Indonesia; Graduate Institute of Biomedical Informatics, College of Medical Science and Technology, Taipei Medical University, Taipei 11031, Taiwan; Clinical Big Data Research Center, Taipei Medical University Hospital, Taipei 11031, Taiwan.
| | - Emily Chia-Yu Su
- Graduate Institute of Biomedical Informatics, College of Medical Science and Technology, Taipei Medical University, Taipei 11031, Taiwan; Clinical Big Data Research Center, Taipei Medical University Hospital, Taipei 11031, Taiwan; Research Center for Artificial Intelligence in Medicine, Taipei Medical University, Taipei 11031, Taiwan.
| |
Collapse
|
25
|
Mufti HN, Hirsch GM, Abidi SR, Abidi SSR. Exploiting Machine Learning Algorithms and Methods for the Prediction of Agitated Delirium After Cardiac Surgery: Models Development and Validation Study. JMIR Med Inform 2019; 7:e14993. [PMID: 31558433 PMCID: PMC6913743 DOI: 10.2196/14993] [Citation(s) in RCA: 32] [Impact Index Per Article: 6.4] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/11/2019] [Revised: 09/02/2019] [Accepted: 09/24/2019] [Indexed: 12/28/2022] Open
Abstract
Background Delirium is a temporary mental disorder that occasionally affects patients undergoing surgery, especially cardiac surgery. It is strongly associated with major adverse events, which in turn leads to increased cost and poor outcomes (eg, need for nursing home due to cognitive impairment, stroke, and death). The ability to foresee patients at risk of delirium will guide the timely initiation of multimodal preventive interventions, which will aid in reducing the burden and negative consequences associated with delirium. Several studies have focused on the prediction of delirium. However, the number of studies in cardiac surgical patients that have used machine learning methods is very limited. Objective This study aimed to explore the application of several machine learning predictive models that can pre-emptively predict delirium in patients undergoing cardiac surgery and compare their performance. Methods We investigated a number of machine learning methods to develop models that can predict delirium after cardiac surgery. A clinical dataset comprising over 5000 actual patients who underwent cardiac surgery in a single center was used to develop the models using logistic regression, artificial neural networks (ANN), support vector machines (SVM), Bayesian belief networks (BBN), naïve Bayesian, random forest, and decision trees. Results Only 507 out of 5584 patients (11.4%) developed delirium. We addressed the underlying class imbalance, using random undersampling, in the training dataset. The final prediction performance was validated on a separate test dataset. Owing to the target class imbalance, several measures were used to evaluate algorithm’s performance for the delirium class on the test dataset. Out of the selected algorithms, the SVM algorithm had the best F1 score for positive cases, kappa, and positive predictive value (40.2%, 29.3%, and 29.7%, respectively) with a P=.01, .03, .02, respectively. The ANN had the best receiver-operator area-under the curve (78.2%; P=.03). The BBN had the best precision-recall area-under the curve for detecting positive cases (30.4%; P=.03). Conclusions Although delirium is inherently complex, preventive measures to mitigate its negative effect can be applied proactively if patients at risk are prospectively identified. Our results highlight 2 important points: (1) addressing class imbalance on the training dataset will augment machine learning model’s performance in identifying patients likely to develop postoperative delirium, and (2) as the prediction of postoperative delirium is difficult because it is multifactorial and has complex pathophysiology, applying machine learning methods (complex or simple) may improve the prediction by revealing hidden patterns, which will lead to cost reduction by prevention of complications and will optimize patients’ outcomes.
Collapse
Affiliation(s)
- Hani Nabeel Mufti
- Division of Cardiac Surgery, Department of Cardiac Sciences, King Faisal Cardiac Center, King Abdulaziz Medical City, Ministry of National Guard Health Affairs - Western Region, Jeddah, Saudi Arabia.,College of Medicine-Jeddah, King Saud bin Abdulaziz University for Health, Ministry of National Guard Health Affairs, Jeddah, Saudi Arabia.,King Abdullah International Medical Research Center, Jeddah, Saudi Arabia
| | | | - Samina Raza Abidi
- Department of Community Health and Epidemiology, Faculty of Medicine, Dalhousie University, Halifax, NS, Canada
| | - Syed Sibte Raza Abidi
- kNowledge Intensive Computing for Healthcare Enterprise Research Group, Faculty of Computer Science, Dalhousie University, Halifax, NS, Canada
| |
Collapse
|