Prediction of Incident Delirium Using a Random Forest classifier

Novel opportunities for clinical pharmacy research: development of a machine learning model to identify medication related causes of delirium in different patient groups

The advent of artificial intelligence (AI) technologies has taken the world of science by storm in 2023. The opportunities of this easy to access technology for clinical pharmacy research are yet to be fully understood. The development of a custom-made large language model (LLM) (DELSTAR) trained on a wide range of internationally recognised scientific publication databases, pharmacovigilance sites and international product characteristics to help identify and summarise medication related information on delirium, as a proof-of-concept model, identified new facilitators and barriers for robust clinical pharmacy practice research. This technology holds great promise for the development of much more comprehensive prescribing guidelines, practice support applications for clinical pharmacy, increased patient and prescribing safety and resultant implications for healthcare costs. The challenge will be to ensure its methodologically robust use and the detailed and transparent verification of its information accuracy.

A machine learning-based prediction model for postoperative delirium in cardiac valve surgery using electronic health records

BACKGROUND

Previous models for predicting delirium after cardiac surgery remained inadequate. This study aimed to develop and validate a machine learning-based prediction model for postoperative delirium (POD) in cardiac valve surgery patients.

METHODS

The electronic medical information of the cardiac surgical intensive care unit (CSICU) was extracted from a tertiary and major referral hospital in southern China over 1 year, from June 2019 to June 2020. A total of 507 patients admitted to the CSICU after cardiac valve surgery were included in this study. Seven classical machine learning algorithms (Random Forest Classifier, Logistic Regression, Support Vector Machine Classifier, K-nearest Neighbors Classifier, Gaussian Naive Bayes, Gradient Boosting Decision Tree, and Perceptron.) were used to develop delirium prediction models under full (q = 31) and selected (q = 19) feature sets, respectively.

RESULT

The Random Forest classifier performs exceptionally well in both feature datasets, with an Area Under the Curve (AUC) of 0.92 for the full feature dataset and an AUC of 0.86 for the selected feature dataset. Additionally, it achieves a relatively lower Expected Calibration Error (ECE) and the highest Average Precision (AP), with an AP of 0.80 for the full feature dataset and an AP of 0.73 for the selected feature dataset. To further evaluate the best-performing Random Forest classifier, SHAP (Shapley Additive Explanations) was used, and the importance matrix plot, scatter plots, and summary plots were generated.

CONCLUSIONS

We established machine learning-based prediction models to predict POD in patients undergoing cardiac valve surgery. The random forest model has the best predictive performance in prediction and can help improve the prognosis of patients with POD.

Use of Electronic Medical Records (EMR) in Gerontology: Benefits, Considerations and a Promising Future

Electronic medical records (EMRs) have many benefits in clinical research in gerontology, enabling data analysis, development of prognostic tools and disease risk prediction. EMRs also offer a range of advantages in clinical practice, such as comprehensive medical records, streamlined communication with healthcare providers, remote data access, and rapid retrieval of test results, ultimately leading to increased efficiency, enhanced patient safety, and improved quality of care in gerontology, which includes benefits like reduced medication use and better patient history taking and physical examination assessments. The use of artificial intelligence (AI) and machine learning (ML) approaches on EMRs can further improve disease diagnosis, symptom classification, and support clinical decision-making. However, there are also challenges related to data quality, data entry errors, as well as the ethics and safety of using AI in healthcare. This article discusses the future of EMRs in gerontology and the application of AI and ML in clinical research. Ethical and legal issues surrounding data sharing and the need for healthcare professionals to critically evaluate and integrate these technologies are also emphasized. The article concludes by discussing the challenges related to the use of EMRs in research as well as in their primary intended use, the daily clinical practice.

Identifying encephalopathy in patients admitted to an intensive care unit: Going beyond structured information using natural language processing

Background

Encephalopathy is a severe co-morbid condition in critically ill patients that includes different clinical constellation of neurological symptoms. However, even for the most recognised form, delirium, this medical condition is rarely recorded in structured fields of electronic health records precluding large and unbiased retrospective studies. We aimed to identify patients with encephalopathy using a machine learning-based approach over clinical notes in electronic health records.

Methods

We used a list of ICD-9 codes and clinical concepts related to encephalopathy to define a cohort of patients from the MIMIC-III dataset. Clinical notes were annotated with MedCAT and vectorized with a bag-of-word approach or word embedding using clinical concepts normalised to standard nomenclatures as features. Machine learning algorithms (support vector machines and random forest) trained with clinical notes from patients who had a diagnosis of encephalopathy (defined by ICD-9 codes) were used to classify patients with clinical concepts related to encephalopathy in their clinical notes but without any ICD-9 relevant code. A random selection of 50 patients were reviewed by a clinical expert for model validation.

Results

Among 46,520 different patients, 7.5% had encephalopathy related ICD-9 codes in all their admissions (group 1, definite encephalopathy), 45% clinical concepts related to encephalopathy only in their clinical notes (group 2, possible encephalopathy) and 38% did not have encephalopathy related concepts neither in structured nor in clinical notes (group 3, non-encephalopathy). Length of stay, mortality rate or number of co-morbid conditions were higher in groups 1 and 2 compared to group 3. The best model to classify patients from group 2 as patients with encephalopathy (SVM using embeddings) had F1 of 85% and predicted 31% patients from group 2 as having encephalopathy with a probability >90%. Validation on new cases found a precision ranging from 92% to 98% depending on the criteria considered.

Conclusions

Natural language processing techniques can leverage relevant clinical information that might help to identify patients with under-recognised clinical disorders such as encephalopathy. In the MIMIC dataset, this approach identifies with high probability thousands of patients that did not have a formal diagnosis in the structured information of the EHR.

Decreased default mode network functional connectivity with visual processing regions as potential biomarkers for delayed neurocognitive recovery: A resting-state fMRI study and machine-learning analysis

Objectives

The abnormal functional connectivity (FC) pattern of default mode network (DMN) may be key markers for early identification of various cognitive disorders. However, the whole-brain FC changes of DMN in delayed neurocognitive recovery (DNR) are still unclear. Our study was aimed at exploring the whole-brain FC patterns of all regions in DMN and the potential features as biomarkers for the prediction of DNR using machine-learning algorithms.

Methods

Resting-state functional magnetic resonance imaging (fMRI) was conducted before surgery on 74 patients undergoing non-cardiac surgery. Seed-based whole-brain FC with 18 core regions located in the DMN was performed, and FC features that were statistically different between the DNR and non-DNR patients after false discovery correction were extracted. Afterward, based on the extracted FC features, machine-learning algorithms such as support vector machine, logistic regression, decision tree, and random forest were established to recognize DNR. The machine learning experiment procedure mainly included three following steps: feature standardization, parameter adjustment, and performance comparison. Finally, independent testing was conducted to validate the established prediction model. The algorithm performance was evaluated by a permutation test.

Results

We found significantly decreased DMN connectivity with the brain regions involved in visual processing in DNR patients than in non-DNR patients. The best result was obtained from the random forest algorithm based on the 20 decision trees (estimators). The random forest model achieved the accuracy, sensitivity, and specificity of 84.0, 63.1, and 89.5%, respectively. The area under the receiver operating characteristic curve of the classifier reached 86.4%. The feature that contributed the most to the random forest model was the FC between the left retrosplenial cortex/posterior cingulate cortex and left precuneus.

Conclusion

The decreased FC of DMN with regions involved in visual processing might be effective markers for the prediction of DNR and could provide new insights into the neural mechanisms of DNR.

Clinical Trial Registration

: Chinese Clinical Trial Registry, ChiCTR-DCD-15006096.

Boosting Delirium Identification Accuracy With Sentiment-Based Natural Language Processing: Mixed Methods Study

BACKGROUND

Delirium is an acute neurocognitive disorder that affects up to half of older hospitalized medical patients and can lead to dementia, longer hospital stays, increased health costs, and death. Although delirium can be prevented and treated, it is difficult to identify and predict.

OBJECTIVE

This study aimed to improve machine learning models that retrospectively identify the presence of delirium during hospital stays (eg, to measure the effectiveness of delirium prevention interventions) by using the natural language processing (NLP) technique of sentiment analysis (in this case a feature that identifies sentiment toward, or away from, a delirium diagnosis).

METHODS

Using data from the General Medicine Inpatient Initiative, a Canadian hospital data and analytics network, a detailed manual review of medical records was conducted from nearly 4000 admissions at 6 Toronto area hospitals. Furthermore, 25.74% (994/3862) of the eligible hospital admissions were labeled as having delirium. Using the data set collected from this study, we developed machine learning models with, and without, the benefit of NLP methods applied to diagnostic imaging reports, and we asked the question "can NLP improve machine learning identification of delirium?"

RESULTS

Among the eligible 3862 hospital admissions, 994 (25.74%) admissions were labeled as having delirium. Identification and calibration of the models were satisfactory. The accuracy and area under the receiver operating characteristic curve of the main model with NLP in the independent testing data set were 0.807 and 0.930, respectively. The accuracy and area under the receiver operating characteristic curve of the main model without NLP in the independent testing data set were 0.811 and 0.869, respectively. Model performance was also found to be stable over the 5-year period used in the experiment, with identification for a likely future holdout test set being no worse than identification for retrospective holdout test sets.

CONCLUSIONS

Our machine learning model that included NLP (ie, sentiment analysis in medical image description text mining) produced valid identification of delirium with the sentiment analysis, providing significant additional benefit over the model without NLP.

A Transfer Learning Approach to Correct the Temporal Performance Drift of Clinical Prediction Models: Retrospective Cohort Study

Background

Clinical prediction models suffer from performance drift as the patient population shifts over time. There is a great need for model updating approaches or modeling frameworks that can effectively use the old and new data.

Objective

Based on the paradigm of transfer learning, we aimed to develop a novel modeling framework that transfers old knowledge to the new environment for prediction tasks, and contributes to performance drift correction.

Methods

The proposed predictive modeling framework maintains a logistic regression–based stacking ensemble of 2 gradient boosting machine (GBM) models representing old and new knowledge learned from old and new data, respectively (referred to as transfer learning gradient boosting machine [TransferGBM]). The ensemble learning procedure can dynamically balance the old and new knowledge. Using 2010-2017 electronic health record data on a retrospective cohort of 141,696 patients, we validated TransferGBM for hospital-acquired acute kidney injury prediction.

Results

The baseline models (ie, transported models) that were trained on 2010 and 2011 data showed significant performance drift in the temporal validation with 2012-2017 data. Refitting these models using updated samples resulted in performance gains in nearly all cases. The proposed TransferGBM model succeeded in achieving uniformly better performance than the refitted models.

Conclusions

Under the scenario of population shift, incorporating new knowledge while preserving old knowledge is essential for maintaining stable performance. Transfer learning combined with stacking ensemble learning can help achieve a balance of old and new knowledge in a flexible and adaptive way, even in the case of insufficient new data.

A machine-learning model to predict postoperative delirium following knee arthroplasty using electronic health records

BACKGROUND

Postoperative delirium is a challenging complication due to its adverse outcome such as long hospital stay. The aims of this study were: 1) to identify preoperative risk factors of postoperative delirium following knee arthroplasty, and 2) to develop a machine-learning prediction model.

METHOD

A total of 3,980 patients from two hospitals were included in this study. The model was developed and trained with 1,931 patients from one hospital and externally validated with 2,049 patients from another hospital. Twenty preoperative variables were collected using electronic hospital records. Feature selection was conducted using the sequential feature selection (SFS). Extreme Gradient Boosting algorithm (XGBoost) model as a machine-learning classifier was applied to predict delirium. A tenfold-stratified area under the curve (AUC) served as the metric for variable selection and internal validation.

RESULTS

The incidence rate of delirium was 4.9% (n = 196). The following seven key predictors of postoperative delirium were selected: age, serum albumin, number of hypnotics and sedatives drugs taken preoperatively, total number of drugs (any kinds of oral medication) taken preoperatively, neurologic disorders, depression, and fall-down risk (all p < 0.05). The predictive performance of our model was good for the developmental cohort (AUC: 0.80, 95% CI: 0.77-0.84). It was also good for the external validation cohort (AUC: 0.82, 95% CI: 0.80-0.83). Our model can be accessed at https://safetka.connecteve.com .

CONCLUSIONS

A web-based predictive model for delirium after knee arthroplasty was developed using a machine-learning algorithm featuring seven preoperative variables. This model can be used only with information that can be obtained from pre-operative electronic hospital records. Thus, this model could be used to predict delirium before surgery and may assist physician's effort on delirium prevention.

Prediction of the Values of CRP, eGFR, and Hemoglobin in the Follow-Up of Renal Cell Carcinoma Patients after (Cryo)Surgery Using Machine Learning Algorithms

Background

Artificial intelligence can support clinical decisions by predictive modeling. Using patient-specific characteristics, models may predict the course of clinical parameters, thus guiding monitoring approaches for the individual patient. Here, we present prediction models for inflammation and for the course of renal function and hemoglobin (Hb) in renal cell carcinoma patients after (cryo)surgery.

Methods

Using random forest machine learning in a longitudinal value-based healthcare data set (n = 86) of renal cell carcinoma patients, prediction models were established and optimized using random and grid searches. Data were split into a training and test set in a 70:30 ratio. Inflammation was predicted for a single timepoint, whereas for renal function estimated glomerular filtration rate (eGFR) and Hb time course prediction was performed.

Results

Whereas the last Hb and eGFR values before (cryo)surgery were the main basis for the course of Hb and renal function, age and several time frame features also contributed significantly. For eGFR, the type of (cryo)surgery was also a main predicting feature, and for Hb, tumor location, and body mass index were important predictors. With regard to prediction of inflammation no feature was markedly prominent. Inflammation prediction was based on a combination of patient characteristics, physiological parameters, and time frame features.

Conclusions

This study provided interesting insights into factors influencing complications and recovery in individual renal cell carcinoma patients. The established prediction models provide the basis for development of clinical decision support tools for selection and timing of laboratory analyses after (cryo)surgery, thus contributing to quality and efficiency of care.

Predicting Delayed Neurocognitive Recovery After Non-cardiac Surgery Using Resting-State Brain Network Patterns Combined With Machine Learning

Delayed neurocognitive recovery (DNR) is a common subtype of postoperative neurocognitive disorders. An objective approach for identifying subjects at high risk of DNR is yet lacking. The present study aimed to predict DNR using the machine learning method based on multiple cognitive-related brain network features. A total of 74 elderly patients (≥ 60-years-old) undergoing non-cardiac surgery were subjected to resting-state functional magnetic resonance imaging (rs-fMRI) before the surgery. Seed-based whole-brain functional connectivity (FC) was analyzed with 18 regions of interest (ROIs) located in the default mode network (DMN), limbic network, salience network (SN), and central executive network (CEN). Multiple machine learning models (support vector machine, decision tree, and random forest) were constructed to recognize the DNR based on FC network features. The experiment has three parts, including performance comparison, feature screening, and parameter adjustment. Then, the model with the best predictive efficacy for DNR was identified. Finally, independent testing was conducted to validate the established predictive model. Compared to the non-DNR group, the DNR group exhibited aberrant whole-brain FC in seven ROIs, including the right posterior cingulate cortex, right medial prefrontal cortex, and left lateral parietal cortex in the DMN, the right insula in the SN, the left anterior prefrontal cortex in the CEN, and the left ventral hippocampus and left amygdala in the limbic network. The machine learning experimental results identified a random forest model combined with FC features of DMN and CEN as the best prediction model. The area under the curve was 0.958 (accuracy = 0.935, precision = 0.899, recall = 0.900, F1 = 0.890) on the test set. Thus, the current study indicated that the random forest machine learning model based on rs-FC features of DMN and CEN predicts the DNR following non-cardiac surgery, which could be beneficial to the early prevention of DNR. Clinical Trial Registration: The study was registered at the Chinese Clinical Trial Registry (Identification number: ChiCTR-DCD-15006096).

Machine Learning to Identify Psychomotor Behaviors of Delirium for Patients in Long-Term Care Facility

This study aimed to develop accurate and explainable machine learning models for three psychomotor behaviors of delirium for hospitalized adult patients. A prospective pilot study was conducted with 33 participants admitted to a long-term care facility between August 10 and 25, 2020. During the pilot study, we collected 560 cases that included 33 clinical variables and the survey items from the short confusion assessment method (S-CAM), and developed a mobile-based application. Multiple machine learning algorithms, including four rule-mining algorithms (C4.5, CBA, MCAR, and LEM2) and four other statistical learning algorithms (LR, ANNs, SVMs with three kernel functions, and random forest), were validated by paired Wilcoxon signed-rank tests on both macro-averaged F1 and weighted average F1-measures during the 10-times stratified 2-fold cross-validation. The LEM2 algorithm achieved the best prediction performance (macro-averaged F1-measure of 49.35%; weighted average F1-measure of 96.55%), correctly identifying adult patients at delirium risk. In the pairwise comparison between predictive powers observed from independent models, the LEM2 model showed a medium or large effect size between 0.4925 and 0.8766 when compared with LR, ANN, SVM with RBF, and MCAR models. We have confirmed that acute consciousness in S-CAM assessment is closely associated with different predictors for screening three psychomotor behaviors of delirium: 1) education level, dementia type or its level, sleep disorder, dehydration, and infection in mixed-type delirium; 2) gender, education level, dementia type, dehydration, bedsores, and foley catheter in hyperactive delirium; and 3) pain, sleep disorder, and haloperidol use in hypoactive delirium.

Developing a machine learning model to identify delirium risk in geriatric internal medicine inpatients

PURPOSE

To develop a machine learning model that predicts delirium risk in geriatric internal medicine inpatients.

METHODS

A prospective cohort study of internal medicine wards in a tertiary care hospital in China. Blinded observers assessed delirium using the Confusion Assessment Method (CAM). The data set was randomly divided into a training set (70%) and a test set (30%). The model was trained on the training set using the decision tree and the five-fold cross-validation, and then the model performance was evaluated on the test set. Under-sampling was used to address the class imbalance. The discriminatory power of the model was measured by the area under the receiver operating characteristic curve (AUC) and F1 score. The data set comprised 740 patients from March 2016 to January 2017.

RESULTS

The training set included 518 patients; the median (IQR) age was 84 (79-87) years; 364 (70.3%) were men; 71 (13.7%) with delirium. The test set included 222 patients; the median (IQR) age was 84.5 (79-87) years; 163 (73.4%) were men; 30 (13.5%) with delirium. In total, the data set included 740 hospital admissions with a median (IQR) age of 84 (79-87) years, 527 (71.2%) were men, and 101 (13.6%) with delirium. From 32 potential predictors, we included five variables in the predictive model: depression, cognitive impairment, types of drugs, nutritional status, and activity of daily life (ADL). The mean AUC on the training set was 0.967, the AUC and F1 score on the test set was 0.950 and 0.810, respectively. The model achieved 93.3% sensitivity, 94.3% specificity, 71.8% positive predictive value, 98.9% negative predictive value, and 94.1% accuracy on the test set.

CONCLUSION

This machine learning model may allow more precise targeting of delirium prevention and could support clinical decision making in geriatric internal medicine wards.

Predicting brain function status changes in critically ill patients via Machine learning

OBJECTIVE

In intensive care units (ICUs), a patient's brain function status can shift from a state of acute brain dysfunction (ABD) to one that is ABD-free and vice versa, which is challenging to forecast and, in turn, hampers the allocation of hospital resources. We aim to develop a machine learning model to predict next-day brain function status changes.

MATERIALS AND METHODS

Using multicenter prospective adult cohorts involving medical and surgical ICU patients from 2 civilian and 3 Veteran Affairs hospitals, we trained and externally validated a light gradient boosting machine to predict brain function status changes. We compared the performances of the boosting model against state-of-the-art models-an ABD predictive model and its variants. We applied Shapley additive explanations to identify influential factors to develop a compact model.

RESULTS

There were 1026 critically ill patients without evidence of prior major dementia, or structural brain diseases, from whom 12 295 daily transitions (ABD: 5847 days; ABD-free: 6448 days) were observed. The boosting model achieved an area under the receiver-operating characteristic curve (AUROC) of 0.824 (95% confidence interval [CI], 0.821-0.827), compared with the state-of-the-art models of 0.697 (95% CI, 0.693-0.701) with P < .001. Using 13 identified top influential factors, the compact model achieved 99.4% of the boosting model on AUROC. The boosting and the compact models demonstrated high generalizability in external validation by achieving an AUROC of 0.812 (95% CI, 0.812-0.813).

CONCLUSION

The inputs of the compact model are based on several simple questions that clinicians can quickly answer in practice, which demonstrates the model has direct prospective deployment potential into clinical practice, aiding in critical hospital resource allocation.

Prediction of delirium using data mining: A systematic review

Delirium remains a significant cause of morbidity, mortality and economic burden to society. "Big data" refers to data of significantly large volume, obtained from a variety of resources, which is created and processed at high velocity. We conducted a systematic review and meta-analysis exploring whether big data could predict the incidence of delirium of patients in the inpatient setting. Medline, Embase, the Cochrane Library, Web of Science, CINAHL, clinicaltrials.gov, who.int and IEEE Xplore were searched using MeSH terms "big data", "data mining", "delirium" and "confusion" up to 30th September 2019. We included both randomised and observational studies. The primary outcome of interest was development of delirium and the secondary outcomes of interest were type of statistical methods used, variables included in the mining algorithms and clinically important outcomes such as mortality and length of hospital stay. The quality of studies was graded using the CHARMs checklist. Six retrospective single centre observational studies were included (n = 178,091), of which 17, 574 participants developed delirium. Studies were of generally of low to moderate quality. The most commonly studied method was random forest, followed by support vector machine and artificial neural networks. The model with best performance for delirium prediction was random forest, with area under receiver operating curve (AUROC) ranging from 0.78 to 0.91. Sensitivity ranged from 0.59 to 0.81 and specificity ranged from 0.73 to 0.92. Our systematic review suggests that machine-learning techniques can be utilised to predict delirium.

A Machine Learning-Based Algorithm for the Prediction of Intensive Care Unit Delirium (PRIDE): Retrospective Study

Background

Delirium frequently occurs among patients admitted to the intensive care unit (ICU). There is limited evidence to support interventions to treat or resolve delirium in patients who have already developed delirium. Therefore, the early recognition and prevention of delirium are important in the management of critically ill patients.

Objective

This study aims to develop and validate a delirium prediction model within 24 hours of admission to the ICU using electronic health record data. The algorithm was named the Prediction of ICU Delirium (PRIDE).

Methods

This is a retrospective cohort study performed at a tertiary referral hospital with 120 ICU beds. We only included patients who were 18 years or older at the time of admission and who stayed in the medical or surgical ICU. Patients were excluded if they lacked a Confusion Assessment Method for the ICU record from the day of ICU admission or if they had a positive Confusion Assessment Method for the ICU record at the time of ICU admission. The algorithm to predict delirium was developed using patient data from the first 2 years of the study period and validated using patient data from the last 6 months. Random forest (RF), Extreme Gradient Boosting (XGBoost), deep neural network (DNN), and logistic regression (LR) were used. The algorithms were externally validated using MIMIC-III data, and the algorithm with the largest area under the receiver operating characteristics (AUROC) curve in the external data set was named the PRIDE algorithm.

Results

A total of 37,543 cases were collected. After patient exclusion, 12,409 remained as our study population, of which 3816 (30.8%) patients experienced delirium incidents during the study period. Based on the exclusion criteria, out of the 96,016 ICU admission cases in the MIMIC-III data set, 2061 cases were included, and 272 (13.2%) delirium incidents occurred. The average AUROCs and 95% CIs for internal validation were 0.916 (95% CI 0.916-0.916) for RF, 0.919 (95% CI 0.919-0.919) for XGBoost, 0.881 (95% CI 0.878-0.884) for DNN, and 0.875 (95% CI 0.875-0.875) for LR. Regarding the external validation, the best AUROC were 0.721 (95% CI 0.72-0.721) for RF, 0.697 (95% CI 0.695-0.699) for XGBoost, 0.655 (95% CI 0.654-0.657) for DNN, and 0.631 (95% CI 0.631-0.631) for LR. The Brier score of the RF model is 0.168, indicating that it is well-calibrated.

Conclusions

A machine learning approach based on electronic health record data can be used to predict delirium within 24 hours of ICU admission. RF, XGBoost, DNN, and LR models were used, and they effectively predicted delirium. However, with the potential to advise ICU physicians and prevent ICU delirium, prospective studies are required to verify the algorithm’s performance.

A Machine Learning Approach for Investigating Delirium as a Multifactorial Syndrome

Delirium is a psycho-organic syndrome common in hospitalized patients, especially the elderly, and is associated with poor clinical outcomes. This study aims to identify the predictors that are mostly associated with the risk of delirium episodes using a machine learning technique (MLT). A random forest (RF) algorithm was used to evaluate the association between the subject’s characteristics and the 4AT (the 4 A’s test) score screening tool for delirium. RF algorithm was implemented using information based on demographic characteristics, comorbidities, drugs and procedures. Of the 78 patients enrolled in the study, 49 (63%) were at risk for delirium, 32 (41%) had at least one episode of delirium during the hospitalization (38% in orthopedics and 31% both in internal medicine and in the geriatric ward). The model explained 75.8% of the variability of the 4AT score with a root mean squared error of 3.29. Higher age, the presence of dementia, physical restraint, diabetes and a lower degree are the variables associated with an increase of the 4AT score. Random forest is a valid method for investigating the patients’ characteristics associated with delirium onset also in small case-series. The use of this model may allow for early detection of delirium onset to plan the proper adjustment in healthcare assistance.

Post-stroke Anxiety Analysis via Machine Learning Methods

Post-stroke anxiety (PSA) has caused wide public concern in recent years, and the study on risk factors analysis and prediction is still an open issue. With the deepening of the research, machine learning has been widely applied to various scenarios and make great achievements increasingly, which brings new approaches to this field. In this paper, 395 patients with acute ischemic stroke are collected and evaluated by anxiety scales (i.e., HADS-A, HAMA, and SAS), hence the patients are divided into anxiety group and non-anxiety group. Afterward, the results of demographic data and general laboratory examination between the two groups are compared to identify the risk factors with statistical differences accordingly. Then the factors with statistical differences are incorporated into a multivariate logistic regression to obtain risk factors and protective factors of PSA. Statistical analysis shows great differences in gender, age, serious stroke, hypertension, diabetes mellitus, drinking, and HDL-C level between PSA group and non-anxiety group with HADS-A and HAMA evaluation. Meanwhile, as evaluated by SAS scale, gender, serious stroke, hypertension, diabetes mellitus, drinking, and HDL-C level differ in the PSA group and the non-anxiety group. Multivariate logistic regression analysis of HADS-A, HAMA, and SAS scales suggest that hypertension, diabetes mellitus, drinking, high NIHSS score, and low serum HDL-C level are related to PSA. In other words, gender, age, disability, hypertension, diabetes mellitus, HDL-C, and drinking are closely related to anxiety during the acute stage of ischemic stroke. Hypertension, diabetes mellitus, drinking, and disability increased the risk of PSA, and higher serum HDL-C level decreased the risk of PSA. Several machine learning methods are employed to predict PSA according to HADS-A, HAMA, and SAS scores, respectively. The experimental results indicate that random forest outperforms the competitive methods in PSA prediction, which contributes to early intervention for clinical treatment.

Machine Learning for Risk Group Identification and User Data Collection in a Herpes Simplex Virus Patient Registry: Algorithm Development and Validation Study

BACKGROUND

Researching people with herpes simplex virus (HSV) is challenging because of poor data quality, low user engagement, and concerns around stigma and anonymity.

OBJECTIVE

This project aimed to improve data collection for a real-world HSV registry by identifying predictors of HSV infection and selecting a limited number of relevant questions to ask new registry users to determine their level of HSV infection risk.

METHODS

The US National Health and Nutrition Examination Survey (NHANES, 2015-2016) database includes the confirmed HSV type 1 and type 2 (HSV-1 and HSV-2, respectively) status of American participants (14-49 years) and a wealth of demographic and health-related data. The questionnaires and data sets from this survey were used to form two data sets: one for HSV-1 and one for HSV-2. These data sets were used to train and test a model that used a random forest algorithm (devised using Python) to minimize the number of anonymous lifestyle-based questions needed to identify risk groups for HSV.

RESULTS

The model selected a reduced number of questions from the NHANES questionnaire that predicted HSV infection risk with high accuracy scores of 0.91 and 0.96 and high recall scores of 0.88 and 0.98 for the HSV-1 and HSV-2 data sets, respectively. The number of questions was reduced from 150 to an average of 40, depending on age and gender. The model, therefore, provided high predictability of risk of infection with minimal required input.

CONCLUSIONS

This machine learning algorithm can be used in a real-world evidence registry to collect relevant lifestyle data and identify individuals' levels of risk of HSV infection. A limitation is the absence of real user data and integration with electronic medical records, which would enable model learning and improvement. Future work will explore model adjustments, anonymization options, explicit permissions, and a standardized data schema that meet the General Data Protection Regulation, Health Insurance Portability and Accountability Act, and third-party interface connectivity requirements.

Risk prediction of delirium in hospitalized patients using machine learning: An implementation and prospective evaluation study

OBJECTIVE

Machine learning models trained on electronic health records have achieved high prognostic accuracy in test datasets, but little is known about their embedding into clinical workflows. We implemented a random forest-based algorithm to identify hospitalized patients at high risk for delirium, and evaluated its performance in a clinical setting.

MATERIALS AND METHODS

Delirium was predicted at admission and recalculated on the evening of admission. The defined prediction outcome was a delirium coded for the recent hospital stay. During 7 months of prospective evaluation, 5530 predictions were analyzed. In addition, 119 predictions for internal medicine patients were compared with ratings of clinical experts in a blinded and nonblinded setting.

RESULTS

During clinical application, the algorithm achieved a sensitivity of 74.1% and a specificity of 82.2%. Discrimination on prospective data (area under the receiver-operating characteristic curve = 0.86) was as good as in the test dataset, but calibration was poor. The predictions correlated strongly with delirium risk perceived by experts in the blinded (r = 0.81) and nonblinded (r = 0.62) settings. A major advantage of our setting was the timely prediction without additional data entry.

DISCUSSION

The implemented machine learning algorithm achieved a stable performance predicting delirium in high agreement with expert ratings, but improvement of calibration is needed. Future research should evaluate the acceptance of implemented machine learning algorithms by health professionals.

CONCLUSIONS

Our study provides new insights into the implementation process of a machine learning algorithm into a clinical workflow and demonstrates its predictive power for delirium.

A novel model to label delirium in an intensive care unit from clinician actions

BACKGROUND

In the intensive care unit (ICU), delirium is a common, acute, confusional state associated with high risk for short- and long-term morbidity and mortality. Machine learning (ML) has promise to address research priorities and improve delirium outcomes. However, due to clinical and billing conventions, delirium is often inconsistently or incompletely labeled in electronic health record (EHR) datasets. Here, we identify clinical actions abstracted from clinical guidelines in electronic health records (EHR) data that indicate risk of delirium among intensive care unit (ICU) patients. We develop a novel prediction model to label patients with delirium based on a large data set and assess model performance.

METHODS

EHR data on 48,451 admissions from 2001 to 2012, available through Medical Information Mart for Intensive Care-III database (MIMIC-III), was used to identify features to develop our prediction models. Five binary ML classification models (Logistic Regression; Classification and Regression Trees; Random Forests; Naïve Bayes; and Support Vector Machines) were fit and ranked by Area Under the Curve (AUC) scores. We compared our best model with two models previously proposed in the literature for goodness of fit, precision, and through biological validation.

RESULTS

Our best performing model with threshold reclassification for predicting delirium was based on a multiple logistic regression using the 31 clinical actions (AUC 0.83). Our model out performed other proposed models by biological validation on clinically meaningful, delirium-associated outcomes.

CONCLUSIONS

Hurdles in identifying accurate labels in large-scale datasets limit clinical applications of ML in delirium. We developed a novel labeling model for delirium in the ICU using a large, public data set. By using guideline-directed clinical actions independent from risk factors, treatments, and outcomes as model predictors, our classifier could be used as a delirium label for future clinically targeted models.

Machine Learning to Develop and Internally Validate a Predictive Model for Post-operative Delirium in a Prospective, Observational Clinical Cohort Study of Older Surgical Patients

BACKGROUND

Our objective was to assess the performance of machine learning methods to predict post-operative delirium using a prospective clinical cohort.

METHODS

We analyzed data from an observational cohort study of 560 older adults (≥ 70 years) without dementia undergoing major elective non-cardiac surgery. Post-operative delirium was determined by the Confusion Assessment Method supplemented by a medical chart review (N = 134, 24%). Five machine learning algorithms and a standard stepwise logistic regression model were developed in a training sample (80% of participants) and evaluated in the remaining hold-out testing sample. We evaluated three overlapping feature sets, restricted to variables that are readily available or minimally burdensome to collect in clinical settings, including interview and medical record data. A large feature set included 71 potential predictors. A smaller set of 18 features was selected by an expert panel using a consensus process, and this smaller feature set was considered with and without a measure of pre-operative mental status.

RESULTS

The area under the receiver operating characteristic curve (AUC) was higher in the large feature set conditions (range of AUC, 0.62-0.71 across algorithms) versus the selected feature set conditions (AUC range, 0.53-0.57). The restricted feature set with mental status had intermediate AUC values (range, 0.53-0.68). In the full feature set condition, algorithms such as gradient boosting, cross-validated logistic regression, and neural network (AUC = 0.71, 95% CI 0.58-0.83) were comparable with a model developed using traditional stepwise logistic regression (AUC = 0.69, 95% CI 0.57-0.82). Calibration for all models and feature sets was poor.

CONCLUSIONS

We developed machine learning prediction models for post-operative delirium that performed better than chance and are comparable with traditional stepwise logistic regression. Delirium proved to be a phenotype that was difficult to predict with appreciable accuracy.

Artificial intelligence-assisted prediction of preeclampsia: Development and external validation of a nationwide health insurance dataset of the BPJS Kesehatan in Indonesia

Background

We developed and validated an artificial intelligence (AI)-assisted prediction of preeclampsia applied to a nationwide health insurance dataset in Indonesia.

Methods

The BPJS Kesehatan dataset have been preprocessed using a nested case-control design into preeclampsia/eclampsia (n = 3318) and normotensive pregnant women (n = 19,883) from all women with one pregnancy. The dataset provided 95 features consisting of demographic variables and medical histories started from 24 months to event and ended by delivery as the event. Six algorithms were compared by area under the receiver operating characteristics curve (AUROC) with a subgroup analysis by time to the event. We compared our model to similar prediction models from systematically reviewed studies. In addition, we conducted a text mining analysis based on natural language processing techniques to interpret our modeling results.

Findings

The best model consisted of 17 predictors extracted by a random forest algorithm. Nine∼12 months to the event was the period that had the best AUROC in external validation by either geographical (0.88, 95% confidence interval (CI) 0.88–0.89) or temporal split (0.86, 95% CI 0.85–0.86). We compared this model to prediction models in seven studies from 869 records in PUBMED, EMBASE, and SCOPUS. This model outperformed the previous models in terms of the precision, sensitivity, and specificity in all validation sets.

Interpretation

Our low-cost model improved preliminary prediction to decide pregnant women that will be predicted by the models with high specificity and advanced predictors.

Funding

This work was supported by grant no. MOST108-2221-E-038-018 from the Ministry of Science and Technology of Taiwan.

Exploiting Machine Learning Algorithms and Methods for the Prediction of Agitated Delirium After Cardiac Surgery: Models Development and Validation Study

Background

Delirium is a temporary mental disorder that occasionally affects patients undergoing surgery, especially cardiac surgery. It is strongly associated with major adverse events, which in turn leads to increased cost and poor outcomes (eg, need for nursing home due to cognitive impairment, stroke, and death). The ability to foresee patients at risk of delirium will guide the timely initiation of multimodal preventive interventions, which will aid in reducing the burden and negative consequences associated with delirium. Several studies have focused on the prediction of delirium. However, the number of studies in cardiac surgical patients that have used machine learning methods is very limited.

Objective

This study aimed to explore the application of several machine learning predictive models that can pre-emptively predict delirium in patients undergoing cardiac surgery and compare their performance.

Methods

We investigated a number of machine learning methods to develop models that can predict delirium after cardiac surgery. A clinical dataset comprising over 5000 actual patients who underwent cardiac surgery in a single center was used to develop the models using logistic regression, artificial neural networks (ANN), support vector machines (SVM), Bayesian belief networks (BBN), naïve Bayesian, random forest, and decision trees.

Results

Only 507 out of 5584 patients (11.4%) developed delirium. We addressed the underlying class imbalance, using random undersampling, in the training dataset. The final prediction performance was validated on a separate test dataset. Owing to the target class imbalance, several measures were used to evaluate algorithm’s performance for the delirium class on the test dataset. Out of the selected algorithms, the SVM algorithm had the best F1 score for positive cases, kappa, and positive predictive value (40.2%, 29.3%, and 29.7%, respectively) with a P=.01, .03, .02, respectively. The ANN had the best receiver-operator area-under the curve (78.2%; P=.03). The BBN had the best precision-recall area-under the curve for detecting positive cases (30.4%; P=.03).

Conclusions

Although delirium is inherently complex, preventive measures to mitigate its negative effect can be applied proactively if patients at risk are prospectively identified. Our results highlight 2 important points: (1) addressing class imbalance on the training dataset will augment machine learning model’s performance in identifying patients likely to develop postoperative delirium, and (2) as the prediction of postoperative delirium is difficult because it is multifactorial and has complex pathophysiology, applying machine learning methods (complex or simple) may improve the prediction by revealing hidden patterns, which will lead to cost reduction by prevention of complications and will optimize patients’ outcomes.