Risk prediction with machine learning and regression methods

Prediction models for diagnosis and prognosis of the colonisation or infection of multidrug-resistant organisms in adults: A systematic review, critical appraisal, and meta-analysis

BACKGROUND

Prediction models help target patients at risk of multidrug-resistant organism (MDRO) colonisation or infection and could serve as tools informing clinical practices to prevent MDRO transmission and inappropriate empiric antibiotic therapy. However, limited evidence identifies which among the available models are of low risk of bias and suitable for clinical application.

OBJECTIVES

To identify, describe, appraise, and summarise the performance of all prognostic and diagnostic models developed or validated for predicting MDRO colonisation or infection.

DATA SOURCES

Six electronic literature databases and clinical registration databases were searched until April 2022.

STUDY ELIGIBILITY CRITERIA

Development and validation studies of any multivariable prognostic and diagnostic models to predict MDRO colonisation or infection in adults.

ASSESSMENT OF RISK OF BIAS

The Prediction Model Risk of Bias Assessment Tool was used to assess risk of bias. Evidence certainty was assessed using the GRADE approach.

METHODS OF DATA SYNTHESIS

Meta-analyses were conducted to summarise the discrimination and calibration of the models' external validations conducted in at least two non-overlapping datasets.

RESULTS

We included 162 models (108 studies) developed for diagnosing (n=135) and predicting (n=27) MDRO colonisation or infection. Models exhibited a high risk of bias, especially in statistical analysis. High-frequency predictors were age, recent invasive procedures, antibiotic usage, and prior hospitalisation. Less than 25% of the models underwent external validations, with only seven by independent teams. Meta-analyses for one diagnostic and two prognostic models only produced very-low to low certainty of evidence.

CONCLUSIONS

The review comprehensively described the models for identifying patients at risk of MDRO colonisation or infection. We cannot recommend which models are ready for application due to high risk of bias, limited validations, and low certainty of evidence from meta-analyses, indicating a clear need to improve the conducting and reporting of model development and external validation studies to facilitate clinical application.

Enhancing selection of alcohol consumption-associated genes by random forest

Machine learning methods have been used in identifying omics markers for a variety of phenotypes. We aimed to examine whether a supervised machine learning algorithm can improve identification of alcohol-associated transcriptomic markers. In this study, we analysed array-based, whole-blood derived expression data for 17 873 gene transcripts in 5508 Framingham Heart Study participants. By using the Boruta algorithm, a supervised random forest (RF)-based feature selection method, we selected twenty-five alcohol-associated transcripts. In a testing set (30 % of entire study participants), AUC (area under the receiver operating characteristics curve) of these twenty-five transcripts were 0·73, 0·69 and 0·66 for non-drinkers v. moderate drinkers, non-drinkers v. heavy drinkers and moderate drinkers v. heavy drinkers, respectively. The AUC of the selected transcripts by the Boruta method were comparable to those identified using conventional linear regression models, for example, AUC of 1958 transcripts identified by conventional linear regression models (false discovery rate < 0·2) were 0·74, 0·66 and 0·65, respectively. With Bonferroni correction for the twenty-five Boruta method-selected transcripts and three CVD risk factors (i.e. at P < 6·7e-4), we observed thirteen transcripts were associated with obesity, three transcripts with type 2 diabetes and one transcript with hypertension. For example, we observed that alcohol consumption was inversely associated with the expression of DOCK4, IL4R, and SORT1, and DOCK4 and SORT1 were positively associated with obesity, and IL4R was inversely associated with hypertension. In conclusion, using a supervised machine learning method, the RF-based Boruta algorithm, we identified novel alcohol-associated gene transcripts.

External validation of a paediatric Smart triage model for use in resource limited facilities

Models for digital triage of sick children at emergency departments of hospitals in resource poor settings have been developed. However, prior to their adoption, external validation should be performed to ensure their generalizability. We externally validated a previously published nine-predictor paediatric triage model (Smart Triage) developed in Uganda using data from two hospitals in Kenya. Both discrimination and calibration were assessed, and recalibration was performed by optimizing the intercept for classifying patients into emergency, priority, or non-urgent categories based on low-risk and high-risk thresholds. A total of 2539 patients were eligible at Hospital 1 and 2464 at Hospital 2, and 5003 for both hospitals combined; admission rates were 8.9%, 4.5%, and 6.8%, respectively. The model showed good discrimination, with area under the receiver-operator curve (AUC) of 0.826, 0.784 and 0.821, respectively. The pre-calibrated model at a low-risk threshold of 8% achieved a sensitivity of 93% (95% confidence interval, (CI):89%-96%), 81% (CI:74%-88%), and 89% (CI:85%-92%), respectively, and at a high-risk threshold of 40%, the model achieved a specificity of 86% (CI:84%-87%), 96% (CI:95%-97%), and 91% (CI:90%-92%), respectively. Recalibration improved the graphical fit, but new risk thresholds were required to optimize sensitivity and specificity.The Smart Triage model showed good discrimination on external validation but required recalibration to improve the graphical fit of the calibration plot. There was no change in the order of prioritization of patients following recalibration in the respective triage categories. Recalibration required new site-specific risk thresholds that may not be needed if prioritization based on rank is all that is required. The Smart Triage model shows promise for wider application for use in triage for sick children in different settings.

Interpretable ensemble imbalance learning strategies for the risk assessment of severe-low-level wind shear based on LiDAR and PIREPs

The occurrence of severe low-level wind shear (S-LLWS) events in the vicinity of airport runways poses a significant threat to flight safety and exacerbates a burgeoning problem in civil aviation. Identifying the risk factors that contribute to occurrences of S-LLWS can facilitate the improvement of aviation safety. Despite the significant influence of S-LLWS on aviation safety, its occurrence is relatively infrequent in comparison to non-SLLWS incidents. In this study, we develop an S-LLWS risk prediction model through the utilization of ensemble imbalance learning (EIL) strategies, namely, BalanceCascade, EasyEnsemble, and RUSBoost. The data for this study were obtained from PIREPs and LiDAR at Hong Kong International Airport. The analysis revealed that the BalanceCascade strategy outperforms EasyEnsemble and RUSBoost in terms of prediction performance. Afterward, the SHapley Additive exPlanations (SHAP) interpretation tool was used in conjunction with the BalanceCascade model for the risk assessment of various factors. The four most influential risk factors, according to the SHAP interpretation tool, were hourly temperature, runway 25LD, runway 25LA, and RWY (encounter location of LLWS). S-LLWS was likely to happen at Runway 25LD and Runway 25LA in temperatures ranging from low to moderate. Similarly, a high proportion of S-LLWS events occurred near the runway threshold, and a relatively small proportion occurred away from it. The EIL strategies in conjunction with the SHAP interpretation tool may accurately predict the S-LLWS without the need for data augmentation in the data pre-processing phase.

EXIST: EXamining rIsk of excesS adiposiTy-Machine learning to predict obesity-related complications

Background

Obesity is associated with an increased risk of multiple conditions, ranging from heart disease to cancer. However, there are few predictive models for these outcomes that have been developed specifically for people with overweight/obesity.

Objective

To develop predictive models for obesity-related complications in patients with overweight and obesity.

Methods

Electronic health record data of adults with body mass index 25-80 kg/m2 treated in primary care practices between 2000 and 2019 were utilized to develop and evaluate predictive models for nine long-term clinical outcomes using a) Lasso-Cox models and b) a machine-learning method random survival forests (RSF). Models were trained on a training dataset and evaluated on a test dataset over 100 replicates. Parsimonious models of <10 variables were also developed using Lasso-Cox.

Results

Over a median follow-up of 5.6 years, study outcome incidence in the cohort of 433,272 patients ranged from 1.8% for knee replacement to 11.7% for atherosclerotic cardiovascular disease. Harrell C-index averaged over replicates ranged from 0.702 for liver outcomes to 0.896 for death for RSF, and from 0.694 for liver outcomes to 0.891 for death for Lasso-Cox. The Harrell C-index for parsimonious models ranged from 0.675 for liver outcomes to 0.850 for knee replacement.

Conclusions

Predictive modeling can identify patients at high risk of obesity-related complications. Interpretable Cox models achieve results close to those of machine learning methods and could be helpful for population health management and clinical treatment decisions.

Artificial intelligence in the prediction of venous thromboembolism: A systematic review and pooled analysis

BACKGROUND

Accurate diagnostic and prognostic predictions of venous thromboembolism (VTE) are crucial for VTE management. Artificial intelligence (AI) enables autonomous identification of the most predictive patterns from large complex data. Although evidence regarding its performance in VTE prediction is emerging, a comprehensive analysis of performance is lacking.

AIMS

To systematically review the performance of AI in the diagnosis and prediction of VTE and compare it to clinical risk assessment models (RAMs) or logistic regression models.

METHODS

A systematic literature search was performed using PubMed, MEDLINE, EMBASE, and Web of Science from inception to April 20, 2021. Search terms included "artificial intelligence" and "venous thromboembolism." Eligible criteria were original studies evaluating AI in the prediction of VTE in adults and reporting one of the following outcomes: sensitivity, specificity, positive predictive value, negative predictive value, or area under receiver operating curve (AUC). Risks of bias were assessed using the PROBAST tool. Unpaired t-test was performed to compare the mean AUC from AI versus conventional methods (RAMs or logistic regression models).

RESULTS

A total of 20 studies were included. Number of participants ranged from 31 to 111 888. The AI-based models included artificial neural network (six studies), support vector machines (four studies), Bayesian methods (one study), super learner ensemble (one study), genetic programming (one study), unspecified machine learning models (two studies), and multiple machine learning models (five studies). Twelve studies (60%) had both training and testing cohorts. Among 14 studies (70%) where AUCs were reported, the mean AUC for AI versus conventional methods were 0.79 (95% CI: 0.74-0.85) versus 0.61 (95% CI: 0.54-0.68), respectively (p < .001). However, the good to excellent discriminative performance of AI methods is unlikely to be replicated when used in clinical practice, because most studies had high risk of bias due to missing data handling and outcome determination.

CONCLUSION

The use of AI appears to improve the accuracy of diagnostic and prognostic prediction of VTE over conventional risk models; however, there was a high risk of bias observed across studies. Future studies should focus on transparent reporting, external validation, and clinical application of these models.

Genomic Biomarker Discovery in Disease Progression and Therapy Response in Bladder Cancer Utilizing Machine Learning

Cancer in all its forms of expression is a major cause of death. To identify the genomic reason behind cancer, discovery of biomarkers is needed. In this paper, genomic data of bladder cancer are examined for the purpose of biomarker discovery. Genomic biomarkers are indicators stemming from the study of the genome, either at a very low level based on the genome sequence itself, or more abstractly such as measuring the level of gene expression for different disease groups. The latter method is pivotal for this work, since the available datasets consist of RNA sequencing data, transformed to gene expression levels, as well as data on a multitude of clinical indicators. Based on this, various methods are utilized such as statistical modeling via logistic regression and regularization techniques (elastic-net), clustering, survival analysis through Kaplan-Meier curves, and heatmaps for the experiments leading to biomarker discovery. The experiments have led to the discovery of two gene signatures capable of predicting therapy response and disease progression with considerable accuracy for bladder cancer patients which correlates well with clinical indicators such as Therapy Response and T-Stage at surgery with Disease Progression in a time-to-event manner.

Machine learning-enabled risk prediction of chronic obstructive pulmonary disease with unbalanced data

BACKGROUND AND OBJECTIVE

Since the early symptoms of chronic obstructive pulmonary disease (COPD) are not obvious, patients are not easily identified, causing improper time for prevention and treatment. In present study, machine learning (ML) methods were employed to construct a risk prediction model for COPD to improve its prediction efficiency.

METHODS

We collected data from a sample of 5807 cases with a complete COPD diagnosis from the 2019 COPD Surveillance Program in Shanxi Province and extracted 34 potentially relevant variables from the dataset. Firstly, we used feature selection methods (i.e., Generalized elastic net, Lasso and Adaptive lasso) to select ten variables. Afterwards, we employed supervised classifiers for class imbalanced data by combining the cost-sensitive learning and SMOTE resampling methods with the ML methods (Logistic Regression, SVM, Random Forest, XGBoost, LightGBM, NGBoost and Stacking), respectively. Last, we assessed their performance.

RESULTS

The cough frequently at age 14 and before and other 9 variables are significant parameters for COPD. The Stacking heterogeneous ensemble model showed relatively good performance in the unbalanced datasets. The Logistic Regression with class weighting enjoyed the best classification performance in the balancing data when these composite indicators (AUC, F1-Score and G-mean) were used as criteria for model comparison. The values of F1-Score and G-mean for the top three ML models were 0.290/0.660 for Logistic Regression with class weighting, 0.288/0.649 for Stacking with synthetic minority oversampling technique (SMOTE), and 0.285/0.648 for LightGBM with SMOTE.

CONCLUSIONS

This paper combining feature selection methods, unbalanced data processing methods and machine learning methods with data from disease surveillance questionnaires and physical measurements to identify people at risk of COPD, concluded that machine learning models based on survey questionnaires could provide an automated identification for patients at risk of COPD, and provide a simple and scientific aid for early identification of COPD.

Machine Learning Models of Postoperative Atrial Fibrillation Prediction After Cardiac Surgery

OBJECTIVES

This study aimed to use machine learning algorithms to build an efficient forecasting model of atrial fibrillation after cardiac surgery, and to compare the predictive performance of machine learning to traditional logistic regression.

DESIGN

A retrospective study.

SETTING

Second Affiliated Hospital of Zhejiang University School of Medicine.

PARTICIPANTS

The study comprised 1,400 patients who underwent valve and/or coronary artery bypass grafting surgery with cardiopulmonary bypass from September 1, 2013 to December 31, 2018.

INTERVENTIONS

None.

MEASUREMENTS AND MAIN RESULTS

Two machine learning approaches (gradient-boosting decision tree and support-vector machine) and logistic regression were used to build predictive models. The performance was compared by the area under the curve (AUC). The clinical practicability was assessed using decision curve analysis. Postoperative atrial fibrillation occurred in 519 patients (37.1%). The AUCs of the support-vector machine, logistic regression, and gradient boosting decision tree were 0.777 (95% CI: 0.772-0.781), 0.767 (95% CI: 0.762-0.772), and 0.765 (95% CI: 0.761-0.770), respectively. As decision curve analysis manifested, these models had achieved appropriate net benefit.

CONCLUSION

In the authors' study, the support-vector machine model was the best predictor; it may be an effective tool for predicting atrial fibrillation after cardiac surgery.

Machine Learning for Predicting the Development of Postoperative Acute Kidney Injury After Coronary Artery Bypass Grafting Without Extracorporeal Circulation

Background: Cardiac surgery-associated acute kidney injury (CSA-AKI) is a major complication that increases morbidity and mortality after cardiac surgery. Most established predictive models are limited to the analysis of nonlinear relationships and do not adequately consider intraoperative variables and early postoperative variables. Nonextracorporeal circulation coronary artery bypass grafting (off-pump CABG) remains the procedure of choice for most coronary surgeries, and refined CSA-AKI predictive models for off-pump CABG are notably lacking. Therefore, this study used an artificial intelligence-based machine learning approach to predict CSA-AKI from comprehensive perioperative data.

Methods: In total, 293 variables were analysed in the clinical data of patients undergoing off-pump CABG in the Department of Cardiac Surgery at the First Affiliated Hospital of Guangxi Medical University between 2012 and 2021. According to the KDIGO criteria, postoperative AKI was defined by an elevation of at least 50% within 7 days, or 0.3 mg/dL within 48 hours, with respect to the reference serum creatinine level. Five machine learning algorithms—a simple decision tree, random forest, support vector machine, extreme gradient boosting and gradient boosting decision tree (GBDT)—were used to construct the CSA-AKI predictive model. The performance of these models was evaluated with the area under the receiver operating characteristic curve (AUC). Shapley additive explanation (SHAP) values were used to explain the predictive model.

Results: The three most influential features in the importance matrix plot were 1-day postoperative serum potassium concentration, 1-day postoperative serum magnesium ion concentration, and 1-day postoperative serum creatine phosphokinase concentration.

Conclusion: GBDT exhibited the largest AUC (0.87) and can be used to predict the risk of AKI development after surgery, thus enabling clinicians to optimise treatment strategies and minimise postoperative complications.

Using Machine Learning to make nanomaterials sustainable

Sustainable development is a key challenge for contemporary human societies; failure to achieve sustainability could threaten human survival. In this review article, we illustrate how Machine Learning (ML) could support more sustainable development, covering the basics of data gathering through each step of the Environmental Risk Assessment (ERA). The literature provides several examples showing how ML can be employed in most steps of a typical ERA.A key observation is that there are currently no clear guidance for using such autonomous technologies in ERAs or which standards/checks are required. Steering thus seems to be the most important task for supporting the use of ML in the ERA of nano- and smart-materials. Resources should be devoted to developing a strategy for implementing ML in ERA with a strong emphasis on data foundations, methodologies, and the related sensitivities/uncertainties. We should recognise historical errors and biases (e.g., in data) to avoid embedding them during ML programming.

Regression-based prognostic models for functional independence after postacute brain injury rehabilitation are not transportable: a systematic review

BACKGROUND AND OBJECTIVES

To identify and summarize validated multivariable prognostic models for the Functional Independence Measure® (FIM®) at discharge from post-acute inpatient rehabilitation in adults with acquired brain injury (ABI).

METHODS

This review was conducted based on the recommendations of the Cochrane Prognosis Methods Group and adheres to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines. Three databases were systematically searched in May 2021 and updated in April 2022. Main inclusion criteria were: a) adult patients with ABI, b) validated multivariable prognostic model, c) time of prognostication within 1-week of admission to post-acute rehabilitation, and d) outcome was the FIM® at discharge from post-acute rehabilitation.

RESULTS

The search yielded 3,169 unique articles. Three articles fulfilled the inclusion criteria, accounting for n = 6 internally and n = 2 externally validated prognostic models. Discrimination was estimated as an area under the curve between 0.76 and 0.89. Calibration was deemed to be assessed insufficiently. The included models were judged to be of high risk of bias.

CONCLUSION

Current prognostic models for the FIM® in post-acute rehabilitation for patients with ABI lack the methodological rigor to support clinical use outside the development setting. Future studies addressing functional independence should ensure appropriate model validation and conform to uniform reporting standards for prognosis research.

A comparison of machine learning algorithms and traditional regression-based statistical modeling for predicting hypertension incidence in a Canadian population

Risk prediction models are frequently used to identify individuals at risk of developing hypertension. This study evaluates different machine learning algorithms and compares their predictive performance with the conventional Cox proportional hazards (PH) model to predict hypertension incidence using survival data. This study analyzed 18,322 participants on 24 candidate features from the large Alberta's Tomorrow Project (ATP) to develop different prediction models. To select the top features, we applied five feature selection methods, including two filter-based: a univariate Cox p-value and C-index; two embedded-based: random survival forest and least absolute shrinkage and selection operator (Lasso); and one constraint-based: the statistically equivalent signature (SES). Five machine learning algorithms were developed to predict hypertension incidence: penalized regression Ridge, Lasso, Elastic Net (EN), random survival forest (RSF), and gradient boosting (GB), along with the conventional Cox PH model. The predictive performance of the models was assessed using C-index. The performance of machine learning algorithms was observed, similar to the conventional Cox PH model. Average C-indexes were 0.78, 0.78, 0.78, 0.76, 0.76, and 0.77 for Ridge, Lasso, EN, RSF, GB and Cox PH, respectively. Important features associated with each model were also presented. Our study findings demonstrate little predictive performance difference between machine learning algorithms and the conventional Cox PH regression model in predicting hypertension incidence. In a moderate dataset with a reasonable number of features, conventional regression-based models perform similar to machine learning algorithms with good predictive accuracy.

Innovations in suicide prevention research (INSPIRE): a protocol for a population-based case-control study

BACKGROUND

Suicide deaths have been increasing for the past 20 years in the USA resulting in 45 979 deaths in 2020, a 29% increase since 1999. Lack of data linkage between entities with potential to implement large suicide prevention initiatives (health insurers, health institutions and corrections) is a barrier to developing an integrated framework for suicide prevention.

OBJECTIVES

Data linkage between death records and several large administrative datasets to (1) estimate associations between risk factors and suicide outcomes, (2) develop predictive algorithms and (3) establish long-term data linkage workflow to ensure ongoing suicide surveillance.

METHODS

We will combine six data sources from North Carolina, the 10th most populous state in the USA, from 2006 onward, including death certificate records, violent deaths reporting system, large private health insurance claims data, Medicaid claims data, University of North Carolina electronic health records and data on justice involved individuals released from incarceration. We will determine the incidence of death from suicide, suicide attempts and ideation in the four subpopulations to establish benchmarks. We will use a nested case-control design with incidence density-matched population-based controls to (1) identify short-term and long-term risk factors associated with suicide attempts and mortality and (2) develop machine learning-based predictive algorithms to identify individuals at risk of suicide deaths.

DISCUSSION

We will address gaps from prior studies by establishing an in-depth linked suicide surveillance system integrating multiple large, comprehensive databases that permit establishment of benchmarks, identification of predictors, evaluation of prevention efforts and establishment of long-term surveillance workflow protocols.

Translating Data Analytics Into Improved Spine Surgery Outcomes: A Roadmap for Biomedical Informatics Research in 2021

STUDY DESIGN

Narrative review.

OBJECTIVES

There is growing interest in the use of biomedical informatics and data analytics tools in spine surgery. Yet despite the rapid growth in research on these topics, few analytic tools have been implemented in routine spine practice. The purpose of this review is to provide a health information technology (HIT) roadmap to help translate data assets and analytics tools into measurable advances in spine surgical care.

METHODS

We conducted a narrative review of PubMed and Google Scholar to identify publications discussing data assets, analytical approaches, and implementation strategies relevant to spine surgery practice.

RESULTS

A variety of data assets are available for spine research, ranging from commonly used datasets, such as administrative billing data, to emerging resources, such as mobile health and biobanks. Both regression and machine learning techniques are valuable for analyzing these assets, and researchers should recognize the particular strengths and weaknesses of each approach. Few studies have focused on the implementation of HIT, and a variety of methods exist to help translate analytic tools into clinically useful interventions. Finally, a number of HIT-related challenges must be recognized and addressed, including stakeholder acceptance, regulatory oversight, and ethical considerations.

CONCLUSIONS

Biomedical informatics has the potential to support the development of new HIT that can improve spine surgery quality and outcomes. By understanding the development life-cycle that includes identifying an appropriate data asset, selecting an analytic approach, and leveraging an effective implementation strategy, spine researchers can translate this potential into measurable advances in patient care.

Applied Machine Learning for Spine Surgeons: Predicting Outcome for Patients Undergoing Treatment for Lumbar Disc Herniation Using PRO Data

STUDY DESIGN

Retrospective/prospective study.

OBJECTIVE

Models based on preoperative factors can predict patients' outcome at 1-year follow-up. This study measures the performance of several machine learning (ML) models and compares the results with conventional methods.

METHODS

Inclusion criteria were patients who had lumbar disc herniation (LDH) surgery, identified in the Danish national registry for spine surgery. Initial training of models included 16 independent variables, including demographics and presurgical patient-reported measures. Patients were grouped by reaching minimal clinically important difference or not for EuroQol, Oswestry Disability Index, Visual Analog Scale (VAS) Leg, and VAS Back and by their ability to return to work at 1 year follow-up. Data were randomly split into training, validation, and test sets by 50%/35%/15%. Deep learning, decision trees, random forest, boosted trees, and support vector machines model were trained, and for comparison, multivariate adaptive regression splines (MARS) and logistic regression models were used. Model fit was evaluated by inspecting area under the curve curves and performance during validation.

RESULTS

Seven models were arrived at. Classification errors were within ±1% to 4% SD across validation folds. ML did not yield superior performance compared with conventional models. MARS and deep learning performed consistently well. Discrepancy was greatest among VAS Leg models.

CONCLUSIONS

Five predictive ML and 2 conventional models were developed, predicting improvement for LDH patients at the 1-year follow-up. We demonstrate that it is possible to build an ensemble of models with little effort as a starting point for further model optimization and selection.

Developing a Prediction Model for Pathologic Complete Response Following Neoadjuvant Chemotherapy in Breast Cancer: A Comparison of Model Building Approaches

The optimal characteristics among patients with breast cancer to recommend neoadjuvant chemotherapy is an active area of clinical research. We developed and compared several approaches to developing prediction models for pathologic complete response (pCR) among patients with breast cancer in Alberta.

Comparing LASSO and random forest models for predicting neurological dysfunction among fluoroquinolone users

BACKGROUND

Fluoroquinolones are associated with central (CNS) and peripheral (PNS) nervous system symptoms, and predicting the risk of these outcomes may have important clinical implications. Both LASSO and random forest are appealing modeling methods, yet it is not clear which method performs better for clinical risk prediction.

PURPOSE

To compare models developed using LASSO versus random forest for predicting neurological dysfunction among fluoroquinolone users.

METHODS

We developed and validated risk prediction models using claims data from a commercially insured population. The study cohort included adults dispensed an oral fluoroquinolone, and outcomes were CNS and PNS dysfunction. Model predictors included demographic variables, comorbidities and medications known to be associated with neurological symptoms, and several healthcare utilization predictors. We assessed the accuracy and calibration of these models using measures including AUC, calibration curves, and Brier scores.

RESULTS

The underlying cohort contained 16 533 (1.18%) individuals with CNS dysfunction and 46 995 (3.34%) individuals with PNS dysfunction during 120 days of follow-up. For CNS dysfunction, LASSO had an AUC of 0.81 (95% CI: 0.80, 0.82), while random forest had an AUC of 0.80 (95% CI: 0.80, 0.81). For PNS dysfunction, LASSO had an AUC of 0.75 (95% CI: 0.74, 0.76) versus an AUC of 0.73 (95% CI: 0.73, 0.74) for random forest. Both LASSO models had better calibration, with Brier scores 0.17 (LASSO) versus 0.20 (random forest) for CNS dysfunction and 0.20 (LASSO) versus 0.25 (random forest) for PNS dysfunction.

CONCLUSIONS

LASSO outperformed random forest in predicting CNS and PNS dysfunction among fluoroquinolone users, and should be considered for modeling when the cohort is modest in size, when the number of model predictors is modest, and when predictors are primarily binary.

A Machine Learning Classifier Improves Mortality Prediction Compared With Pediatric Logistic Organ Dysfunction-2 Score: Model Development and Validation

Supplemental Digital Content is available in the text.

Objectives:

To determine whether machine learning algorithms can better predict PICU mortality than the Pediatric Logistic Organ Dysfunction-2 score.

Design:

Retrospective study.

Setting:

Quaternary care medical-surgical PICU.

Patients:

All patients admitted to the PICU from 2013 to 2019.

Interventions:

None.

Measurements and Main Results:

We investigated the performance of various machine learning algorithms using the same variables used to calculate the Pediatric Logistic Organ Dysfunction-2 score to predict PICU mortality. We used 10,194 patient records from 2013 to 2017 for training and 4,043 patient records from 2018 to 2019 as a holdout validation cohort. Mortality rate was 3.0% in the training cohort and 3.4% in the validation cohort. The best performing algorithm was a random forest model (area under the receiver operating characteristic curve, 0.867 [95% CI, 0.863–0.895]; area under the precision-recall curve, 0.327 [95% CI, 0.246–0.414]; F1, 0.396 [95% CI, 0.321–0.468]) and significantly outperformed the Pediatric Logistic Organ Dysfunction-2 score (area under the receiver operating characteristic curve, 0.761 [95% CI, 0.713–0.810]; area under the precision-recall curve (0.239 [95% CI, 0.165–0.316]; F1, 0.284 [95% CI, 0.209–0.360]), although this difference was reduced after retraining the Pediatric Logistic Organ Dysfunction-2 logistic regression model at the study institution. The random forest model also showed better calibration than the Pediatric Logistic Organ Dysfunction-2 score, and calibration of the random forest model remained superior to the retrained Pediatric Logistic Organ Dysfunction-2 model.

Conclusions:

A machine learning model achieved better performance than a logistic regression-based score for predicting ICU mortality. Better estimation of mortality risk can improve our ability to adjust for severity of illness in future studies, although external validation is required before this method can be widely deployed.

Machine Learning Compared With Conventional Statistical Models for Predicting Myocardial Infarction Readmission and Mortality: A Systematic Review

BACKGROUND

Machine learning (ML) methods are increasingly used in addition to conventional statistical modelling (CSM) for predicting readmission and mortality in patients with myocardial infarction (MI). However, the two approaches have not been systematically compared across studies of prognosis in patients with MI.

METHODS

Following PRISMA guidelines, we systematically reviewed the literature via Medline, EPub, Cochrane Central, Embase, Inspec, ACM Digital Library, and Web of Science. Eligible studies included primary research articles published from January 2000 to March 2020, comparing ML and CSM for prognostication after MI.

RESULTS

Of 7,348 articles, 112 underwent full-text review, with the final set composed of 24 articles representing 374,365 patients. ML methods included artificial neural networks (n = 12 studies), random forests (n = 11), decision trees (n = 8), support vector machines (n = 8), and Bayesian techniques (n = 7). CSM included logistic regression (n = 19 studies), existing CSM-derived risk scores (n = 12), and Cox regression (n = 2). Thirteen of 19 studies examining mortality reported higher C-indexes with the use of ML compared with CSM. One study examined readmissions at 2 different time points, with C-indexes that were higher for ML than CSM. Across all studies, a total of 29 comparisons were performed, but the majority (n = 26, 90%) found small (< 0.05) absolute differences in the C-index between ML and CSM. With the use of a modified CHARMS checklist, sources of bias were identifiable in the majority of studies, and only 2 were externally validated.

CONCLUSION

Although ML algorithms tended to have higher C-indexes than CSM for predicting death or readmission after MI, these studies exhibited threats to internal validity and were often unvalidated. Further comparisons are needed, with adherence to clinical quality standards for prognosis research. (Trial registration: PROSPERO CRD42019134896).

Prognostic models will be victims of their own success, unless…

Predictive analytics have begun to change the workflows of healthcare by giving insight into our future health. Deploying prognostic models into clinical workflows should change behavior and motivate interventions that affect outcomes. As users respond to model predictions, downstream characteristics of the data, including the distribution of the outcome, may change. The ever-changing nature of healthcare necessitates maintenance of prognostic models to ensure their longevity. The more effective a model and intervention(s) are at improving outcomes, the faster a model will appear to degrade. Improving outcomes can disrupt the association between the model's predictors and the outcome. Model refitting may not always be the most effective response to these challenges. These problems will need to be mitigated by systematically incorporating interventions into prognostic models and by maintaining robust performance surveillance of models in clinical use. Holistically modeling the outcome and intervention(s) can lead to resilience to future compromises in performance.

AI and Big Data in Healthcare: Towards a More Comprehensive Research Framework for Multimorbidity

Multimorbidity refers to the coexistence of two or more chronic diseases in one person. Therefore, patients with multimorbidity have multiple and special care needs. However, in practice it is difficult to meet these needs because the organizational processes of current healthcare systems tend to be tailored to a single disease. To improve clinical decision making and patient care in multimorbidity, a radical change in the problem-solving approach to medical research and treatment is needed. In addition to the traditional reductionist approach, we propose interactive research supported by artificial intelligence (AI) and advanced big data analytics. Such research approach, when applied to data routinely collected in healthcare settings, provides an integrated platform for research tasks related to multimorbidity. This may include, for example, prediction, correlation, and classification problems based on multiple interaction factors. However, to realize the idea of this paradigm shift in multimorbidity research, the optimization, standardization, and most importantly, the integration of electronic health data into a common national and international research infrastructure is needed. Ultimately, there is a need for the integration and implementation of efficient AI approaches, particularly deep learning, into clinical routine directly within the workflows of the medical professionals.

Second opinion needed: communicating uncertainty in medical machine learning

There is great excitement that medical artificial intelligence (AI) based on machine learning (ML) can be used to improve decision making at the patient level in a variety of healthcare settings. However, the quantification and communication of uncertainty for individual predictions is often neglected even though uncertainty estimates could lead to more principled decision-making and enable machine learning models to automatically or semi-automatically abstain on samples for which there is high uncertainty. In this article, we provide an overview of different approaches to uncertainty quantification and abstention for machine learning and highlight how these techniques could improve the safety and reliability of current ML systems being used in healthcare settings. Effective quantification and communication of uncertainty could help to engender trust with healthcare workers, while providing safeguards against known failure modes of current machine learning approaches. As machine learning becomes further integrated into healthcare environments, the ability to say "I'm not sure" or "I don't know" when uncertain is a necessary capability to enable safe clinical deployment.

Precision Medicine: Academic dreaming or clinical reality?

Precision medicine can be distilled into a concept of accounting for an individual's unique collection of clinical, physiologic, genetic, and sociodemographic characteristics to provide patient-level predictions of disease course and response to therapy. Abundant evidence now allows us to determine how an average person with epilepsy will respond to specific medical and surgical treatments. This is useful, but not readily applicable to an individual patient. This has brought into sharp focus the desire for a more individualized approach through which we counsel people based on individual characteristics, as opposed to population-level data. We are now accruing data at unprecedented rates, allowing us to convert this ideal into reality. In addition, we have access to growing volumes of administrative and electronic health records data, biometric, imaging, genetics data, microbiome, and other "omics" data, thus paving the way toward phenome-wide association studies and "the epidemiology of one." Despite this, there are many challenges ahead. The collating, integrating, and storing sensitive multimodal data for advanced analytics remains difficult as patient consent and data security issues increase in complexity. Agreement on many aspects of epilepsy remains imperfect, rendering models sensitive to misclassification due to a lack of "ground truth." Even with existing data, advanced analytics models are prone to overfitting and often failure to generalize externally. Finally, uptake by clinicians is often hindered by opaque, "black box" algorithms. Systematic approaches to data collection and model generation, and an emphasis on education to promote uptake and knowledge translation, are required to propel epilepsy-based precision medicine from the realm of the theoretical into routine clinical practice.

Application of Machine Learning Methods in Nursing Home Research

Background: A machine learning (ML) system is able to construct algorithms to continue improving predictions and generate automated knowledge through data-driven predictors or decisions. Objective: The purpose of this study was to compare six ML methods (random forest (RF), logistics regression, linear support vector machine (SVM), polynomial SVM, radial SVM, and sigmoid SVM) of predicting falls in nursing homes (NHs). Methods: We applied three representative six-ML algorithms to the preprocessed dataset to develop a prediction model (N = 60). We used an accuracy measure to evaluate prediction models. Results: RF was the most accurate model (0.883), followed by the logistic regression model, SVM linear, and polynomial SVM (0.867). Conclusions: RF was a powerful algorithm to discern predictors of falls in NHs. For effective fall management, researchers should consider organizational characteristics as well as personal factors. Recommendations for Future Research: To confirm the superiority of ML in NH research, future studies are required to discern additional potential factors using newly introduced ML methods.

Neurocritical Care: Bench to Bedside (Eds. Claude Hemphill, Michael James) Integrating and Using Big Data in Neurocritical Care

The critical care environment drives huge volumes of data, and clinicians are tasked with quickly processing this data and responding to it urgently. The neurocritical care environment increasingly involves EEG, multimodal intracranial monitoring, and complex imaging which preclude comprehensive human synthesis, and requires new concepts to integrate data into clinical care. By definition, Big Data is data that cannot be handled using traditional infrastructures and is characterized by the volume, variety, velocity, and variability of the data being produced. Big Data in the neurocritical care unit requires rethinking of data storage infrastructures and the development of tools and analytics to drive advancements in the field. Preprocessing, feature extraction, statistical inference, and analytic tools are required in order to achieve the primary goals of Big Data for clinical use: description, prediction, and prescription. Barriers to its use at bedside include a lack of infrastructure development within the healthcare industry, lack of standardization of data inputs, and ultimately existential and scientific concerns about the outputs that result from the use of tools such as artificial intelligence. However, as implied by the fundamental theorem of biomedical informatics, physicians remain central to the development and utility of Big Data to improve patient care.

ADAPTS: An Intelligent Sustainable Conceptual Framework for Engineering Projects

This paper presents a conceptual framework for the optimization of environmental sustainability in engineering projects, both for products and industrial facilities or processes. The main objective of this work is to propose a conceptual framework to help researchers to approach optimization under the criteria of sustainability of engineering projects, making use of current Machine Learning techniques. For the development of this conceptual framework, a bibliographic search has been carried out on the Web of Science. From the selected documents and through a hermeneutic procedure the texts have been analyzed and the conceptual framework has been carried out. A graphic representation pyramid shape is shown to clearly define the variables of the proposed conceptual framework and their relationships. The conceptual framework consists of 5 dimensions; its acronym is ADAPTS. In the base are: (1) the Application to which it is intended, (2) the available DAta, (3) the APproach under which it is operated, and (4) the machine learning Tool used. At the top of the pyramid, (5) the necessary Sensing. A study case is proposed to show its applicability. This work is part of a broader line of research, in terms of optimization under sustainability criteria.

Logistic regression was as good as machine learning for predicting major chronic diseases

OBJECTIVE

To evaluate the performance of machine learning (ML) algorithms and to compare them with logistic regression for the prediction of risk of cardiovascular diseases (CVDs), chronic kidney disease (CKD), diabetes (DM), and hypertension (HTN) and in a prospective cohort study using simple clinical predictors.

STUDY DESIGN AND SETTING

We conducted analyses in a population-based cohort study in Asian adults (n = 6,762). Five different ML models were considered-single-hidden-layer neural network, support vector machine, random forest, gradient boosting machine, and k-nearest neighbor-and were compared with standard logistic regression.

RESULTS

The incidences at 6 years of CVD, CKD, DM, and HTN cases were 4.0%, 7.0%, 9.2%, and 34.6%, respectively. Logistic regression reached the highest area under the receiver operating characteristic curve for CKD (0.905 [0.88, 0.93]) and DM (0.768 [0.73, 0.81]) predictions. For CVD and HTN, the best models were neural network (0.753 [0.70, 0.81]) and support vector machine (0.780 [0.747, 0.812]), respectively. However, the differences with logistic regression were small (less than 1%) and nonsignificant. Logistic regression, gradient boosting machine, and neural network were systematically ranked among the best models.

CONCLUSION

Logistic regression yields as good performance as ML models to predict the risk of major chronic diseases with low incidence and simple clinical predictors.

FibroScan-AST (FAST) score for the non-invasive identification of patients with non-alcoholic steatohepatitis with significant activity and fibrosis: a prospective derivation and global validation study

BACKGROUND

The burden of non-alcoholic fatty liver disease (NAFLD) is increasing globally, and a major priority is to identify patients with non-alcoholic steatohepatitis (NASH) who are at greater risk of progression to cirrhosis, and who will be candidates for clinical trials and emerging new pharmacotherapies. We aimed to develop a score to identify patients with NASH, elevated NAFLD activity score (NAS≥4), and advanced fibrosis (stage 2 or higher [F≥2]).

METHODS

This prospective study included a derivation cohort before validation in multiple international cohorts. The derivation cohort was a cross-sectional, multicentre study of patients aged 18 years or older, scheduled to have a liver biopsy for suspicion of NAFLD at seven tertiary care liver centres in England. This was a prespecified secondary outcome of a study for which the primary endpoints have already been reported. Liver stiffness measurement (LSM) by vibration-controlled transient elastography and controlled attenuation parameter (CAP) measured by FibroScan device were combined with aspartate aminotransferase (AST), alanine aminotransferase (ALT), or AST:ALT ratio. To identify those patients with NASH, an elevated NAS, and significant fibrosis, the best fitting multivariable logistic regression model was identified and internally validated using boot-strapping. Score calibration and discrimination performance were determined in both the derivation dataset in England, and seven independent international (France, USA, China, Malaysia, Turkey) histologically confirmed cohorts of patients with NAFLD (external validation cohorts). This study is registered with ClinicalTrials.gov, number NCT01985009.

FINDINGS

Between March 20, 2014, and Jan 17, 2017, 350 patients with suspected NAFLD attending liver clinics in England were prospectively enrolled in the derivation cohort. The most predictive model combined LSM, CAP, and AST, and was designated FAST (FibroScan-AST). Performance was satisfactory in the derivation dataset (C-statistic 0·80, 95% CI 0·76-0·85) and was well calibrated. In external validation cohorts, calibration of the score was satisfactory and discrimination was good across the full range of validation cohorts (C-statistic range 0·74-0·95, 0·85; 95% CI 0·83-0·87 in the pooled external validation patients' cohort; n=1026). Cutoff was 0·35 for sensitivity of 0·90 or greater and 0·67 for specificity of 0·90 or greater in the derivation cohort, leading to a positive predictive value (PPV) of 0·83 (84/101) and a negative predictive value (NPV) of 0·85 (93/110). In the external validation cohorts, PPV ranged from 0·33 to 0·81 and NPV from 0·73 to 1·0.

INTERPRETATION

The FAST score provides an efficient way to non-invasively identify patients at risk of progressive NASH for clinical trials or treatments when they become available, and thereby reduce unnecessary liver biopsy in patients unlikely to have significant disease.

FUNDING

Echosens and UK National Institute for Health Research.

A nonparametric updating method to correct clinical prediction model drift

OBJECTIVE

Clinical prediction models require updating as performance deteriorates over time. We developed a testing procedure to select updating methods that minimizes overfitting, incorporates uncertainty associated with updating sample sizes, and is applicable to both parametric and nonparametric models.

MATERIALS AND METHODS

We describe a procedure to select an updating method for dichotomous outcome models by balancing simplicity against accuracy. We illustrate the test's properties on simulated scenarios of population shift and 2 models based on Department of Veterans Affairs inpatient admissions.

RESULTS

In simulations, the test generally recommended no update under no population shift, no update or modest recalibration under case mix shifts, intercept correction under changing outcome rates, and refitting under shifted predictor-outcome associations. The recommended updates provided superior or similar calibration to that achieved with more complex updating. In the case study, however, small update sets lead the test to recommend simpler updates than may have been ideal based on subsequent performance.

DISCUSSION

Our test's recommendations highlighted the benefits of simple updating as opposed to systematic refitting in response to performance drift. The complexity of recommended updating methods reflected sample size and magnitude of performance drift, as anticipated. The case study highlights the conservative nature of our test.

CONCLUSIONS

This new test supports data-driven updating of models developed with both biostatistical and machine learning approaches, promoting the transportability and maintenance of a wide array of clinical prediction models and, in turn, a variety of applications relying on modern prediction tools.

Comparison of Machine Learning Techniques for Prediction of Hospitalization in Heart Failure Patients

The present study aims to compare the performance of eight Machine Learning Techniques (MLTs) in the prediction of hospitalization among patients with heart failure, using data from the Gestione Integrata dello Scompenso Cardiaco (GISC) study. The GISC project is an ongoing study that takes place in the region of Puglia, Southern Italy. Patients with a diagnosis of heart failure are enrolled in a long-term assistance program that includes the adoption of an online platform for data sharing between general practitioners and cardiologists working in hospitals and community health districts. Logistic regression, generalized linear model net (GLMN), classification and regression tree, random forest, adaboost, logitboost, support vector machine, and neural networks were applied to evaluate the feasibility of such techniques in predicting hospitalization of 380 patients enrolled in the GISC study, using data about demographic characteristics, medical history, and clinical characteristics of each patient. The MLTs were compared both without and with missing data imputation. Overall, models trained without missing data imputation showed higher predictive performances. The GLMN showed better performance in predicting hospitalization than the other MLTs, with an average accuracy, positive predictive value and negative predictive value of 81.2%, 87.5%, and 75%, respectively. Present findings suggest that MLTs may represent a promising opportunity to predict hospital admission of heart failure patients by exploiting health care information generated by the contact of such patients with the health care system.

Fundamentals of Clinical Prediction Modeling for the Neurosurgeon

Clinical prediction models in neurosurgery are increasingly reported. These models aim to provide an evidence-based approach to the estimation of the probability of a neurosurgical outcome by combining 2 or more prognostic variables. Model development and model reporting are often suboptimal. A basic understanding of the methodology of clinical prediction modeling is needed when interpreting these models. We address basic statistical background, 7 modeling steps, and requirements of these models such that they may fulfill their potential for major impact for our daily clinical practice and for future scientific work.

A cardiovascular risk prediction model for older people: Development and validation in a primary care population

Cardiovascular risk prediction is mainly based on traditional risk factors that have been validated in middle‐aged populations. However, associations between these risk factors and cardiovascular disease (CVD) attenuate with increasing age. Therefore, for older people the authors developed and internally validated risk prediction models for fatal and non‐fatal CVD, (re)evaluated the predictive value of traditional and new factors, and assessed the impact of competing risks of non‐cardiovascular death. Post hoc analyses of 1811 persons aged 70‐78 year and free from CVD at baseline from the preDIVA study (Prevention of Dementia by Intensive Vascular care, 2006‐2015), a primary care‐based trial that included persons free from dementia and conditions likely to hinder successful long‐term follow‐up, were performed. In 2017‐2018, Cox‐regression analyses were performed for a model including seven traditional risk factors only, and a model to assess incremental predictive ability of the traditional and eleven new factors. Analyses were repeated accounting for competing risk of death, using Fine‐Gray models. During an average of 6.2 years of follow‐up, 277 CVD events occurred. Age, sex, smoking, and type 2 diabetes mellitus were traditional predictors for CVD, whereas total cholesterol, HDL‐cholesterol, and systolic blood pressure (SBP) were not. Of the eleven new factors, polypharmacy and apathy symptoms were predictors. Discrimination was moderate (concordance statistic 0.65). Accounting for competing risks resulted in slightly smaller predicted absolute risks. In conclusion, we found, SBP, HDL, and total cholesterol no longer predict CVD in older adults, whereas polypharmacy and apathy symptoms are two new relevant predictors. Building on the selected risk factors in this study may improve CVD prediction in older adults and facilitate targeting preventive interventions to those at high risk.

External validation of the International Risk Prediction Algorithm for the onset of generalized anxiety and/or panic syndromes (The Predict A) in the US general population

INTRODUCTION

Multivariable risk prediction algorithms are useful for making clinical decisions and health planning. While prediction algorithms for new onset of anxiety disorders in Europe and elsewhere have been developed, the performance of these algorithms in the Americas is not known. The objective of this study was to validate the PredictA algorithm for new onset of anxiety and/or panic disorders in the US general population.

METHODS

Longitudinal study design was conducted with approximate 2-year follow-up data from a total of 24 626 individuals who participated in Wave 1 and 2 of the US National Epidemiologic Survey on Alcohol and Related Conditions (NESARC) and who did not have generalized anxiety disorder (GAD) and panic disorder in the past year at Wave 1. The PredictA algorithm was directly applied to the selected participants.

RESULTS

Among the participants, 5.4% developed GAD and/or panic disorder over two years. The PredictA algorithm had a discriminative power (C-statistics = 0.62, 95%CI: 0.61; 0.64), but poor calibration (p < 0.001) with the NESARC data. The observed and the mean predicted risk of GAD and/or panic disorders in the NESARC were 5.3% and 3.6%, respectively. Particularly, the observed and predicted risks of GAD and/or panic disorders in the highest decile of risk score in the NESARC participants were 13.3% and 10.4%, respectively.

CONCLUSION

The PredictA algorithm has acceptable discrimination, but the calibration with the NESARC data was poor. The PredictA algorithm is likely to underestimate the risk of GAD/panic disorders in the US population. Therefore, the use of PredictA in the US general population for predicting individual risk of GAD and/or panic disorders is not encouraged.

Exploration, Inference, and Prediction in Neuroscience and Biomedicine

Recent decades have seen dramatic progress in brain research. These advances were often buttressed by probing single variables to make circumscribed discoveries, typically through null hypothesis significance testing. New ways for generating massive data fueled tension between the traditional methodology that is used to infer statistically relevant effects in carefully chosen variables, and pattern-learning algorithms that are used to identify predictive signatures by searching through abundant information. In this article we detail the antagonistic philosophies behind two quantitative approaches: certifying robust effects in understandable variables, and evaluating how accurately a built model can forecast future outcomes. We discourage choosing analytical tools via categories such as 'statistics' or 'machine learning'. Instead, to establish reproducible knowledge about the brain, we advocate prioritizing tools in view of the core motivation of each quantitative analysis: aiming towards mechanistic insight or optimizing predictive accuracy.

Replicability of Machine Learning Models in the Social Sciences

Abstract. Machine learning tools are increasingly used in social sciences and policy fields due to their increase in predictive accuracy. However, little research has been done on how well the models of machine learning methods replicate across samples. We compare machine learning methods with regression on the replicability of variable selection, along with predictive accuracy, using an empirical dataset as well as simulated data with additive, interaction, and non-linear squared terms added as predictors. Methods analyzed include support vector machines (SVM), random forests (RF), multivariate adaptive regression splines (MARS), and the regularized regression variants, least absolute shrinkage and selection operator (LASSO), and elastic net. In simulations with additive and linear interactions, machine learning methods performed similarly to regression in replicating predictors; they also performed mostly equal or below regression on measures of predictive accuracy. In simulations with square terms, machine learning methods SVM, RF, and MARS improved predictive accuracy and replicated predictors better than regression. Thus, in simulated datasets, the gap between machine learning methods and regression on predictive measures foreshadowed the gap in variable selection. In replications on the empirical dataset, however, improved prediction by machine learning methods was not accompanied by a visible improvement in replicability in variable selection. This disparity is explained by the overall explanatory power of the models. When predictors have small effects and noise predominates, improved global measures of prediction in a sample by machine learning methods may not lead to the robust selection of predictors; thus, in the presence of weak predictors and noise, regression remains a useful tool for model building and replication.

Supervised Machine-learning Predictive Analytics for Prediction of Postinduction Hypotension

Editor’s Perspective

What We Already Know about This Topic

What This Article Tells Us That Is New

Background

Hypotension is a risk factor for adverse perioperative outcomes. Machine-learning methods allow large amounts of data for development of robust predictive analytics. The authors hypothesized that machine-learning methods can provide prediction for the risk of postinduction hypotension.

Methods

Data was extracted from the electronic health record of a single quaternary care center from November 2015 to May 2016 for patients over age 12 that underwent general anesthesia, without procedure exclusions. Multiple supervised machine-learning classification techniques were attempted, with postinduction hypotension (mean arterial pressure less than 55 mmHg within 10 min of induction by any measurement) as primary outcome, and preoperative medications, medical comorbidities, induction medications, and intraoperative vital signs as features. Discrimination was assessed using cross-validated area under the receiver operating characteristic curve. The best performing model was tuned and final performance assessed using split-set validation.

Results

Out of 13,323 cases, 1,185 (8.9%) experienced postinduction hypotension. Area under the receiver operating characteristic curve using logistic regression was 0.71 (95% CI, 0.70 to 0.72), support vector machines was 0.63 (95% CI, 0.58 to 0.60), naive Bayes was 0.69 (95% CI, 0.67 to 0.69), k-nearest neighbor was 0.64 (95% CI, 0.63 to 0.65), linear discriminant analysis was 0.72 (95% CI, 0.71 to 0.73), random forest was 0.74 (95% CI, 0.73 to 0.75), neural nets 0.71 (95% CI, 0.69 to 0.71), and gradient boosting machine 0.76 (95% CI, 0.75 to 0.77). Test set area for the gradient boosting machine was 0.74 (95% CI, 0.72 to 0.77).

Conclusions

The success of this technique in predicting postinduction hypotension demonstrates feasibility of machine-learning models for predictive analytics in the field of anesthesiology, with performance dependent on model selection and appropriate tuning.

Calibration drift in regression and machine learning models for acute kidney injury

Objective

Predictive analytics create opportunities to incorporate personalized risk estimates into clinical decision support. Models must be well calibrated to support decision-making, yet calibration deteriorates over time. This study explored the influence of modeling methods on performance drift and connected observed drift with data shifts in the patient population.

Materials and Methods

Using 2003 admissions to Department of Veterans Affairs hospitals nationwide, we developed 7 parallel models for hospital-acquired acute kidney injury using common regression and machine learning methods, validating each over 9 subsequent years.

Results

Discrimination was maintained for all models. Calibration declined as all models increasingly overpredicted risk. However, the random forest and neural network models maintained calibration across ranges of probability, capturing more admissions than did the regression models. The magnitude of overprediction increased over time for the regression models while remaining stable and small for the machine learning models. Changes in the rate of acute kidney injury were strongly linked to increasing overprediction, while changes in predictor-outcome associations corresponded with diverging patterns of calibration drift across methods.

Conclusions

Efficient and effective updating protocols will be essential for maintaining accuracy of, user confidence in, and safety of personalized risk predictions to support decision-making. Model updating protocols should be tailored to account for variations in calibration drift across methods and respond to periods of rapid performance drift rather than be limited to regularly scheduled annual or biannual intervals.

Moving beyond regression techniques in cardiovascular risk prediction: applying machine learning to address analytic challenges

Risk prediction plays an important role in clinical cardiology research. Traditionally, most risk models have been based on regression models. While useful and robust, these statistical methods are limited to using a small number of predictors which operate in the same way on everyone, and uniformly throughout their range. The purpose of this review is to illustrate the use of machine-learning methods for development of risk prediction models. Typically presented as black box approaches, most machine-learning methods are aimed at solving particular challenges that arise in data analysis that are not well addressed by typical regression approaches. To illustrate these challenges, as well as how different methods can address them, we consider trying to predicting mortality after diagnosis of acute myocardial infarction. We use data derived from our institution's electronic health record and abstract data on 13 regularly measured laboratory markers. We walk through different challenges that arise in modelling these data and then introduce different machine-learning approaches. Finally, we discuss general issues in the application of machine-learning methods including tuning parameters, loss functions, variable importance, and missing data. Overall, this review serves as an introduction for those working on risk modelling to approach the diffuse field of machine learning.

Prediction of transition to psychosis in patients with a clinical high risk for psychosis: a systematic review of methodology and reporting

BACKGROUND

To enhance indicated prevention in patients with a clinical high risk (CHR) for psychosis, recent research efforts have been increasingly directed towards estimating the risk of developing psychosis on an individual level using multivariable clinical prediction models. The aim of this study was to systematically review the methodological quality and reporting of studies developing or validating such models.

METHOD

A systematic literature search was carried out (up to 14 March 2016) to find all studies that developed or validated a clinical prediction model predicting the transition to psychosis in CHR patients. Data were extracted using a comprehensive item list which was based on current methodological recommendations.

RESULTS

A total of 91 studies met the inclusion criteria. None of the retrieved studies performed a true external validation of an existing model. Only three studies (3.5%) had an event per variable ratio of at least 10, which is the recommended minimum to avoid overfitting. Internal validation was performed in only 14 studies (15%) and seven of these used biased internal validation strategies. Other frequently observed modeling approaches not recommended by methodologists included univariable screening of candidate predictors, stepwise variable selection, categorization of continuous variables, and poor handling and reporting of missing data.

CONCLUSIONS

Our systematic review revealed that poor methods and reporting are widespread in prediction of psychosis research. Since most studies relied on small sample sizes, did not perform internal or external cross-validation, and used poor model development strategies, most published models are probably overfitted and their reported predictive accuracy is likely to be overoptimistic.