A comparative study of forest methods for time-to-event data: variable selection and predictive performance

Comparison between traditional logistic regression and machine learning for predicting mortality in adult sepsis patients

Background

Sepsis is a life-threatening disease associated with a high mortality rate, emphasizing the need for the exploration of novel models to predict the prognosis of this patient population. This study compared the performance of traditional logistic regression and machine learning models in predicting adult sepsis mortality.

Objective

To develop an optimum model for predicting the mortality of adult sepsis patients based on comparing traditional logistic regression and machine learning methodology.

Methods

Retrospective analysis was conducted on 606 adult sepsis inpatients at our medical center between January 2020 and December 2022, who were randomly divided into training and validation sets in a 7:3 ratio. Traditional logistic regression and machine learning methods were employed to assess the predictive ability of mortality in adult sepsis. Univariate analysis identified independent risk factors for the logistic regression model, while Least Absolute Shrinkage and Selection Operator (LASSO) regression facilitated variable shrinkage and selection for the machine learning model. Among various machine learning models, which included Bagged Tree, Boost Tree, Decision Tree, LightGBM, Naïve Bayes, Nearest Neighbors, Support Vector Machine (SVM), and Random Forest (RF), the one with the maximum area under the curve (AUC) was chosen for model construction. Model validation and comparison with the Sequential Organ Failure Assessment (SOFA) and the Acute Physiology and Chronic Health Evaluation (APACHE) scores were performed using receiver operating characteristic (ROC) curves, calibration curves, and decision curve analysis (DCA) curves in the validation set.

Results

Univariate analysis was employed to assess 17 variables, namely gender, history of coronary heart disease (CHD), systolic pressure, white blood cell (WBC), neutrophil count (NEUT), lymphocyte count (LYMP), lactic acid, neutrophil-to-lymphocyte ratio (NLR), red blood cell distribution width (RDW), interleukin-6 (IL-6), prothrombin time (PT), international normalized ratio (INR), fibrinogen (FBI), D-dimer, aspartate aminotransferase (AST), total bilirubin (Tbil), and lung infection. Significant differences (p < 0.05) between the survival and non-survival groups were observed for these variables. Utilizing stepwise regression with the "backward" method, independent risk factors, including systolic pressure, lactic acid, NLR, RDW, IL-6, PT, and Tbil, were identified. These factors were then incorporated into a logistic regression model, chosen based on the minimum Akaike Information Criterion (AIC) value (98.65). Machine learning techniques were also applied, and the RF model, demonstrating the maximum Area Under the Curve (AUC) of 0.999, was selected. LASSO regression, employing the lambda.1SE criteria, identified systolic pressure, lactic acid, NEUT, RDW, IL6, INR, and Tbil as variables for constructing the RF model, validated through ten-fold cross-validation. For model validation and comparison with traditional logistic models, SOFA, and APACHE scoring.

Conclusion

Based on deep machine learning principles, the RF model demonstrates advantages over traditional logistic regression models in predicting adult sepsis prognosis. The RF model holds significant potential for clinical surveillance and interventions to enhance outcomes for sepsis patients.

Machine learning algorithms that predict the risk of prostate cancer based on metabolic syndrome and sociodemographic characteristics: a prospective cohort study

BACKGROUND

Given the rapid increase in the prevalence of prostate cancer (PCa), identifying its risk factors and developing suitable risk prediction models has important implications for public health. We used machine learning (ML) approach to screen participants with high risk of PCa and, specifically, investigated whether participants with metabolic syndrome (MetS) exhibited an elevated PCa risk.

METHODS

A prospective cohort study was performed with 41,837 participants in South Korea. We predicted PCa based on MetS, its components, and sociodemographic factors using Cox proportional hazards and five ML models. Integrated Brier score (IBS) and C-index were used to assess model performance.

RESULTS

A total of 210 incident PCa cases were identified. We found good calibration and discrimination for all models (C-index ≥ 0.800 and IBS = 0.01). Importantly, performance increased after excluding MetS and its components from the models; the highest C-index was 0.862 for survival support vector machine. In contrast, first-degree family history of PCa, alcohol consumption, age, and income were valuable for PCa prediction.

CONCLUSION

ML models are an effective approach to develop prediction models for survival analysis. Furthermore, MetS and its components do not seem to influence PCa susceptibility, in contrast to first-degree family history of PCa, age, alcohol consumption, and income.

Predicting deep infection in pilon and tibial plateau fractures: a secondary analysis of the VANCO and OXYGEN trials

Objectives

To develop and validate a prediction model for a deep surgical site infection (SSI) after fixation of a tibial plateau or pilon fracture.

Design

Pooled data from 2 randomized trials (VANCO and OXYGEN).

Setting

Fifty-two US trauma centers.

Patients

In total, 1847 adult patients with operatively treated tibial plateau or pilon fractures who met criteria for a high risk of infection.

Intervention

We considered 13 baseline patient characteristics and developed and externally validated prediction models using 3 approaches (logistic regression, stepwise elimination, and machine learning).

Main Outcomes and Measures

The primary prediction model outcome was a deep SSI requiring operative debridement within 182 days of definitive fixation. Our primary prognostic performance metric for evaluating the models was area under the receiver operating characteristic curve (AUC) with clinical utility set at 0.7.

Results

Deep SSI occurred in 75 VANCO patients (8%) and in 56 OXYGEN patients (6%). The machine learning model for VANCO (AUC = 0.65) and stepwise elimination model for OXYGEN (AUC = 0.62) had the highest internal validation AUCs. However, none of the external validation AUCs exceeded 0.64 (range, 0.58 to 0.64).

Conclusions

The predictive models did not reach the prespecified clinical utility threshold. Our models' inability to distinguish high-risk from low-risk patients is likely due to strict eligibility criteria and, therefore, homogeneous patient populations.

Survival analysis in breast cancer: evaluating ensemble learning techniques for prediction

Breast cancer is most commonly faced with form of cancer amongst women worldwide. In spite of the fact that the breast cancer research and awareness have gained considerable momentum, there is still no one treatment due to disease heterogeneity. Survival data may be of specific interest in breast cancer studies to understand its dynamic and complex trajectories. This study copes with the most important covariates affecting the disease progression. The study utilizes the German Breast Cancer Study Group 2 (GBSG2) and the Molecular Taxonomy of Breast Cancer International Consortium dataset (METABRIC) datasets. In both datasets, interests lie in relapse of the disease and the time when the relapse happens. The three models, namely the Cox proportional hazards (PH) model, random survival forest (RSF) and conditional inference forest (Cforest) were employed to analyse the breast cancer datasets. The goal of this study is to apply these methods in prediction of breast cancer progression and compare their performances based on two different estimation methods: the bootstrap estimation and the bootstrap .632 estimation. The model performance was evaluated in concordance index (C-index) and prediction error curves (pec) for discrimination. The Cox PH model has a lower C-index and bigger prediction error compared to the RSF and the Cforest approach for both datasets. The analysis results of GBSG2 and METABRIC datasets reveal that the RSF and the Cforest algorithms provide non-parametric alternatives to Cox PH model for estimation of the survival probability of breast cancer patients.

Time to progression is the main predictor of survival in patients with high-risk nonmuscle invasive bladder cancer: Results from a machine learning-based analysis of a large multi-institutional database

BACKGROUND

In patients affected by high-risk nonmuscle invasive bladder cancer (HR-NMIBC) progression to muscle invasive status is considered as the main indicator of local treatment failure. We aimed to investigate the effect of progression and time to progression on overall survival (OS) and to investigate their validity as surrogate endpoints.

METHODS

A total of 1,510 patients from 18 different institutions treated for T1 high grade NMIBC, followed by a secondary transurethral resection and BCG intravesical instillation. We relied on random survival forest (RSF) to rank covariates based on OS prediction. Cox's regression models were used to quantify the effect of covariates on mortality.

RESULTS

During a median follow-up of 49.0 months, 485 (32.1%) patients progressed to MIBC, while 163 (10.8%) patients died. The median time to progression was 82 (95%CI: 78.0-93.0) months. In RSF time-to-progression and age were the most predictive covariates of OS. The survival tree defined 5 groups of risk. In multivariable Cox's regression models accounting for progression status as time-dependent covariate, shorter time to progression (as continuous covariate) was associated with longer OS (HR: 9.0, 95%CI: 3.0-6.7; P < 0.001). Virtually same results after time to progression stratification (time to progression ≥10.5 months as reference).

CONCLUSION

Time to progression is the main predictor of OS in patients with high risk NMIBC treated with BCG and might be considered a coprimary endpoint. In addition, models including time to progression could be considered for patients' stratification in clinical practice and at the time of clinical trials design.

Use of survival support vector machine combined with random survival forest to predict the survival of nasopharyngeal carcinoma patients

Background

The Cox regression model is not sufficiently accurate to predict the survival prognosis of nasopharyngeal carcinoma (NPC) patients. It is impossible to calculate and rank the importance of impact factors due to the low predictive accuracy of the Cox regression model. So, we developed a system. Using the SEER (The Surveillance, Epidemiology, and End Results) database data on NPC patients, we proposed the use of random survival forest (RSF) and survival-support vector machine (SVM) from the machine learning methods to develop a survival prediction system specifically for NPC patients. This approach aimed to make up for the insufficiency of the Cox regression model. We also used the Cox regression model to validate the development of the nomogram and compared it with machine learning methods.

Methods

A total of 1,683 NPC patients were extracted from the SEER database from January 2010 to December 2015. We used R language for modeling work, established the nomogram of survival prognosis of NPC patients by Cox regression model, ranked the correlation of influencing factors by RSF model VIMP (variable important) method, developed a survival prognosis system for NPC patients based on survival-SVM, and used C-index for model evaluation and performance comparison.

Results

Although the Cox regression models can be developed to predict the prognosis of NPC patients, their accuracy was lower than that of machine learning methods. When we substituted the data for the Cox model, the C-index for the training set was only 0.740, and the C-index for the test set was 0.721. In contrast, the C index of the survival-SVM model was 0.785. The C-index of the RSF model was 0.729. The importance ranking of each variable could be obtained according to the VIMP method.

Conclusions

The prediction results from the Cox model are not as good as those of the RSF method and survival-SVM based on the machine learning method. For the survival prognosis of NPC patients, the machine learning method can be considered for clinical application.

A comparison of machine learning models and Cox proportional hazards models regarding their ability to predict the risk of gastrointestinal cancer based on metabolic syndrome and its components

Background

Little is known about applying machine learning (ML) techniques to identify the important variables contributing to the occurrence of gastrointestinal (GI) cancer in epidemiological studies. We aimed to compare different ML models to a Cox proportional hazards (CPH) model regarding their ability to predict the risk of GI cancer based on metabolic syndrome (MetS) and its components.

Methods

A total of 41,837 participants were included in a prospective cohort study. Incident cancer cases were identified by following up with participants until December 2019. We used CPH, random survival forest (RSF), survival trees (ST), gradient boosting (GB), survival support vector machine (SSVM), and extra survival trees (EST) models to explore the impact of MetS on GI cancer prediction. We used the C-index and integrated Brier score (IBS) to compare the models.

Results

In all, 540 incident GI cancer cases were identified. The GB and SSVM models exhibited comparable performance to the CPH model concerning the C-index (0.725). We also recorded a similar IBS for all models (0.017). Fasting glucose and waist circumference were considered important predictors.

Conclusions

Our study found comparably good performance concerning the C-index for the ML models and CPH model. This finding suggests that ML models may be considered another method for survival analysis when the CPH model's conditions are not satisfied.