The Performance of Risk Prediction Models

Aortic stiffness: an age-related prognostic marker?

Aortic stiffness, a consequence of vascular aging, is an independent predictor of cardiovascular morbidity and mortality. However, the impact of age and sex on its predictive performance remains unclear. We have included 6046 individuals from the population-based Rotterdam study. Survival analyses were performed to investigate the impact of age and sex on the link between aortic stiffness and outcomes, including coronary heart disease (CHD), stroke, cardiovascular disease (CVD), cardiovascular and all-cause mortality. The added predictive value of aortic stiffness across age categories and by sex was assessed by using explained variation, Harrell's C index and Integrated Discrimination Improvement (IDI). Aortic stiffness was independently associated with all outcomes [hazard ratio (95% confidence interval; CI): 1.16 (1.04-1.22) for CHD, 1.09 (1.00-1.19) for stroke, 1.11 (1.05-1.18) for CVD, 1.14 (1.05-1.23) for cardiovascular mortality, 1.08 (1.03-1.13) for all-cause mortality]. The strength of the association between aortic stiffness and stroke, cardiovascular and all-cause mortality decreased significantly by advancing age. The variance of the outcome (R2) explained by aortic stiffness alone was noticeable in individuals younger than 60 years and negligible in the other age categories. The association of aortic stiffness and CHD was stronger in women than in men. Similarly, the difference in R2 between women and men was greater for CHD than for the other considered outcomes. Our findings suggest that the gain in explained variation caused by aortic stiffness for CVD and mortality might be limited to individuals younger than 60 years.

Utilizing a comprehensive machine learning approach to identify patients at high risk for extended length of stay following spinal deformity surgery in pediatric patients with early onset scoliosis

PURPOSE

Early onset scoliosis (EOS) patient diversity makes outcome prediction challenging. Machine learning offers an innovative approach to analyze patient data and predict results, including LOS in pediatric spinal deformity surgery.

METHODS

Children under 10 with EOS were chosen from the American College of Surgeon's NSQIP database. Extended LOS, defined as over 5 days, was predicted using feature selection and machine learning in Python. The best model, determined by the area under the curve (AUC), was optimized and used to create a risk calculator for prolonged LOS.

RESULTS

The study included 1587 patients, mostly young (average age: 6.94 ± 2.58 years), with 33.1% experiencing prolonged LOS (n = 526). Most patients were female (59.2%, n = 940), with an average BMI of 17.0 ± 8.7. Factors influencing LOS were operative time, age, BMI, ASA class, levels operated on, etiology, nutritional support, pulmonary and neurologic comorbidities. The gradient boosting model performed best with a test accuracy of 0.723, AUC of 0.630, and a Brier score of 0.189, leading to a patient-specific risk calculator for prolonged LOS.

CONCLUSIONS

Machine learning algorithms accurately predict extended LOS across a national patient cohort and characterize key preoperative drivers of increased LOS after PSIF in pediatric patients with EOS.

Identifying potentially depressed older Chinese adults in the community: Hong Kong's Elderly Health Service cohort

BACKGROUND

Depression is common at older ages, but is under-recognized due to stigma, misperception, and under-diagnosis; its manifestations may vary by setting. Identifying older adults at risk of depression in the community is urgently needed for timely support and early interventions. We assessed the performance of an existing risk prediction model developed in a European setting (i.e., Depression Risk Assessment Tool (DRAT-up)), and developed a new model (i.e., EHS-Depress model) to predict 2-year risk of the onset of later life depressive symptoms in older Chinese adults.

METHODS

Among 185,538 participants aged ≥65 years from Hong Kong's Elderly Health Service (EHS) cohort, 174,806 without depressive symptoms at baseline were included. Two-thirds were randomly sampled for recalibration and new model development using Cox proportional-hazards models with backward elimination. Overall predictive performance, discrimination, and calibration were assessed using the remaining.

RESULTS

The original DRAT-up model underestimated the risk of developing depressive symptoms in older Chinese adults; recalibrating it improved calibration. The new EHS-Depress model had better discrimination (Harrell's C statistic 0.68 and D statistic 2.74) and similarly good calibration (calibration slope 1.18 and intercept -0.002) probably due to the inclusion of more specific health measures, socio-demographics, lifestyle factors, and regular social contact as predictors.

LIMITATIONS

Predictors of depressive symptoms included in our models depend on the data availability.

CONCLUSIONS

The EHS-Depress model predicted 2-year risk of developing depressive symptoms better than the original and recalibrated DRAT-up models. The setting-specific risk prediction model is more applicable to older Chinese adults in primary care settings.

Development and Validation of a User Friendly Morphology Grading System (PATENT) Predicting Aortic Remodelling After Thoracic Endovascular Aortic Repair in High Risk Uncomplicated Type B Aortic Dissection

OBJECTIVE

This study aimed to create a morphology grading system, solely based on 2D images from computed tomography angiography, to predict negative aortic remodelling (NAR) for patients with high risk uncomplicated type B aortic dissection (TBAD) after thoracic endovascular aortic repair (TEVAR).

METHODS

This single centre retrospective cohort study extracted and analysed consecutive patients diagnosed with high risk uncomplicated TBAD. Negative aortic remodelling was defined as an increase in the false lumen or total aortic diameter, or decrease in the true lumen diameter. The multivariable Cox regression model identified risk factors and a prediction model was created for two year freedom from NAR. A three category grading system, in which patients were classified into low, medium, and high risk groups, was further developed and internally validated.

RESULTS

Of 351 patients included, 99 (28%) developed NAR. The median age was 52 years (interquartile range 45, 62 years) and 56 (16%) were female. The rate of two year freedom from NAR was 71% (95% CI 65 - 77%). After the multivariable Cox regression analysis, Patent false lumen, Aberrant right subclavian artery, Taper ratio, abdominal circumferential Extent, coeliac artery or reNal artery involved, and four channel dissection (Three false lumens) remained independent predictors and were included in the PATENT grading system. The risk score was statistically significantly associated with NAR (HR 1.21; 95% CI 1.14 - 1.29; p < .001). The medium and high risk groups demonstrated a higher rate of NAR (medium risk, HR 2.82; 95% CI 1.57 - 5.01; p = .001; high risk, HR 4.39; 95% CI 2.58 - 7.48; p < .001). The grading system was characterised by robust discrimination with Harrell's C index of 0.68 (95% CI 0.63 - 0.75).

CONCLUSION

The PATENT grading system was characterised by good discrimination and calibration, which may serve as a clinician friendly tool to aid risk stratification for TBAD patients after TEVAR.

Development and Validation of the Hospital Medicine Safety Sepsis Initiative Mortality Model

BACKGROUND

When comparing outcomes after sepsis, it is essential to account for patient case mix to make fair comparisons. We developed a model to assess risk-adjusted 30-day mortality in the Michigan Hospital Medicine Safety sepsis initiative (HMS-Sepsis).

RESEARCH QUESTION

Can HMS-Sepsis registry data adequately predict risk of 30-day mortality? Do performance assessments using adjusted vs unadjusted data differ?

STUDY DESIGN AND METHODS

Retrospective cohort of community-onset sepsis hospitalizations in the HMS-Sepsis registry (April 2022-September 2023), with split-derivation (70%) and validation (30%) cohorts. We fit a risk-adjustment model (HMS-Sepsis mortality model) incorporating acute physiologic, demographic, and baseline health data and assessed model performance using concordance (C) statistics, Brier's scores, and comparisons of predicted vs observed mortality by deciles of risk. We compared hospital performance (first quintile, middle quintiles, fifth quintile) using observed vs adjusted mortality to understand the extent to which risk adjustment impacted hospital performance assessment.

RESULTS

Among 17,514 hospitalizations from 66 hospitals during the study period, 12,260 hospitalizations (70%) were used for model derivation and 5,254 hospitalizations (30%) were used for model validation. Thirty-day mortality for the total cohort was 19.4%. The final model included 13 physiologic variables, two physiologic interactions, and 16 demographic and chronic health variables. The most significant variables were age, metastatic solid tumor, temperature, altered mental status, and platelet count. The model C statistic was 0.82 for the derivation cohort, 0.81 for the validation cohort, and ≥ 0.78 for all subgroups assessed. Overall calibration error was 0.0%, and mean calibration error across deciles of risk was 1.5%. Standardized mortality ratios yielded different assessments than observed mortality for 33.9% of hospitals.

INTERPRETATION

The HMS-Sepsis mortality model showed strong discrimination and adequate calibration and reclassified one-third of hospitals to a different performance category from unadjusted mortality. Based on its strong performance, the HMS-Sepsis mortality model can aid in fair hospital benchmarking, assessment of temporal changes, and observational causal inference analysis.

Development and validation of the Ex-Care BR model: a multicentre initiative for identifying Brazilian surgical patients at risk of 30-day in-hospital mortality

BACKGROUND

Surgical risk stratification is crucial for enhancing perioperative assistance and allocating resources efficiently. However, existing models may not capture the complexity of surgical care in Brazil. Using data from various healthcare settings nationwide, we developed a new risk model for 30-day in-hospital mortality (the Ex-Care BR model).

METHODS

A retrospective cohort study was conducted in 10 hospitals from different geographic regions in Brazil. Data were analysed using multilevel logistic regression models. Model performance was assessed using the area under the receiver operating characteristic curve (AUROC), Brier score, and calibration plots. Derivation and validation cohorts were randomly assigned.

RESULTS

A total of 107,372 patients were included, and 30-day in-hospital mortality was 2.1% (n=2261). The final risk model comprised four predictors related to the patient and surgery (age, ASA physical status classification, surgical urgency, and surgical size), and the random effect related to hospitals. The model showed excellent discrimination (AUROC=0.93, 95% confidence interval [CI], 0.93-0.94), calibration, and overall performance (Brier score=0.017) in the derivation cohort (n=75,094). Similar results were observed in the validation cohort (n=32,278) (AUROC=0.93, 95% CI, 0.92-0.93).

CONCLUSIONS

The Ex-Care BR is the first model to consider regional and organisational peculiarities of the Brazilian surgical scene, in addition to patient and surgical factors. It is particularly useful for identifying high-risk surgical patients in situations demanding efficient allocation of limited resources. However, a thorough exploration of mortality variations among hospitals is essential for a comprehensive understanding of risk.

CLINICAL TRIAL REGISTRATION

NCT05796024.

Acute Kidney Injury After Heart Transplantation: Risk Factors and Clinical Outcomes

OBJECTIVE

Acute kidney injury (AKI) requiring renal-replacement therapy (RRT) after heart transplantation (OHT) is common and impairs outcomes. This study aimed to identify independent donor and recipient risk factors associated with RRT after OHT.

DESIGN

A retrospective data analysis.

SETTING

Data were collected from clinical routines in a maximum-care university hospital.

PARTICIPANTS

Patients who underwent OHT.

INTERVENTIONS

The authors retrospectively analyzed data from 264 patients who underwent OHT between 2012 and 2021; 189 patients were eligible and included in the final analysis.

MEASUREMENTS AND MAIN RESULTS

The mean age was 48.0 ± 12.3 years, and 71.4% of patients were male. Ninety (47.6%) patients were on long-term mechanical circulatory support (lt-MCS). Posttransplant AKI with RRT occurred in 123 (65.1%) patients. In a multivariate analysis, preoperative body mass index >25 kg/m² (odds ratio [OR] 4.74, p < 0.001), elevated preoperative creatinine levels (OR for each mg/dL increase 3.44, p = 0.004), administration of red blood cell units during transplantation procedure (OR 2.31, p = 0.041) and ischemia time (OR for each hour increase 1.77, p = 0.004) were associated with a higher incidence of RRT. The use of renin-angiotensin-aldosterone system blockers before transplantation was associated with a reduced risk of RRT (OR 0.36, p = 0.013). The risk of mortality was 6.9-fold higher in patients who required RRT (hazard ratio 6.9, 95% CI: 2.1-22.6 p = 0.001). Previous lt-MCS, as well as donor parameters, were not associated with RRT after OHT.

CONCLUSIONS

The implementation of guideline-directed medical therapy, weight reduction, minimizing ischemia time (ie, organ perfusion systems, workflow optimization), and comprehensive patient blood management potentially influences renal function and outcomes after OHT.

Evaluation of short-term mortality in patients with Medicare undergoing endovascular interventions for chronic limb-threatening ischemia

INTRODUCTION

Patients with chronic limb-threatening ischemia (CLTI) have high mortality rates after revascularization. Risk stratification for short-term outcomes is challenging. We aimed to develop machine-learning models to rank predictive variables for 30-day and 90-day all-cause mortality after peripheral vascular intervention (PVI).

METHODS

Patients undergoing PVI for CLTI in the Medicare-linked Vascular Quality Initiative were included. Sixty-six preprocedural variables were included. Random survival forest (RSF) models were constructed for 30-day and 90-day all-cause mortality in the training sample and evaluated in the testing sample. Predictive variables were ranked based on the frequency that they caused branch splitting nearest the root node by importance-weighted relative importance plots. Model performance was assessed by the Brier score, continuous ranked probability score, out-of-bag error rate, and Harrell's C-index.

RESULTS

A total of 10,114 patients were included. The crude mortality rate was 4.4% at 30 days and 10.6% at 90 days. RSF models commonly identified stage 5 chronic kidney disease (CKD), dementia, congestive heart failure (CHF), age, urgent procedures, and need for assisted care as the most predictive variables. For both models, eight of the top 10 variables were either medical comorbidities or functional status variables. Models showed good discrimination (C-statistic 0.72 and 0.73) and calibration (Brier score 0.03 and 0.10).

CONCLUSION

RSF models for 30-day and 90-day all-cause mortality commonly identified CKD, dementia, CHF, need for assisted care at home, urgent procedures, and age as the most predictive variables as critical factors in CLTI. Results may help guide individualized risk-benefit treatment conversations regarding PVI.

Development and internal validation of machine learning-based models and external validation of existing risk scores for outcome prediction in patients with ischaemic stroke

Aims

We developed new machine learning (ML) models and externally validated existing statistical models [ischaemic stroke predictive risk score (iScore) and totalled health risks in vascular events (THRIVE) scores] for predicting the composite of recurrent stroke or all-cause mortality at 90 days and at 3 years after hospitalization for first acute ischaemic stroke (AIS).

Methods and results

In adults hospitalized with AIS from January 2005 to November 2016, with follow-up until November 2019, we developed three ML models [random forest (RF), support vector machine (SVM), and extreme gradient boosting (XGBOOST)] and externally validated the iScore and THRIVE scores for predicting the composite outcomes after AIS hospitalization, using data from 721 patients and 90 potential predictor variables. At 90 days and 3 years, 11 and 34% of patients, respectively, reached the composite outcome. For the 90-day prediction, the area under the receiver operating characteristic curve (AUC) was 0.779 for RF, 0.771 for SVM, 0.772 for XGBOOST, 0.720 for iScore, and 0.664 for THRIVE. For 3-year prediction, the AUC was 0.743 for RF, 0.777 for SVM, 0.773 for XGBOOST, 0.710 for iScore, and 0.675 for THRIVE.

Conclusion

The study provided three ML-based predictive models that achieved good discrimination and clinical usefulness in outcome prediction after AIS and broadened the application of the iScore and THRIVE scoring system for long-term outcome prediction. Our findings warrant comparative analyses of ML and existing statistical method-based risk prediction tools for outcome prediction after AIS in new data sets.

Validation of the GERAADA score to predict 30-day mortality in acute type A aortic dissection in a single high-volume aortic centre

OBJECTIVES

This study aimed to evaluate employing the German Registry of Acute Aortic Dissection Type A (GERAADA) score to predict 30-day mortality in an aortic centre in the USA.

METHODS

Between January 2010 and June 2021, 689 consecutive patients underwent surgery for acute type A dissection at a single institution. Excluded were patients with missing clinical data (N = 4). The GERAADA risk score was retrospectively calculated via a web-based application. Model discrimination power was calculated with c-statistics from logistic regression and reported as the area under the receiver operating characteristic curve with 95% confidence intervals. The calibration was measured by calculating the observed versus estimated mortality ratio. The Brier score was used for the overall model evaluation.

RESULTS

Included were 685 patients [mean age 60.6 years (SD: 13.5), 64.8% male] who underwent surgery for acute type A aortic dissection. The 30-day mortality rate was 12.0%. The GERAADA score demonstrated very good discrimination power with an area under the receiver operating characteristic curve of 0.762 (95% confidence interval 0.703-0.821). The entire cohort's observed versus estimated mortality ratio was 0.543 (0.439-0.648), indicating an overestimation of the model-calculated risk. The Brier score was 0.010, thus revealing the model's acceptable overall performance.

CONCLUSIONS

The GERAADA score is a practical and easily accessible tool for reliably estimating the 30-day mortality risk of patients undergoing surgery for acute type A aortic dissection. This model may naturally overestimate risk in patients undergoing surgery in experienced aortic centres.

Development and Validation of the CANHEART Population-Based Laboratory Prediction Models for Atherosclerotic Cardiovascular Disease

BACKGROUND

Prediction of atherosclerotic cardiovascular disease (ASCVD) in primary prevention assessments exclusively with laboratory results may facilitate automated risk reporting and improve uptake of preventive therapies.

OBJECTIVE

To develop and validate sex-specific prediction models for ASCVD using age and routine laboratory tests and compare their performance with that of the pooled cohort equations (PCEs).

DESIGN

Derivation and validation of the CANHEART (Cardiovascular Health in Ambulatory Care Research Team) Lab Models.

SETTING

Population-based cohort study in Ontario, Canada.

PARTICIPANTS

A derivation and internal validation cohort of adults aged 40 to 75 years without cardiovascular disease from April 2009 to December 2015; an external validation cohort of primary care patients from January 2010 to December 2014.

MEASUREMENTS

Age and laboratory predictors measured in the outpatient setting included serum total cholesterol, high-density lipoprotein cholesterol, triglycerides, hemoglobin, mean corpuscular volume, platelets, leukocytes, estimated glomerular filtration rate, and glucose. The ASCVD outcomes were defined as myocardial infarction, stroke, and death from ischemic heart or cerebrovascular disease within 5 years.

RESULTS

Sex-specific models were developed and internally validated in 2 160 497 women and 1 833 147 men. They were well calibrated, with relative differences less than 1% between mean predicted and observed risk for both sexes. The c-statistic was 0.77 in women and 0.71 in men. External validation in 31 697 primary care patients showed a relative difference less than 14% and an absolute difference less than 0.3 percentage points in mean predicted and observed risks for both sexes. The c-statistics for the laboratory models were 0.72 for both sexes and were not statistically significantly different from those for the PCEs in women (change in c-statistic, -0.01 [95% CI, -0.03 to 0.01]) or men (change in c-statistic, -0.01 [CI, -0.04 to 0.02]).

LIMITATION

Medication use was not available at the population level.

CONCLUSION

The CANHEART Lab Models predict ASCVD with similar accuracy to more complex models, such as the PCEs.

PRIMARY FUNDING SOURCE

Canadian Institutes of Health Research.

Evaluating the cost of simplicity in score building: An example from alcohol research

Building a score from a questionnaire to predict a binary gold standard is a common research question in psychology and health sciences. When building this score, researchers may have to choose between statistical performance and simplicity. A practical question is to what extent it is worth sacrificing the former to improve the latter. We investigated this research question using real data, in which the aim was to predict an alcohol use disorder (AUD) diagnosis from 20 self-reported binary questions in young Swiss men (n = 233, mean age = 26). We compared the statistical performance using the area under the ROC curve (AUC) of (a) a "refined score" obtained by logistic regression and several simplified versions of it ("simple scores"): with (b) 3, (c) 2, and (d) 1 digit(s), and (e) a "sum score" that did not allow negative coefficients. We used four estimation methods: (a) maximum likelihood, (b) backward selection, (c) LASSO, and (d) ridge penalty. We also used bootstrap procedures to correct for optimism. Simple scores, especially sum scores, performed almost identically or even slightly better than the refined score (respective ranges of corrected AUCs for refined and sum scores: 0.828-0.848, 0.835-0.850), with the best performance been achieved by LASSO. Our example data demonstrated that simplifying a score to predict a binary outcome does not necessarily imply a major loss in statistical performance, while it may improve its implementation, interpretation, and acceptability. Our study thus provides further empirical evidence of the potential benefits of using sum scores in psychology and health sciences.

Developing and externally validating a machine learning risk prediction model for 30-day mortality after stroke using national stroke registers in the UK and Sweden

OBJECTIVES

We aimed to develop and externally validate a generalisable risk prediction model for 30-day stroke mortality suitable for supporting quality improvement analytics in stroke care using large nationwide stroke registers in the UK and Sweden.

DESIGN

Registry-based cohort study.

SETTING

Stroke registries including the Sentinel Stroke National Audit Programme (SSNAP) in England, Wales and Northern Ireland (2013-2019) and the national Swedish stroke register (Riksstroke 2015-2020).

PARTICIPANTS AND METHODS

Data from SSNAP were used for developing and temporally validating the model, and data from Riksstroke were used for external validation. Models were developed with the variables available in both registries using logistic regression (LR), LR with elastic net and interaction terms and eXtreme Gradient Boosting (XGBoost). Performances were evaluated with discrimination, calibration and decision curves.

OUTCOME MEASURES

The primary outcome was all-cause 30-day in-hospital mortality after stroke.

RESULTS

In total, 488 497 patients who had a stroke with 12.4% 30-day in-hospital mortality were used for developing and temporally validating the model in the UK. A total of 128 360 patients who had a stroke with 10.8% 30-day in-hospital mortality and 13.1% all mortality were used for external validation in Sweden. In the SSNAP temporal validation set, the final XGBoost model achieved the highest area under the receiver operating characteristic curve (AUC) (0.852 (95% CI 0.848 to 0.855)) and was well calibrated. The performances on the external validation in Riksstroke were as good and achieved AUC at 0.861 (95% CI 0.858 to 0.865) for in-hospital mortality. For Riksstroke, the models slightly overestimated the risk for in-hospital mortality, while they were better calibrated at the risk for all mortality.

CONCLUSION

The risk prediction model was accurate and externally validated using high quality registry data. This is potentially suitable to be deployed as part of quality improvement analytics in stroke care to enable the fair comparison of stroke mortality outcomes across hospitals and health systems across countries.

Evaluating dynamic and predictive discrimination for recurrent event models: use of a time-dependent C-index

Interest in analyzing recurrent event data has increased over the past few decades. One essential aspect of a risk prediction model for recurrent event data is to accurately distinguish individuals with different risks of developing a recurrent event. Although the concordance index (C-index) effectively evaluates the overall discriminative ability of a regression model for recurrent event data, a local measure is also desirable to capture dynamic performance of the regression model over time. Therefore, in this study, we propose a time-dependent C-index measure for inferring the model's discriminative ability locally. We formulated the C-index as a function of time using a flexible parametric model and constructed a concordance-based likelihood for estimation and inference. We adapted a perturbation-resampling procedure for variance estimation. Extensive simulations were conducted to investigate the proposed time-dependent C-index's finite-sample performance and estimation procedure. We applied the time-dependent C-index to three regression models of a study of re-hospitalization in patients with colorectal cancer to evaluate the models' discriminative capability.

Development of a Multivariable Seizure Likelihood Assessment Based on Clinical Information and Short Autonomic Activity Recordings for Children With Epilepsy

BACKGROUND

Predicting seizure likelihood for the following day would enable clinicians to extend or potentially schedule video-electroencephalography (EEG) monitoring when seizure risk is high. Combining standardized clinical data with short-term recordings of wearables to predict seizure likelihood could have high practical relevance as wearable data is easy and fast to collect. As a first step toward seizure forecasting, we classified patients based on whether they had seizures or not during the following recording.

METHODS

Pediatric patients admitted to the epilepsy monitoring unit wore a wearable that recorded the heart rate (HR), heart rate variability (HRV), electrodermal activity (EDA), and peripheral body temperature. We utilized short recordings from 9:00 to 9:15 pm and compared mean values between patients with and without impending seizures. In addition, we collected clinical data: age, sex, age at first seizure, generalized slowing, focal slowing, and spikes on EEG, magnetic resonance imaging findings, and antiseizure medication reduction. We used conventional machine learning techniques with cross-validation to classify patients with and without impending seizures.

RESULTS

We included 139 patients: 78 had no seizures and 61 had at least one seizure after 9 pm during the concurrent video-EEG and E4 recordings. HR (P < 0.01) and EDA (P < 0.01) were lower and HRV (P = 0.02) was higher for patients with than for patients without impending seizures. The average accuracy of group classification was 66%, and the mean area under the receiver operating characteristics was 0.72.

CONCLUSIONS

Short-term wearable recordings in combination with clinical data have great potential as an easy-to-use seizure likelihood assessment tool.

Predictive abilities comparison from multiple dynamic prediction models

With the development of personalized medicine, the study of individual prognosis appears to be a major contemporary scientific issue. Dynamic models are particularly well adapted to such studies by allowing some potential changes in the follow-up to be taken into account. In particular, this leads to more accurate predictions by updating the available information throughout the patient monitoring. Some mathematical tools have been developed to quantify and compare the effectiveness of dynamic predictions using dynamic versions of the area under the receiver operating characteristic curve and the Brier score in the competing risks setting. Nevertheless, only two predictive abilities can be compared. This may be too restrictive in a clinical context where more and more information can be collected during patient follow-up thanks to recent technological advances. Here we propose a new procedure that allows multiple comparisons of the predictive abilities of different biomarkers, based on the dynamic area under the receiver operating characteristic curve or Brier score. Performances of our testing procedure were assessed by simulations. Moreover, a motivating application in hepatology will be presented. Finally, this work compares more than two dynamic predictive abilities of biomarkers and is available via R functions on GitHub.

Machine learning models of healthcare expenditures predicting mortality: A cohort study of spousal bereaved Danish individuals

BACKGROUND

The ability to accurately predict survival in older adults is crucial as it guides clinical decision making. The added value of using health care usage for predicting mortality remains unexplored. The aim of this study was to investigate if temporal patterns of healthcare expenditures, can improve the predictive performance for mortality, in spousal bereaved older adults, next to other widely used sociodemographic variables.

METHODS

This is a population-based cohort study of 48,944 Danish citizens 65 years of age and older suffering bereavement within 2013-2016. Individuals were followed from date of spousal loss until death from all causes or 31st of December 2016, whichever came first. Healthcare expenditures were available on weekly basis for each person during the follow-up and used as predictors for mortality risk in Extreme Gradient Boosting models. The extent to which medical spending trajectories improved mortality predictions compared to models with sociodemographics, was assessed with respect to discrimination (AUC), overall prediction error (Brier score), calibration, and clinical benefit (decision curve analysis).

RESULTS

The AUC of age and sex for mortality the year after spousal loss was 70.8% [95% CI 68.8, 72.8]. The addition of sociodemographic variables led to an increase of AUC ranging from 0.9% to 3.1% but did not significantly reduce the overall prediction error. The AUC of the model combining the variables above plus medical spending usage was 80.8% [79.3, 82.4] also exhibiting smaller Brier score and better calibration. Overall, patterns of healthcare expenditures improved mortality predictions the most, also exhibiting the highest clinical benefit among the rest of the models.

CONCLUSION

Temporal patterns of medical spending have the potential to significantly improve our assessment on who is at high risk of dying after suffering spousal loss. The proposed methodology can assist in a more efficient risk profiling and prognosis of bereaved individuals.

Comparing the performance of machine learning and conventional models for predicting atherosclerotic cardiovascular disease in a general Chinese population

BACKGROUND

Accurately predicting the risk of atherosclerotic cardiovascular disease (ASCVD) is crucial for implementing individualized prevention strategies and improving patient outcomes. Our objective is to develop machine learning (ML)-based models for predicting ASCVD risk in a prospective Chinese population and compare their performance with conventional regression models.

METHODS

A hybrid dataset consisting of 551 features was used, including 98 demographic, behavioral, and psychological features, 444 Electrocardiograph (ECG) features, and 9 Echocardiography (Echo) features. Seven machine learning (ML)-based models were trained, validated, and tested after selecting the 30 most informative features. We compared the discrimination, calibration, net benefit, and net reclassification improvement (NRI) of the ML models with those of conventional ASCVD risk calculators, such as the Pooled Cohort Equations (PCE) and Prediction for ASCVD Risk in China (China-PAR).

RESULTS

The study included 9,609 participants (mean age 53.4 ± 10.4 years, 53.7% female), and during a median follow-up of 4.7 years, 431 (4.5%) participants developed ASCVD. In the testing set, the final ML-based ANN model outperformed PCE, China-PAR, recalibrated PCE, and recalibrated China-PAR in predicting ASCVD. This was demonstrated by the model's higher area under the curve (AUC) of 0.800, compared to 0.777, 0.780, 0.779, and 0.779 for the other models, respectively. Additionally, the model had a lower Hosmer-Lemeshow χ2 of 9.1, compared to 37.3, 67.6, 126.6, and 18.6 for the other models. The net benefit at a threshold of 5% was also higher for the ML-based ANN model at 0.017, compared to 0.016, 0.013, 0.017, and 0.016 for the other models, respectively. Furthermore, the NRI was 0.089 for the ML-based ANN model, while it was 0.355, 0.098, and 0.088 for PCE, China-PAR, and recalibrated PCE, respectively.

CONCLUSIONS

Compared to conventional regression ASCVD risk calculators, such as PCE and China-PAR, the ANN prediction model may help optimize identification of individuals at heightened cardiovascular risk by flexibly incorporating a wider range of potential predictors. The findings may help guide clinical decision-making and ultimately contribute to ASCVD prevention and management.

A biomarker discovery framework for childhood anxiety

Introduction

Anxiety is the most common manifestation of psychopathology in youth, negatively affecting academic, social, and adaptive functioning and increasing risk for mental health problems into adulthood. Anxiety disorders are diagnosed only after clinical symptoms emerge, potentially missing opportunities to intervene during critical early prodromal periods. In this study, we used a new empirical approach to extracting nonlinear features of the electroencephalogram (EEG), with the goal of discovering differences in brain electrodynamics that distinguish children with anxiety disorders from healthy children. Additionally, we examined whether this approach could distinguish children with externalizing disorders from healthy children and children with anxiety.

Methods

We used a novel supervised tensor factorization method to extract latent factors from repeated multifrequency nonlinear EEG measures in a longitudinal sample of children assessed in infancy and at ages 3, 5, and 7 years of age. We first examined the validity of this method by showing that calendar age is highly correlated with latent EEG complexity factors (r = 0.77). We then computed latent factors separately for distinguishing children with anxiety disorders from healthy controls using a 5-fold cross validation scheme and similarly for distinguishing children with externalizing disorders from healthy controls.

Results

We found that latent factors derived from EEG recordings at age 7 years were required to distinguish children with an anxiety disorder from healthy controls; recordings from infancy, 3 years, or 5 years alone were insufficient. However, recordings from two (5, 7 years) or three (3, 5, 7 years) recordings gave much better results than 7 year recordings alone. Externalizing disorders could be detected using 3- and 5 years EEG data, also giving better results with two or three recordings than any single snapshot. Further, sex assigned at birth was an important covariate that improved accuracy for both disorder groups, and birthweight as a covariate modestly improved accuracy for externalizing disorders. Recordings from infant EEG did not contribute to the classification accuracy for either anxiety or externalizing disorders.

Conclusion

This study suggests that latent factors extracted from EEG recordings in childhood are promising candidate biomarkers for anxiety and for externalizing disorders if chosen at appropriate ages.

Supervised two-dimensional functional principal component analysis with time-to-event outcomes and mammogram imaging data

Screening mammography aims to identify breast cancer early and secondarily measures breast density to classify women at higher or lower than average risk for future breast cancer in the general population. Despite the strong association of individual mammography features to breast cancer risk, the statistical literature on mammogram imaging data is limited. While functional principal component analysis (FPCA) has been studied in the literature for extracting image-based features, it is conducted independently of the time-to-event response variable. With the consideration of building a prognostic model for precision prevention, we present a set of flexible methods, supervised FPCA (sFPCA) and functional partial least squares (FPLS), to extract image-based features associated with the failure time while accommodating the added complication from right censoring. Throughout the article, we hope to demonstrate that one method is favored over the other under different clinical setups. The proposed methods are applied to the motivating data set from the Joanne Knight Breast Health cohort at Siteman Cancer Center. Our approaches not only obtain the best prediction performance compared to the benchmark model, but also reveal different risk patterns within the mammograms.

Depiction of neuroendocrine features associated with immunotherapy response using a novel one-class predictor in lung adenocarcinoma

BACKGROUND

Tumours with no evidence of neuroendocrine transformation histologically but harbouring neuroendocrine features are collectively referred to as non-small cell lung cancer (NSCLC) with neuroendocrine differentiation (NED). Investigating the mechanisms underlying NED is conducive to designing appropriate treatment options for NSCLC patients.

METHODS

In the present study, we integrated multiple lung cancer datasets to identify neuroendocrine features using a one-class logistic regression (OCLR) machine learning algorithm trained on small cell lung cancer (SCLC) cells, a pulmonary neuroendocrine cell type, based on the transcriptome of NSCLC and named the NED index (NEDI). Single-sample gene set enrichment analysis, pathway enrichment analysis, ESTIMATE algorithm analysis, and unsupervised subclass mapping (SubMap) were performed to assess the altered pathways and immune characteristics of lung cancer samples with different NEDI values.

RESULTS

We developed and validated a novel one-class predictor based on the expression values of 13,279 mRNAs to quantitatively evaluate neuroendocrine features in NSCLC. We observed that a higher NEDI correlated with better prognosis in patients with LUAD. In addition, we observed that a higher NEDI was significantly associated with reduced immune cell infiltration and immune effector molecule expression. Furthermore, we found that etoposide-based chemotherapy might be more effective in the treatment of LUAD with high NEDI values. Moreover, we noted that tumours with low NEDI values had better responses to immunotherapy than those with high NEDI values.

CONCLUSIONS

Our findings improve the understanding of NED and provide a useful strategy for applying NEDI-based risk stratification to guide decision-making in the treatment of LUAD.

Statistical analysis of high-dimensional biomedical data: a gentle introduction to analytical goals, common approaches and challenges

BACKGROUND

In high-dimensional data (HDD) settings, the number of variables associated with each observation is very large. Prominent examples of HDD in biomedical research include omics data with a large number of variables such as many measurements across the genome, proteome, or metabolome, as well as electronic health records data that have large numbers of variables recorded for each patient. The statistical analysis of such data requires knowledge and experience, sometimes of complex methods adapted to the respective research questions.

METHODS

Advances in statistical methodology and machine learning methods offer new opportunities for innovative analyses of HDD, but at the same time require a deeper understanding of some fundamental statistical concepts. Topic group TG9 "High-dimensional data" of the STRATOS (STRengthening Analytical Thinking for Observational Studies) initiative provides guidance for the analysis of observational studies, addressing particular statistical challenges and opportunities for the analysis of studies involving HDD. In this overview, we discuss key aspects of HDD analysis to provide a gentle introduction for non-statisticians and for classically trained statisticians with little experience specific to HDD.

RESULTS

The paper is organized with respect to subtopics that are most relevant for the analysis of HDD, in particular initial data analysis, exploratory data analysis, multiple testing, and prediction. For each subtopic, main analytical goals in HDD settings are outlined. For each of these goals, basic explanations for some commonly used analysis methods are provided. Situations are identified where traditional statistical methods cannot, or should not, be used in the HDD setting, or where adequate analytic tools are still lacking. Many key references are provided.

CONCLUSIONS

This review aims to provide a solid statistical foundation for researchers, including statisticians and non-statisticians, who are new to research with HDD or simply want to better evaluate and understand the results of HDD analyses.

Predicting the onset of breast cancer using mammogram imaging data with irregular boundary

With mammography being the primary breast cancer screening strategy, it is essential to make full use of the mammogram imaging data to better identify women who are at higher and lower than average risk. Our primary goal in this study is to extract mammogram-based features that augment the well-established breast cancer risk factors to improve prediction accuracy. In this article, we propose a supervised functional principal component analysis (sFPCA) over triangulations method for extracting features that are ordered by the magnitude of association with the failure time outcome. The proposed method accommodates the irregular boundary issue posed by the breast area within the mammogram imaging data with flexible bivariate splines over triangulations. We also provide an eigenvalue decomposition algorithm that is computationally efficient. Compared to the conventional unsupervised FPCA method, the proposed method results in a lower Brier Score and higher area under the ROC curve (AUC) in simulation studies. We apply our method to data from the Joanne Knight Breast Health Cohort at Siteman Cancer Center. Our approach not only obtains the best prediction performance comparing to unsupervised FPCA and benchmark models but also reveals important risk patterns within the mammogram images. This demonstrates the importance of utilizing additional supervised image-based features to clarify breast cancer risk.

CA-125 Early Dynamics to Predict Overall Survival in Women with Newly Diagnosed Advanced Ovarian Cancer Based on Meta-Analysis Data

(1) Background: Cancer antigen 125 (CA-125) is a protein produced by ovarian cancer cells that is used for patients' monitoring. However, the best ways to analyze its decline and prognostic role are poorly quantified. (2) Methods: We leveraged individual patient data from the Gynecologic Cancer Intergroup (GCIG) meta-analysis (N = 5573) to compare different approaches summarizing the early trajectory of CA-125 before the prediction time (called the landmark time) at 3 or 6 months after treatment initiation in order to predict overall survival. These summaries included observed and estimated measures obtained by a linear mixed model (LMM). Their performances were evaluated by 10-fold cross-validation with the Brier score and the area under the ROC (AUC). (3) Results: The estimated value and the last observed value at 3 months were the best measures used to predict overall survival, with an AUC of 0.75 CI 95% [0.70; 0.80] at 24 and 36 months and 0.74 [0.69; 0.80] and 0.75 [0.69; 0.80] at 48 months, respectively, considering that CA-125 over 6 months did not improve the AUC, with 0.74 [0.68; 0.78] at 24 months and 0.71 [0.65; 0.76] at 36 and 48 months. (4) Conclusions: A 3-month surveillance provided reliable individual information on overall survival until 48 months for patients receiving first-line chemotherapy.

Age and BMI equal Modified Frailty Index, Modified Charlson Comorbidity Index and ASA in predicting adverse events in spinal surgery for cervical degenerative diseases

OBJECTIVE

To compare Modified Frailty Index (mFI), Modified Charlson Comorbidity (mCCI) and ASA with demographic data such as age, BMI and gender in the prediction of AEs obtained using a validated systematic reporting system in a prospective cohort undergoing cervical spine surgery.

METHODS

All adult patients undergoing spine surgery for cervical degenerative disease at our academic tertiary referral center from February 1, 2016, to January 31, 2017, were included. Morbidity and mortality were determined according to the predefined adverse event (AE) variables using the Spinal Adverse Events Severity (SAVES) System. Area under the curve (AUC) analyses from receiver operating characteristics (ROC) curves were used to assess the discriminative ability in predicting AEs for the comorbidity indices mFI, mCCI, ASA and for BMI, age and gender.

RESULTS

A total of 288 consecutive cervical cases were included. BMI was the most predictive demographic factor for an AE (AUC = 0.58), the most predictive comorbidity index was mCCI (AUC = 0.52). No combination of comorbidity indices or demographic factors reached a threshold of AUC ≥ 0.7 for AEs. As predictor of extended length of stay: age (AUC = 0.77), mFI (AUC = 0.70) and ASA (AUC = 0.70) were similar and fair.

CONCLUSION

Age and BMI equal mFI, mCCI and ASA in predicting postoperative AEs, amongst patients operated for cervical degenerative disease. No significant difference was found between mFI, mCCI and ASA in the discriminative abilities in predicting morbidity, based on prospectively collected AEs according to the SAVES grading system.

Predicting Alcohol Consumption Patterns for Individuals with a User-Friendly Parsimonious Statistical Model

Many studies on the relationship between alcohol and health outcome focus primarily on average consumption over time and do not consider how heavy per-occasion drinking may influence apparent relationships. Improved methods concerning the most recent drinking occasion are essential to inform the extent of alcohol-related health problems. We aimed to develop a user-friendly and readily replicable computational model that predicts: (i) an individual's probability of consuming alcohol ≥2, 3, 4… drinks; and (ii) the total number of days during which consumption is ≥2, 3, 4… drinks over a specified period. Data from the 2010 and 2011 National Survey on Drug Use and Health (NSDUH) were used to develop and validate the model. Predictors used in model development were age, gender, usual number of drinks consumed per day, and number of drinking days in the past 30 days. Main outcomes were number of drinks consumed on the last drinking occasion in the past 30 days, and number of days of risky levels of consumption. The area under ROC curves ranged between 0.86 and 0.91 when predicting the number of drinks consumed. Coefficients were very close to 1 for all outcomes, indicating closeness between the predicted and observed values. This straightforward modelling approach can be easily adopted by public health behavioral studies.

Development of a prognostic nomogram for sepsis associated-acute respiratory failure patients on 30-day mortality in intensive care units: a retrospective cohort study

BACKGROUND

Acute respiratory failure is a type of sepsis complicated by severe organ failure. We have developed a new nomogram for predicting the 30-day risk of death in patients through a retrospective study.

METHOD

Data was collected and extracted from MIMICIV, with 768 eligible cases randomly assigned to the primary cohort (540) and the validation cohort (228). The final six factors were included by Cox regression analysis to create the Nomogram, the accuracy of the Nomogram was assessed using the C-index and calibration curve, and finally, the clinical usefulness of the Nomogram was evaluated using DCA in.

RESULTS

Multivariate Cox regression analysis showed that age, DBP, lactate, PaO2, platelet, mechanical ventilation were independent factors for 30-day mortality of SA-ARF. The nomogram established based on the six factors. The C-index of nomogram in the primary cohort is 0.731 (95% CI 0.657-0.724) and 0.722 (95%CI 0.622-0.759) in the validation cohort. Besides, the decision curve analysis (DCA) confirmed the clinical usefulness of the nomogram.

CONCLUSION

The study developed and validated a risk prediction model for SA-ARF patients that can help clinicians reasonably determine disease risk and further confirm its clinical utility using internal validation.

A Nomogram for Early Diagnosis of Community-Acquired Pneumonia Based on Bronchoalveolar Lavage Fluid Metabolomics

Purpose

There is a high disease burden associated with community-acquired pneumonia (CAP) around the world. A timely and correct diagnosis of CAP can facilitate early treatment and prevent illness progression. The present study aimed to find some novel biomarkers of CAP by metabolic analysis and construct a nomogram model for precise diagnosis and individualized treatment of CAP patients.

Patients and Methods

42 CAP patients and 20 controls were enrolled in this study. The metabolic profiles of bronchoalveolar lavage fluid (BALF) samples were identified by untargeted LC-MS/MS analysis. With a VIP score ≥ 1 in OPLS-DA analysis and P < 0.05, the significantly dysregulated metabolites were estimated as potential biomarkers of CAP, which were further included in the construction of the diagnostic prediction model along with laboratory inflammatory indexes via stepwise backward regression analysis. Discrimination, calibration, and clinical applicability of the nomogram were evaluated by the C-index, the calibration curve, and the decision curve analysis (DCA) estimated by bootstrap resampling.

Results

The metabolic profiles differed obviously between CAP patients and healthy controls, as shown by PCA and OPLS-DA plots. Seven metabolites significantly dysregulated in CAP were established: dimethyl disulfide, oleic acid (d5), N-acetyl-a-neuraminic acid, pyrimidine, choline, LPC (12:0/0:0) and PA (20:4/2:0). Multivariate logistic regression revealed that the expression levels of PA (20:4/2:0), N-acetyl-a-neuraminic acid, and CRP were associated with CAP. After being validated by bootstrap resampling, this model showed satisfactory diagnostic performance.

Conclusion

A novel nomogram prediction model containing metabolic potential biomarkers in BALF that was developed for the early diagnosis of CAP offers insights into the pathogenesis and host response in CAP.

Establishment and validation of a torsade de pointes prediction model based on human iPSC‑derived cardiomyocytes

Drug-induced cardiotoxicity is one of the main causes of drug failure, which leads to subsequent withdrawal from pharmaceutical development. Therefore, identifying the potential toxic candidate in the early stages of drug development is important. Human induced pluripotent stem cell-derived cardiomyocytes (hiPSC-CMs) are a useful tool for assessing candidate compounds for arrhythmias. However, a suitable model using hiPSC-CMs to predict the risk of torsade de pointes (TdP) has not been fully established. The present study aimed to establish a predictive TdP model based on hiPSC-CMs. In the current study, 28 compounds recommended by the Comprehensive in vitro Proarrhythmia Assay (CiPA) were used as training set and models were established in different risk groups, high- and intermediate-risk versus low-risk groups. Subsequently, six endpoints of electrophysiological responses were used as potential model predictors. Accuracy, sensitivity and area under the curve (AUC) were used as evaluation indices of the models and seven compounds with known TdP risk were used to verify model differentiation and calibration. The results showed that among the seven models, the AUC of logistic regression and AdaBoost model was higher and had little difference in both training and test sets, which indicated that the discriminative ability and model stability was good and excellent, respectively. Therefore, these two models were taken as submodels, similar weight was configured and a new TdP risk prediction model was constructed using a soft voting strategy. The classification accuracy, sensitivity and AUC of the new model were 0.93, 0.95 and 0.92 on the training set, respectively and all 1.00 on the test set, which indicated good discrimination ability on both training and test sets. The risk threshold was defined as 0.50 and the consistency between the predicted and observed results were 92.8 and 100% on the training and test sets, respectively. Overall, the present study established a risk prediction model for TdP based on hiPSC-CMs which could be an effective predictive tool for compound-induced arrhythmias.

Performance of multilabel machine learning models and risk stratification schemas for predicting stroke and bleeding risk in patients with non-valvular atrial fibrillation

BACKGROUND

Appropriate anticoagulant therapy for patients with atrial fibrillation (AF) requires assessment of stroke and bleeding risks. However, risk stratification schemas such as CHA2DS2-VASc and HAS-BLED have modest predictive capacity for patients with AF. Multilabel machine learning (ML) techniques may improve predictive performance and support decision-making for anticoagulant therapy. We compared the performance of multilabel ML models with the currently used risk scores for predicting outcomes in AF patients.

METHODS

This was a retrospective cohort study of 9670 patients, mean age 76.9 years, 46% women, who were hospitalized with non-valvular AF, and had 1-year follow-up. The outcomes were ischemic stroke (167), major bleeding (430) admissions, all-cause death (1912) and event-free survival (7387). Discrimination and calibration of ML models were compared with clinical risk scores by area under the curve (AUC). Risk stratification was assessed using net reclassification index (NRI).

RESULTS

Multilabel gradient boosting classifier chain provided the best AUCs for stroke (0.685 95% CI 0.676, 0.694), major bleeding (0.709 95% CI 0.703, 0.716) and death (0.765 95% CI 0.763, 0.768) compared to multi-layer neural networks and classifier chain using support vector machine. It provided modest performance improvement for stroke compared to AUC of CHA2DS2-VASc (0.652, NRI = 3.2%, p-value = 0.1), but significantly improved major bleeding prediction compared to AUC of HAS-BLED (0.522, NRI = 22.8%, p-value < 0.05). It also achieved greater discriminant power for death compared with AUC of CHA2DS2-VASc (0.606, p-value < 0.05). ML models identified additional risk features such as hemoglobin level, renal function.

CONCLUSIONS

Multilabel ML models can outperform clinical risk stratification scores for predicting the risk of major bleeding and death in non-valvular AF patients.

Efficient Evaluation of Prediction Rules in Semi-Supervised Settings under Stratified Sampling

In many contemporary applications, large amounts of unlabeled data are readily available while labeled examples are limited. There has been substantial interest in semi-supervised learning (SSL) which aims to leverage unlabeled data to improve estimation or prediction. However, current SSL literature focuses primarily on settings where labeled data is selected uniformly at random from the population of interest. Stratified sampling, while posing additional analytical challenges, is highly applicable to many real world problems. Moreover, no SSL methods currently exist for estimating the prediction performance of a fitted model when the labeled data is not selected uniformly at random. In this paper, we propose a two-step SSL procedure for evaluating a prediction rule derived from a working binary regression model based on the Brier score and overall misclassification rate under stratified sampling. In step I, we impute the missing labels via weighted regression with nonlinear basis functions to account for stratified sampling and to improve efficiency. In step II, we augment the initial imputations to ensure the consistency of the resulting estimators regardless of the specification of the prediction model or the imputation model. The final estimator is then obtained with the augmented imputations. We provide asymptotic theory and numerical studies illustrating that our proposals outperform their supervised counterparts in terms of efficiency gain. Our methods are motivated by electronic health record (EHR) research and validated with a real data analysis of an EHR-based study of diabetic neuropathy.

PROM1, CXCL8, RUNX1, NAV1 and TP73 genes as independent markers predictive of prognosis or response to treatment in two cohorts of high-grade serous ovarian cancer patients

Considering the vast biological diversity and high mortality rate in high-grade ovarian cancers, identification of novel biomarkers, enabling precise diagnosis and effective, less aggravating treatment, is of paramount importance. Based on scientific literature data, we selected 80 cancer-related genes and evaluated their mRNA expression in 70 high-grade serous ovarian cancer (HGSOC) samples by Real-Time qPCR. The results were validated in an independent Northern American cohort of 85 HGSOC patients with publicly available NGS RNA-seq data. Detailed statistical analyses of our cohort with multivariate Cox and logistic regression models considering clinico-pathological data and different TP53 mutation statuses, revealed an altered expression of 49 genes to affect the prognosis and/or treatment response. Next, these genes were investigated in the validation cohort, to confirm the clinical significance of their expression alterations, and to identify genetic variants with an expected high or moderate impact on their products. The expression changes of five genes, PROM1, CXCL8, RUNX1, NAV1, TP73, were found to predict prognosis or response to treatment in both cohorts, depending on the TP53 mutation status. In addition, we revealed novel and confirmed known SNPs in these genes, and showed that SNPs in the PROM1 gene correlated with its elevated expression.

Comparing Machine Learning to Regression Methods for Mortality Prediction Using Veterans Affairs Electronic Health Record Clinical Data

BACKGROUND

It is unclear whether machine learning methods yield more accurate electronic health record (EHR) prediction models compared with traditional regression methods.

OBJECTIVE

The objective of this study was to compare machine learning and traditional regression models for 10-year mortality prediction using EHR data.

DESIGN

This was a cohort study.

SETTING

Veterans Affairs (VA) EHR data.

PARTICIPANTS

Veterans age above 50 with a primary care visit in 2005, divided into separate training and testing cohorts (n= 124,360 each).

MEASUREMENTS AND ANALYTIC METHODS

The primary outcome was 10-year all-cause mortality. We considered 924 potential predictors across a wide range of EHR data elements including demographics (3), vital signs (9), medication classes (399), disease diagnoses (293), laboratory results (71), and health care utilization (149). We compared discrimination (c-statistics), calibration metrics, and diagnostic test characteristics (sensitivity, specificity, and positive and negative predictive values) of machine learning and regression models.

RESULTS

Our cohort mean age (SD) was 68.2 (10.5), 93.9% were male; 39.4% died within 10 years. Models yielded testing cohort c-statistics between 0.827 and 0.837. Utilizing all 924 predictors, the Gradient Boosting model yielded the highest c-statistic [0.837, 95% confidence interval (CI): 0.835-0.839]. The full (unselected) logistic regression model had the highest c-statistic of regression models (0.833, 95% CI: 0.830-0.835) but showed evidence of overfitting. The discrimination of the stepwise selection logistic model (101 predictors) was similar (0.832, 95% CI: 0.830-0.834) with minimal overfitting. All models were well-calibrated and had similar diagnostic test characteristics.

LIMITATION

Our results should be confirmed in non-VA EHRs.

CONCLUSION

The differences in c-statistic between the best machine learning model (924-predictor Gradient Boosting) and 101-predictor stepwise logistic models for 10-year mortality prediction were modest, suggesting stepwise regression methods continue to be a reasonable method for VA EHR mortality prediction model development.

Risk prediction of 30-day mortality after stroke using machine learning: a nationwide registry-based cohort study

Backgrounds

We aimed to develop and validate machine learning (ML) models for 30-day stroke mortality for mortality risk stratification and as benchmarking models for quality improvement in stroke care.

Methods

Data from the UK Sentinel Stroke National Audit Program between 2013 to 2019 were used. Models were developed using XGBoost, Logistic Regression (LR), LR with elastic net with/without interaction terms using 80% randomly selected admissions from 2013 to 2018, validated on the 20% remaining admissions, and temporally validated on 2019 admissions. The models were developed with 30 variables. A reference model was developed using LR and 4 variables. Performances of all models was evaluated in terms of discrimination, calibration, reclassification, Brier scores and Decision-curves.

Results

In total, 488,497 stroke patients with a 12.3% 30-day mortality rate were included in the analysis. In 2019 temporal validation set, XGBoost model obtained the lowest Brier score (0.069 (95% CI: 0.068–0.071)) and the highest area under the ROC curve (AUC) (0.895 (95% CI: 0.891–0.900)) which outperformed LR reference model by 0.04 AUC (p < 0.001) and LR with elastic net and interaction term model by 0.003 AUC (p < 0.001). All models were perfectly calibrated for low (< 5%) and moderate risk groups (5–15%) and ≈1% underestimation for high-risk groups (> 15%). The XGBoost model reclassified 1648 (8.1%) low-risk cases by the LR reference model as being moderate or high-risk and gained the most net benefit in decision curve analysis.

Conclusions

All models with 30 variables are potentially useful as benchmarking models in stroke-care quality improvement with ML slightly outperforming others.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12883-022-02722-1.

Validated Risk-Score Model Predicting Lymph Node Metastases in Patients with Non-Functional Gastroenteropancreatic Neuroendocrine Tumors

BACKGROUND

The incidence of, and factors associated with, lymph node metastasis (LN+) in non-functional gastroenteropancreatic (GEP) neuroendocrine tumors (NETs) are not well characterized.

METHODS

Patients were identified from the 2010-2015 National Cancer Database who underwent surgical resection with lymphadenectomy for clinical stage I-III non-functional GEP NETs. Among a randomly selected training subset of 75% of the study population, variables associated with LN+ were identified using multivariable logistic regression analysis, and these variables were used to create a risk-score model for LN+, which was internally validated among the remaining 25% of the cohort.

RESULTS

Of 12,228 patients evaluated, 6,902 (56.4%) had LN+. Among the training set, variables associated with LN+ included age (70 years of age or older: odds ratio [OR] 1.12, 95% CI 1.00-1.24; ref: less than 70 years), tumor location (stomach: OR 3.72, 95% CI 2.94-4.71; small intestine: OR 19.60, 95% CI 17.31-22.19; ref: pancreas), tumor grade (moderately differentiated: OR 1.47, 95% CI 1.30-1.67; poorly differentiated/anaplastic: OR 1.53, 95% CI 1.21-1.95; ref: well-differentiated), tumor size (2-4 cm: OR 2.40, 95% CI 2.13-2.70; >4 cm: OR 5.25, 95% CI 4.47-6.17; ref: <2 cm), and lymphovascular invasion (OR 5.62, 95% CI 5.08-6.21; ref: no lymphovascular invasion). After internal validation, a risk-score model for LN+ using these variables was developed composed of low- (N = 2,779), intermediate- (N = 2,598), high- (N = 3,433), and very-high-risk (N = 3,418) groups; within each group the rate of LN+ was 8.7%, 48.6%, 64.9%, and 92.8%, respectively.

CONCLUSIONS

This developed risk-score model, including both patient and tumor variables, can be used to calculate the risk for LN metastases in patients with GEP NETs.

The Development and Validation of a Nomogram Incorporating Clinical, Pathological, and Therapeutic Features to Predict Overall Survival in Patients With Penile Cancer: A SEER-Based Study

Objective

This study aimed to investigate the prognostic factors of penile cancer and establish a comprehensive predictive model for clinical application.

Methods

A total of 581 patients from the Surveillance, Epidemiology, and End Results (SEER) program (2000-2018) were used to develop the prognostic model. The multivariate Cox proportional hazards regression was performed to identify independent prognostic factors to develop the nomogram. The performance of this model was validated internally by a cohort with 143 patients from the SEER database and validated externally by a cohort with 70 patients from the West China Hospital, Sichuan University (2010-2020).

Results

Age, marital status, size of the primary lesion, primary tumor (T), regional lymph nodes status, distant metastasis (M), and the surgery of regional lymph node (LND) were the independent prognostic factors for overall survival (OS) and were incorporated in the prognostic model. The prognostic nomogram showed a good risk stratification ability for OS in the development cohort, internal validation cohort, and external validation cohort.

Conclusion

This study incorporates the clinical, pathological, and therapeutic features comprehensively to develop a novel and clinically effective prognostic model for patients with penile cancer.

Estimation and inference of predictive discrimination for survival outcome risk prediction models

Accurate risk prediction has been the central goal in many studies of survival outcomes. In the presence of multiple risk factors, a censored regression model can be employed to estimate a risk prediction rule. Before the prediction tool can be popularized for practical use, it is crucial to rigorously assess its prediction performance. In our motivating example, researchers are interested in developing and validating a risk prediction tool to identify future lung cancer cases by integrating demographic information, disease characteristics and smoking-related data. Considering the long latency period of cancer, it is desirable for a prediction tool to achieve discriminative performance that does not weaken over time. We propose estimation and inferential procedures to comprehensively assess both the overall predictive discrimination and the temporal pattern of an estimated prediction rule. The proposed methods readily accommodate commonly used censored regression models, including the Cox proportional hazards model and the accelerated failure time model. The estimators are consistent and asymptotically normal, and reliable variance estimators are also developed. The proposed methods offer an informative tool for inferring time-dependent predictive discrimination, as well as for comparing the discrimination performance between candidate models. Applications of the proposed methods demonstrate enduring performance of the risk prediction tool in the PLCO study and detected decaying performance in a study of liver disease.

Impact of Pre-Treatment NLR and Other Hematologic Biomarkers on the Outcomes of Early-Stage Non-Small-Cell Lung Cancer Treated with Stereotactic Body Radiation Therapy

Introduction: We evaluated the association of pre-treatment immunologic biomarkers on the outcomes of early-stage non-small-cell lung cancer (NSCLC) patients treated with stereotactic body radiation therapy (SBRT). Materials and methods: In this retrospective study, all newly diagnosed early-stage NSCLC treated with SBRT between January 2010 and December 2017 were screened and included for further analysis. The pre-treatment neutrophil-lymphocyte ratio (NLR), monocyte lymphocyte ratio (MLR), and platelet-lymphocyte ratio (PLR) were calculated. Overall survival (OS) and recurrence-free survival (RFS) were estimated by Kaplan–Meier. Multivariable models were constructed to determine the impact of different biomarkers and the Akaike information criterion (AIC), index of adequacy, and scaled Brier scores were calculated. Results: A total of 72 patients were identified and 61 were included in final analysis. The median neutrophil count at baseline was 5.4 × 10⁹/L (IQR: 4.17–7.05 × 10⁹/L). Median lymphocyte count was 1.63 × 10⁹/L (IQR: 1.29–2.10 × 10⁹/L), median monocyte count was 0.65 × 10⁹/L (IQR: 0.54–0.83 × 10⁹/L), median platelet count was 260.0 × 10⁹/L (IQR: 211.0–302.0 × 10⁹/L). The median NLR was 3.42 (IQR: 2.38–5.04), median MLR was 0.39 (IQR: 0.31–0.53), and median PLR was 156.4 (IQR: 117.2–197.5). On multivariable regression a higher NLR was associated with worse OS (p = 0.01; HR-1.26; 95% CI 1.04–1.53). The delta AIC between the two multivariable models was 3.4, suggesting a moderate impact of NLR on OS. On multivariable analysis, higher NLR was associated with poor RFS (p = 0.001; NLR^1 HR 0.36; 0.17–0.78; NLR^2 HR-1.16; 95% CI 1.06–1.26) with a nonlinear relationship. The delta AIC between the two multivariable models was 16.2, suggesting a strong impact of NLR on RFS. In our cohort, MLR and PLR were not associated with RFS or OS in multivariable models. Conclusions: Our study suggests NLR, as a biomarker of systemic inflammation, is an independent prognostic factor for OS and RFS. The nonlinear relationship with RFS may indicate a suitable immunological environment is needed for optimal SBRT action and tumoricidal mechanisms. These findings require further validation in independent cohorts.

Urinary L-FABP is a promising prognostic biomarker of ACLF and mortality in patients with decompensated cirrhosis

BACKGROUND & AIMS

Decompensated cirrhosis (DC) is associated with high mortality, mainly owing to the development of acute-on-chronic liver failure (ACLF). Identifying the patients with DC who are at high risk of mortality and ACLF development is an unmet clinical need. Liver fatty acid-binding protein (L-FABP) is expressed in several organs and correlates with liver and systemic inflammation. Herein, we aimed to assess the prognostic value of L-FABP in patients with DC.

METHODS

A prospective series of 444 patients hospitalized for DC was divided into 2 cohorts: study cohort (305 patients) and validation cohort (139 patients). L-FABP was measured in urine and plasma samples collected at admission. Neutrophil gelatinase-associated lipocalin (NGAL) was also measured in urine samples for comparison.

RESULTS

Urine but not plasma L-FABP correlated with 3-month survival on univariate analysis. On multivariate analysis, urine L-FABP and model for end-stage liver disease (MELD)-Na were the only independent predictors of prognosis. Urine L-FABP levels were higher in patients with ACLF than in those without and also predicted the development of ACLF, together with MELD-Na, during follow-up. In patients with ACLF, urine L-FABP correlated with liver, coagulation, and circulatory failure. Urine L-FABP levels were also increased in patients with acute kidney injury, particularly in those with acute tubular necrosis. The ability of urinary L-FABP to predict survival and ACLF development was confirmed in the validation cohort. Urine NGAL predicted outcome on univariate but not multivariate analysis.

CONCLUSIONS

Urinary L-FABP levels are independently associated with the 3-month clinical course in patients with DC, in terms of mortality and ACLF development. Urinary L-FABP is a promising prognostic biomarker for patients with DC.

LAY SUMMARY

Increased levels of liver fatty acid-binding protein (L-FABP), a protein related to lipid metabolism, have been associated with liver-related diseases. The present study analyzed urinary L-FABP levels in 2 independent groups of patients with decompensated cirrhosis and showed that higher urinary L-FABP levels correlated with increased mortality and risk of acute-on-chronic liver failure development. Therefore, urinary L-FABP levels could be useful as a new tool to predict complications in patients with decompensated cirrhosis.

Polygenic risk scores: the future of cancer risk prediction, screening, and precision prevention

Genome-wide association studies (GWASs) have shown that the genetic architecture of cancers are highly polygenic and enabled researchers to identify genetic risk loci for cancers. The genetic variants associated with a cancer can be combined into a polygenic risk score (PRS), which captures part of an individual's genetic susceptibility to cancer. Recently, PRSs have been widely used in cancer risk prediction and are shown to be capable of identifying groups of individuals who could benefit from the knowledge of their probabilistic susceptibility to cancer, which leads to an increased interest in understanding the potential utility of PRSs that might further refine the assessment and management of cancer risk. In this context, we provide an overview of the major discoveries from cancer GWASs. We then review the methodologies used for PRS construction, and describe steps for the development and evaluation of risk prediction models that include PRS and/or conventional risk factors. Potential utility of PRSs in cancer risk prediction, screening, and precision prevention are illustrated. Challenges and practical considerations relevant to the implementation of PRSs in health care settings are discussed.

Weekly pattern of dialysis unit blood pressure is a promising marker for prognosis evaluation in hemodialysis population

BACKGROUND

Dialysis unit blood pressure (BP) pattern showed superiority in prognostic evaluation and interdialytic BP burden assessment. However previous studies mainly focused on the recurrent BP pattern within a session (intradialytic BP change or intradialytic BP slope), the clinical value of the weekly pattern of dialysis unit BP is unknown.

METHODS

We performed a prospective cohort study in adult end stage renal disease (ESRD) patients on thrice weekly hemodialysis (HD). The slope and the change of the postdialysis systolic BP (SBP) in the course of a week (post-SBP slope and post-SBP change) were used to characterize the weekly pattern of dialysis unit BP. Outcomes included all-cause mortality, cardiovascular mortality, and first cardiovascular event. We also measured the home BP in our cohort.

RESULTS

One hundred and twenty-nine subjects were followed over a median of 31 months. Higher post-SBP slope (≥0.185) was independently associated with increased risk of all-cause mortality, cardiovascular mortality, and first cardiovascular event. Results were similar for increased post-SBP change. HD patients with a higher post-SBP slope or an increased post-SBP change also had significant increased interdialytic BP burden measured by home SBP on both dialysis days and non-dialysis days.

CONCLUSIONS

Post-SBP slope and post-SBP change might be promising dialysis unit BP markers for prognostic evaluation and interdialytic BP burden assessment.

Study of dosimetry and clinical factors for assessment of xerostomia in head and neck squamous cell carcinoma treated by intensity-modulated radiotherapy: A prospective study

AIM:

Clinical and dosimetric factors related to toxicity in terms of xerostomia in patients with head and neck squamous cell cancer (HNSCC) treated with intensity-modulated radiotherapy (IMRT).

MATERIALS AND METHODS:

Patients older than 18 years, with the WHO Performance Status Score <2 with primary diagnosis of HNSCC Stage II, III, and IV who had undergone primary or postoperative radiotherapy (RT) treated by IMRT at the center, from November 2015 to November 2016 were included in the study. Patients were assessed by physical examination and questioned to score their quality of life for dryness (HNDR) and stickiness (HNSS) by EORTC-HN-35 (Hindi or English version) at baseline (before treatment), at 3, 6, and 12 months following treatment. The validation of EORTC-HN-35 for HNDR and HNSS in patients was handed.

RESULTS:

Thirty patients were included in the study. The mean symptom score values for HNSS at baseline, 3, 6, and 12 months' post-RT treatment were 17.8, 62.2, 64.4, and 20.8, respectively. Dryness and stickiness also increased over 3–6 months in follow-up but slightly relieved at 12 months, but it could not reach to baseline. In subgroup analysis, at baseline mean score of dryness of mouth in elderly patients (≥60 years) (P = 0.248), poor performance status (Eastern Cooperative Oncology Group 2) (P = 0.80) and patients with advanced stage (Stage III and IVA) (P = 0.185) was higher. Correlation of normal tissue complication probability for xerostomia with contralateral mean parotid gland showed insignificant linearity with shallow curve.

CONCLUSION:

Patients remained symptomatic for xerostomia chiefly till 6 months' postirradiation, but it was slightly relieved in 12 months but could not reach the baseline. Dosimetric sparing ofcontralateral parotid resulted in decreased probability of developing xerostomia.

Subclinical Vascular Brain Lesions in Young Adults With Acute Ischemic Stroke

BACKGROUND AND PURPOSE

Subclinical vascular brain lesions are highly prevalent in elderly patients with stroke. Little is known about predisposing factors and their impact on long-term outcome of patients with stroke at a young age.

METHODS

We quantified magnetic resonance-defined subclinical vascular brain lesions, including lacunes and white matter hyperintensities, perivascular spaces and cerebral microbleeds, and assessed total small-vessel disease (SVD) score in patients with first-ever acute ischemic stroke aged 18 to 45 years, and followed them up, as part of the multicentre Italian Project on Stroke in Young Adults. The primary end point was a composite of ischemic stroke, transient ischemic attack, myocardial infarction, or other arterial events. We assessed the predictive accuracy of magnetic resonance features and whether the addition of these markers improves outcome prediction over a validated clinical tool, such as the Italian Project on Stroke in Young Adults score.

RESULTS

Among 591 patients (males, 53.8%; mean age, 37.5±6.4 years), 117 (19.8%) had subclinical vascular brain lesions. Family history of stroke was associated with lacunes (odds ratio, 2.24 [95% CI, 1.30-3.84]) and total SVD score (odds ratio, 2.06 [95% CI, 1.20-3.53] for score≥1), hypertension with white matter hyperintensities (odds ratio, 2.29 [95% CI, 1.22-4.32]). After a median follow-up of 36.0 months (25th-75th percentile, 38.0), lacunes and total SVD score were associated with primary end point (hazard ratio, 2.13 [95% CI, 1.17-3.90] for lacunes; hazard ratio, 2.17 [95% CI, 1.20-3.90] for total SVD score ≥1), and the secondary end point brain ischemia (hazard ratio, 2.55 [95% CI, 1.36-4.75] for lacunes; hazard ratio, 2.61 [95% CI, 1.42-4.80] for total SVD score ≥1). The predictive performances of the models, including magnetic resonance features were comparable to those of the random model. Adding individual magnetic resonance features to the Italian Project on Stroke in Young Adults score did not improve model prediction.

CONCLUSIONS

Subclinical vascular brain lesions affect ≈2 in 10 young adults with ischemic stroke. Although lacunes and total SVD score are associated with thrombotic recurrence, they do not improve accuracy of outcome prediction over validated clinical predictors.

To screen or not to screen? The development of a prediction model for aorto-iliac stenosis in kidney transplant candidates

Screening for aorto-iliac stenosis is important in kidney transplant candidates as its presence affects pre-transplantation decisions regarding side of implantation and the need for an additional vascular procedure. Reliable imaging techniques to identify this condition require contrast fluid, which can be harmful in these patients. To guide patient selection for these imaging techniques, we aimed to develop a prediction model for the presence of aorto-iliac stenosis. Patients with contrast-enhanced imaging available in the pre-transplant screening between January 1st, 2000 and December 31st, 2018 were included. A prediction model was developed using multivariable logistic regression analysis and internally validated using bootstrap resampling. Model performance was assessed with the concordance index and calibration slope. Three hundred and seventy-three patients were included, 90 patients (24.1%) had imaging-proven aorto-iliac stenosis. Our final model included age, smoking, peripheral arterial disease, coronary artery disease, a previous transplant, intermittent claudication and the presence of a femoral artery murmur. The model yielded excellent discrimination (optimism-corrected concordance index: 0.83) and calibration (optimism-corrected calibration slope: 0.91). In conclusion, this prediction model can guide the development of standardized protocols to decide which patients should receive vascular screening to identify aorto-iliac stenosis. External validation is needed before this model can be implemented in patient care.

Netboost: Boosting-Supported Network Analysis Improves High-Dimensional Omics Prediction in Acute Myeloid Leukemia and Huntington's Disease

State-of-the art selection methods fail to identify weak but cumulative effects of features found in many high-dimensional omics datasets. Nevertheless, these features play an important role in certain diseases. We present Netboost, a three-step dimension reduction technique. First, a boosting-based filter is combined with the topological overlap measure to identify the essential edges of the network. Second, sparse hierarchical clustering is applied on the selected edges to identify modules and finally module information is aggregated by the first principal components. We demonstrate the application of the newly developed Netboost in combination with CoxBoost for survival prediction of DNA methylation and gene expression data from 180 acute myeloid leukemia (AML) patients and show, based on cross-validated prediction error curve estimates, its prediction superiority over variable selection on the full dataset as well as over an alternative clustering approach. The identified signature related to chromatin modifying enzymes was replicated in an independent dataset, the phase II AMLSG 12-09 study. In a second application we combine Netboost with Random Forest classification and improve the disease classification error in RNA-sequencing data of Huntington's disease mice. Netboost is a freely available Bioconductor R package for dimension reduction and hypothesis generation in high-dimensional omics applications.

Prediction of first cardiovascular disease event in 2.9 million individuals using Danish administrative healthcare data: a nationwide, registry-based derivation and validation study

Aims

The aim of this study was to derive and validate a risk prediction model with nationwide coverage to predict the individual and population-level risk of cardiovascular disease (CVD).

Methods and results

All 2.98 million Danish residents aged 30–85 years free of CVD were included on 1 January 2014 and followed through 31 December 2018 using nationwide administrative healthcare registries. Model predictors and outcome were pre-specified. Predictors were age, sex, education, use of antithrombotic, blood pressure-lowering, glucose-lowering, or lipid-lowering drugs, and a smoking proxy of smoking-cessation drug use or chronic obstructive pulmonary disease. Outcome was 5-year risk of first CVD event, a combination of ischaemic heart disease, heart failure, peripheral artery disease, stroke, or cardiovascular death. Predictions were computed using cause-specific Cox regression models. The final model fitted in the full data was internally-externally validated in each Danish Region. The model was well-calibrated in all regions. Area under the receiver operating characteristic curve (AUC) and Brier scores ranged from 76.3% to 79.6% and 3.3 to 4.4. The model was superior to an age-sex benchmark model with differences in AUC and Brier scores ranging from 1.2% to 1.5% and −0.02 to −0.03. Average predicted risks in each Danish municipality ranged from 2.8% to 5.9%. Predicted risks for a 66-year old ranged from 2.6% to 25.3%. Personalized predicted risks across ages 30–85 were presented in an online calculator (https://hjerteforeningen.shinyapps.io/cvd-risk-manuscript/).

Conclusion

A CVD risk prediction model based solely on nationwide administrative registry data provided accurate prediction of personal and population-level 5-year first CVD event risk in the Danish population. This may inform clinical and public health primary prevention efforts.

Development and Validation of Unplanned Extubation Prediction Models Using Intensive Care Unit Data: Retrospective, Comparative, Machine Learning Study

Background

Patient safety in the intensive care unit (ICU) is one of the most critical issues, and unplanned extubation (UE) is considered the most adverse event for patient safety. Prevention and early detection of such an event is an essential but difficult component of quality care.

Objective

This study aimed to develop and validate prediction models for UE in ICU patients using machine learning.

Methods

This study was conducted in an academic tertiary hospital in Seoul, Republic of Korea. The hospital had approximately 2000 inpatient beds and 120 ICU beds. As of January 2019, the hospital had approximately 9000 outpatients on a daily basis. The number of annual ICU admissions was approximately 10,000. We conducted a retrospective study between January 1, 2010, and December 31, 2018. A total of 6914 extubation cases were included. We developed a UE prediction model using machine learning algorithms, which included random forest (RF), logistic regression (LR), artificial neural network (ANN), and support vector machine (SVM). For evaluating the model’s performance, we used the area under the receiver operating characteristic curve (AUROC). The sensitivity, specificity, positive predictive value, negative predictive value, and F1 score were also determined for each model. For performance evaluation, we also used a calibration curve, the Brier score, and the integrated calibration index (ICI) to compare different models. The potential clinical usefulness of the best model at the best threshold was assessed through a net benefit approach using a decision curve.

Results

Among the 6914 extubation cases, 248 underwent UE. In the UE group, there were more males than females, higher use of physical restraints, and fewer surgeries. The incidence of UE was higher during the night shift as compared to the planned extubation group. The rate of reintubation within 24 hours and hospital mortality were higher in the UE group. The UE prediction algorithm was developed, and the AUROC for RF was 0.787, for LR was 0.762, for ANN was 0.763, and for SVM was 0.740.

Conclusions

We successfully developed and validated machine learning–based prediction models to predict UE in ICU patients using electronic health record data. The best AUROC was 0.787 and the sensitivity was 0.949, which was obtained using the RF algorithm. The RF model was well-calibrated, and the Brier score and ICI were 0.129 and 0.048, respectively. The proposed prediction model uses widely available variables to limit the additional workload on the clinician. Further, this evaluation suggests that the model holds potential for clinical usefulness.

Uniform criteria for total hip replacement surgery in patients with hip osteoarthritis: a decision tool to guide treatment decisions

Background

Uniform criteria for performing hip replacement surgery in hip osteoarthritis patients are currently lacking. As a result, variation in surgery and inappropriateness of care may occur. The aim of this study was to develop a consensus-based decision tool to support the decision-making process for hip replacement surgery.

Methods

Patients with a diagnosis of unilateral or bilateral osteoarthritis were included. Consensus rounds with orthopedic surgeons were organized to blindly reassess medical files and to decide whether surgery is indicated or not, based on all available pre-treatment information. We compared the outcomes obtained from the blind reassessment by the consensus group with the actual treatment. Furthermore, prediction models were fitted on the reassessment outcome to identify which set of clinical parameters would be most predictive and uniformly shared in the decision to operate.

Two prediction models were fitted, one model without radiologic outcomes and one model where radiologic outcomes were included.

Results

In total, 364 medical files of osteoarthritis patients were included and reassessed in the analyses. Key predictors in the prediction model without radiology were age, flexion, internal rotation and the Hip disability and Osteoarthritis Outcome Score–quality of life. The discriminative power was high (Area Under Receiver Operating Curve (AUC) = 0.86). Key predictors in the prediction model with radiology were age, internal rotation and Kellgren and Lawrence severity score (AUC = 0.94).

Conclusion

The study yielded a decision tool with uniform criteria for hip replacement surgery in osteoarthritis patients. The tool will guide the clinical decision-making process of physicians on whether to perform hip surgery and should be used together with information about patient preferences and social context.

A comparison of time to event analysis methods, using weight status and breast cancer as a case study

Survival analysis with cohort study data has been traditionally performed using Cox proportional hazards models. Random survival forests (RSFs), a machine learning method, now present an alternative method. Using the UK Women's Cohort Study (n = 34,493) we evaluate two methods: a Cox model and an RSF, to investigate the association between Body Mass Index and time to breast cancer incidence. Robustness of the models were assessed by cross validation and bootstraping. Histograms of bootstrap coefficients are reported. C-Indices and Integrated Brier Scores are reported for all models. In post-menopausal women, the Cox model Hazard Ratios (HR) for Overweight (OW) and Obese (O) were 1.25 (1.04, 1.51) and 1.28 (0.98, 1.68) respectively and the RSF Odds Ratios (OR) with partial dependence on menopause for OW and O were 1.34 (1.31, 1.70) and 1.45 (1.42, 1.48). HR are non-significant results. Only the RSF appears confident about the effect of weight status on time to event. Bootstrapping demonstrated Cox model coefficients can vary significantly, weakening interpretation potential. An RSF was used to produce partial dependence plots (PDPs) showing OW and O weight status increase the probability of breast cancer incidence in post-menopausal women. All models have relatively low C-Index and high Integrated Brier Score. The RSF overfits the data. In our study, RSF can identify complex non-proportional hazard type patterns in the data, and allow more complicated relationships to be investigated using PDPs, but it overfits limiting extrapolation of results to new instances. Moreover, it is less easily interpreted than Cox models. The value of survival analysis remains paramount and therefore machine learning techniques like RSF should be considered as another method for analysis.

EASIX for Prediction of Outcome in Hospitalized SARS-CoV-2 Infected Patients

Background

The coronavirus disease 2019 (COVID-19) is caused by severe acute respiratory syndrome coronavirus-2 (SARS-CoV-2) and has evoked a pandemic that challenges public health-care systems worldwide. Endothelial cell dysfunction plays a key role in pathophysiology, and simple prognosticators may help to optimize allocation of limited resources. Endothelial activation and stress index (EASIX) is a validated predictor of endothelial complications and outcome after allogeneic stem cell transplantation. Aim of this study was to test if EASIX could predict life-threatening complications in patients with COVID-19.

Methods

SARS-CoV-2-positive, hospitalized patients were enrolled onto a prospective non-interventional register study (n=100). Biomarkers were assessed at hospital admission. Primary endpoint was severe course of disease (mechanical ventilation and/or death, V/D). Results were validated in 126 patients treated in two independent institutions.

Results

EASIX at admission was a strong predictor of severe course of the disease (odds ratio for a two-fold change 3.4, 95%CI 1.8-6.3, p<0.001), time to V/D (hazard ratio (HR) for a two-fold change 2.0, 95%CI 1.5-2.6, p<0.001) as well as survival (HR for a two-fold change 1.7, 95%CI 1.2-2.5, p=0.006). The effect was retained in multivariable analysis adjusting for age, gender, and comorbidities and could be validated in the independent cohort. At hospital admission EASIX correlated with increased suppressor of tumorigenicity-2, soluble thrombomodulin, angiopoietin-2, CXCL8, CXCL9 and interleukin-18, but not interferon-alpha.

Conclusion

EASIX is a validated predictor of COVID19 outcome and an easy-to-access tool to segregate patients in need for intensive surveillance.