Measuring the performance of prediction models to personalize treatment choice

Development and Validation of PRE-SARC (PREdiction of SARCopenia Risk in Community Older Adults) Sarcopenia Prediction Model

OBJECTIVE

Reliable identification of high-risk older adults who are likely to develop sarcopenia is essential to implement targeted preventive measures and follow-up. However, no sarcopenia prediction model is currently available for community use. Our objective was to develop and validate a risk prediction model for calculating the 1-year absolute risk of developing sarcopenia in an aging population.

METHODS

One prospective population-based cohort of non-sarcopenic individuals aged 60 years or older were used for the development of a sarcopenia risk prediction model and model validation. Sarcopenia was defined according to the 2019 Asian Working Group for Sarcopenia consensus. Stepwise logistic regression was used to identify risk factors for sarcopenia incidence within a 1-year follow-up. Model performance was evaluated using the area under the receiver operating characteristics curve (AUROC) and calibration plot, respectively.

RESULTS

The development cohort included 1042 older adults, among whom 87 participants developed sarcopenia during a 1-year follow-up. The PRE-SARC (PREdiction of SARCopenia Risk in community older adults) model can accurately predict the 1-year risk of sarcopenia by using 7 easily accessible community-based predictors. The PRE-SARC model performed well in predicting sarcopenia, with an AUROC of 87% (95% CI, 0.83-0.90) and good calibration. Internal validation showed minimal optimism, with an adjusted AUROC of 0.85. The prediction score was categorized into 4 risk groups: low (0%-10%), moderate (>10%-20%), high (>20%-40%), and very high (>40%). The PRE-SARC model has been incorporated into an online risk calculator, which is freely accessible for daily clinical applications (https://sarcopeniariskprediction.shinyapps.io/dynnomapp/).

CONCLUSIONS

In community-dwelling individuals, the PRE-SARC model can accurately predict 1-year sarcopenia incidence. This model serves as a readily available and free accessible tool to identify older adults at high risk of sarcopenia, thereby facilitating personalized early preventive approaches and optimizing the utilization of health care resources.

Combining randomized and non-randomized data to predict heterogeneous effects of competing treatments

Some patients benefit from a treatment while others may do so less or do not benefit at all. We have previously developed a two-stage network meta-regression prediction model that synthesized randomized trials and evaluates how treatment effects vary across patient characteristics. In this article, we extended this model to combine different sources of types in different formats: aggregate data (AD) and individual participant data (IPD) from randomized and non-randomized evidence. In the first stage, a prognostic model is developed to predict the baseline risk of the outcome using a large cohort study. In the second stage, we recalibrated this prognostic model to improve our predictions for patients enrolled in randomized trials. In the third stage, we used the baseline risk as effect modifier in a network meta-regression model combining AD, IPD randomized clinical trial to estimate heterogeneous treatment effects. We illustrated the approach in the re-analysis of a network of studies comparing three drugs for relapsing-remitting multiple sclerosis. Several patient characteristics influence the baseline risk of relapse, which in turn modifies the effect of the drugs. The proposed model makes personalized predictions for health outcomes under several treatment options and encompasses all relevant randomized and non-randomized evidence.

Framework for personalized prediction of treatment response in relapsing-remitting multiple sclerosis: a replication study in independent data

BACKGROUND

Individualizing and optimizing treatment of relapsing-remitting multiple sclerosis patients is a challenging problem, which would benefit from a clinically valid decision support. Stühler et al. presented black box models for this aim which were developed and internally evaluated in a German registry but lacked external validation.

METHODS

In patients from the French OFSEP registry, we independently built and validated models predicting being free of relapse and free of confirmed disability progression (CDP), following the methodological roadmap and predictors reported by Stühler. Hierarchical Bayesian models were fit to predict the outcomes under 6 disease-modifying treatments given the individual disease course up to the moment of treatment change. Data was temporally split on 2017, and models were developed in patients treated earlier (n = 5517). Calibration curves, discrimination, mean squared error (MSE) and relative percentage of root MSE (RMSE%) were assessed by external validation of models in more-recent patients (n = 3768). Non-Bayesian fixed-effects GLMs were also applied and their outcomes were compared to these of the Bayesian ones. For both, we modelled the number of on-therapy relapses with a negative binomial distribution, and CDP occurrence with a binomial distribution.

RESULTS

The performance of our temporally-validated relapse model (MSE: 0.326, C-Index: 0.639) is potentially superior to that of Stühler's (MSE: 0.784, C-index: 0.608). Calibration plots revealed miscalibration. Our CDP model (MSE: 0.072, C-Index: 0.777) was also better than its counterpart (MSE: 0.131, C-index: 0.554). Results from non-Bayesian fixed-effects GLM models were similar to the Bayesian ones.

CONCLUSIONS

The relapse and CDP models rebuilt and externally validated in independent data could compare and strengthen the credibility of the Stühler models. Their model-building strategy was replicable.

Prediction Under Interventions: Evaluation of Counterfactual Performance Using Longitudinal Observational Data

Predictions under interventions are estimates of what a person's risk of an outcome would be if they were to follow a particular treatment strategy, given their individual characteristics. Such predictions can give important input to medical decision-making. However, evaluating the predictive performance of interventional predictions is challenging. Standard ways of evaluating predictive performance do not apply when using observational data, because prediction under interventions involves obtaining predictions of the outcome under conditions that are different from those that are observed for a subset of individuals in the validation dataset. This work describes methods for evaluating counterfactual performance of predictions under interventions for time-to-event outcomes. This means we aim to assess how well predictions would match the validation data if all individuals had followed the treatment strategy under which predictions are made. We focus on counterfactual performance evaluation using longitudinal observational data, and under treatment strategies that involve sustaining a particular treatment regime over time. We introduce an estimation approach using artificial censoring and inverse probability weighting that involves creating a validation dataset mimicking the treatment strategy under which predictions are made. We extend measures of calibration, discrimination (c-index and cumulative/dynamic AUCt) and overall prediction error (Brier score) to allow assessment of counterfactual performance. The methods are evaluated using a simulation study, including scenarios in which the methods should detect poor performance. Applying our methods in the context of liver transplantation shows that our procedure allows quantification of the performance of predictions supporting crucial decisions on organ allocation.

Development and validation of a prognostic score to identify the optimal candidate for preemptive TIPS in patients with cirrhosis and acute variceal bleeding

BACKGROUND AND AIM

Baveno VII workshop recommends the use of preemptive TIPS (p-TIPS) in patients with cirrhosis and acute variceal bleeding (AVB) at high- risk of treatment failure. However, the criteria defining "high-risk" have low clinical accessibility or include subjective variables. We aimed to develop and externally validate a model for better identification of p-TIPS candidates.

APPROACH AND RESULTS

The derivation cohort included 1554 patients with cirrhosis and AVB who were treated with endoscopy plus drug (n = 1264) or p-TIPS (n = 290) from 12 hospitals in China between 2010 and 2017. We first used competing risk regression to develop a score for predicting 6-week and 1-year mortality in patients treated with endoscopy plus drugs, which included age, albumin, bilirubin, international normalized ratio, white blood cell, creatinine, and sodium. The score was internally validated with the bootstrap method, which showed good discrimination (6 wk/1 y concordance-index: 0.766/0.740) and calibration, and outperformed other currently available models. In the second stage, the developed score was combined with treatment and their interaction term to predicate the treatment effect of p-TIPS (mortality risk difference between treatment groups) in the whole derivation cohort. The estimated treatment effect of p-TIPS varied substantially among patients. The prediction model had good discriminative ability (6 wk/1 y c -for-benefit: 0.696/0.665) and was well calibrated. These results were confirmed in the validation dataset of 445 patients with cirrhosis with AVB from 6 hospitals in China between 2017 and 2019 (6-wk/1-y c-for-benefit: 0.675/0.672).

CONCLUSIONS

We developed and validated a clinical prediction model that can help to identify individuals who will benefit from p-TIPS, which may guide clinical decision-making.

Comparison of causal forest and regression-based approaches to evaluate treatment effect heterogeneity: an application for type 2 diabetes precision medicine

OBJECTIVE

Precision medicine requires reliable identification of variation in patient-level outcomes with different available treatments, often termed treatment effect heterogeneity. We aimed to evaluate the comparative utility of individualized treatment selection strategies based on predicted individual-level treatment effects from a causal forest machine learning algorithm and a penalized regression model.

METHODS

Cohort study characterizing individual-level glucose-lowering response (6 month reduction in HbA1c) in people with type 2 diabetes initiating SGLT2-inhibitor or DPP4-inhibitor therapy. Model development set comprised 1,428 participants in the CANTATA-D and CANTATA-D2 randomised clinical trials of SGLT2-inhibitors versus DPP4-inhibitors. For external validation, calibration of observed versus predicted differences in HbA1c in patient strata defined by size of predicted HbA1c benefit was evaluated in 18,741 patients in UK primary care (Clinical Practice Research Datalink).

RESULTS

Heterogeneity in treatment effects was detected in clinical trial participants with both approaches (proportion predicted to have a benefit on SGLT2-inhibitor therapy over DPP4-inhibitor therapy: causal forest: 98.6%; penalized regression: 81.7%). In validation, calibration was good with penalized regression but sub-optimal with causal forest. A strata with an HbA1c benefit > 10 mmol/mol with SGLT2-inhibitors (3.7% of patients, observed benefit 11.0 mmol/mol [95%CI 8.0-14.0]) was identified using penalized regression but not causal forest, and a much larger strata with an HbA1c benefit 5-10 mmol with SGLT2-inhibitors was identified with penalized regression (regression: 20.9% of patients, observed benefit 7.8 mmol/mol (95%CI 6.7-8.9); causal forest 11.6%, observed benefit 8.7 mmol/mol (95%CI 7.4-10.1).

CONCLUSIONS

Consistent with recent results for outcome prediction with clinical data, when evaluating treatment effect heterogeneity researchers should not rely on causal forest or other similar machine learning algorithms alone, and must compare outputs with standard regression, which in this evaluation was superior.