External validation of NTCP-models for radiation pneumonitis in lung cancer patients treated with chemoradiotherapy

PURPOSE

Normal tissue complication probability (NTCP) models can be used to estimate the risk of radiation pneumonitis (RP). The aim of this study was to externally validate the most frequently used prediction models for RP, i.e., the QUANTEC and APPELT models, in a large cohort of lung cancer patients treated with IMRT or VMAT. [1-2] METHODS AND MATERIALS: This prospective cohort study, included lung cancer patients treated between 2013 and 2018. A closed testing procedure was performed to test the need for model updating. To improve model performance, modification or removal of variables was considered. Performance measures included tests for goodness of fit, discrimination, and calibration.

RESULTS

In this cohort of 612 patients, the incidence of RP ≥ grade 2 was 14.5%. For the QUANTEC-model, recalibration was recommended which resulted in a revised intercept and adjusted regression coefficient (from 0.126 to 0.224) of the mean lung dose (MLD),. The APPELT-model needed revision including model updating with modification and elimination of variables. After revision, the New RP-model included the following predictors (and regression coefficients): MLD (B = 0.250), age (B = 0.049, and smoking status (B = 0.902). The discrimination of the updated APPELT-model was higher compared to the recalibrated QUANTEC-model (AUC: 0.79 vs. 0.73).

CONCLUSIONS

This study demonstrated that both the QUANTEC- and APPELT-model needed revision. Next to changes of the intercept and regression coefficients, the APPELT model improved further by model updating and performed better than the recalibrated QUANTEC model. This New RP-model is widely applicable containing non-tumour site specific variables, which can easily be collected.

Comparison of dynamic updating strategies for clinical prediction models

BACKGROUND

Prediction models inform many medical decisions, but their performance often deteriorates over time. Several discrete-time update strategies have been proposed in the literature, including model recalibration and revision. However, these strategies have not been compared in the dynamic updating setting.

METHODS

We used post-lung transplant survival data during 2010-2015 and compared the Brier Score (BS), discrimination, and calibration of the following update strategies: (1) never update, (2) update using the closed testing procedure proposed in the literature, (3) always recalibrate the intercept, (4) always recalibrate the intercept and slope, and (5) always refit/revise the model. In each case, we explored update intervals of every 1, 2, 4, and 8 quarters. We also examined how the performance of the update strategies changed as the amount of old data included in the update (i.e., sliding window length) increased.

RESULTS

All methods of updating the model led to meaningful improvement in BS relative to never updating. More frequent updating yielded better BS, discrimination, and calibration, regardless of update strategy. Recalibration strategies led to more consistent improvements and less variability over time compared to the other updating strategies. Using longer sliding windows did not substantially impact the recalibration strategies, but did improve the discrimination and calibration of the closed testing procedure and model revision strategies.

CONCLUSIONS

Model updating leads to improved BS, with more frequent updating performing better than less frequent updating. Model recalibration strategies appeared to be the least sensitive to the update interval and sliding window length.

Permutation inference for canonical correlation analysis

Canonical correlation analysis (CCA) has become a key tool for population neuroimaging, allowing investigation of associations between many imaging and non-imaging measurements. As age, sex and other variables are often a source of variability not of direct interest, previous work has used CCA on residuals from a model that removes these effects, then proceeded directly to permutation inference. We show that a simple permutation test, as typically used to identify significant modes of shared variation on such data adjusted for nuisance variables, produces inflated error rates. The reason is that residualisation introduces dependencies among the observations that violate the exchangeability assumption. Even in the absence of nuisance variables, however, a simple permutation test for CCA also leads to excess error rates for all canonical correlations other than the first. The reason is that a simple permutation scheme does not ignore the variability already explained by previous canonical variables. Here we propose solutions for both problems: in the case of nuisance variables, we show that transforming the residuals to a lower dimensional basis where exchangeability holds results in a valid permutation test; for more general cases, with or without nuisance variables, we propose estimating the canonical correlations in a stepwise manner, removing at each iteration the variance already explained, while dealing with different number of variables in both sides. We also discuss how to address the multiplicity of tests, proposing an admissible test that is not conservative, and provide a complete algorithm for permutation inference for CCA.