Pitfalls in Prediction Modeling for Normal Tissue Toxicity in Radiation Therapy: An Illustration With the Individual Radiation Sensitivity and Mammary Carcinoma Risk Factor Investigation Cohorts

Comparing Performances of Predictive Models of Toxicity after Radiotherapy for Breast Cancer Using Different Machine Learning Approaches

PURPOSE

Different ML models were compared to predict toxicity in RT on a large cohort (n = 1314).

METHODS

The endpoint was RTOG G2/G3 acute toxicity, resulting in 204/1314 patients with the event. The dataset, including 25 clinical, anatomical, and dosimetric features, was split into 984 for training and 330 for internal tests. The dataset was standardized; features with a high p-value at univariate LR and with Spearman ρ>0.8 were excluded; synthesized data of the minority were generated to compensate for class imbalance. Twelve ML methods were considered. Model optimization and sequential backward selection were run to choose the best models with a parsimonious feature number. Finally, feature importance was derived for every model.

RESULTS

The model's performance was compared on a training-test dataset over different metrics: the best performance model was LightGBM. Logistic regression with three variables (LR3) selected via bootstrapping showed performances similar to the best-performing models. The AUC of test data is slightly above 0.65 for the best models (highest value: 0.662 with LightGBM).

CONCLUSIONS

No model performed the best for all metrics: more complex ML models had better performances; however, models with just three features showed performances comparable to the best models using many (n = 13-19) features.

Machine-learning prediction model for acute skin toxicity after breast radiation therapy using spectrophotometry

Purpose

Radiation-induced skin toxicity is a common and distressing side effect of breast radiation therapy (RT). We investigated the use of quantitative spectrophotometric markers as input parameters in supervised machine learning models to develop a predictive model for acute radiation toxicity.

Methods and materials

One hundred twenty-nine patients treated for adjuvant whole-breast radiotherapy were evaluated. Two spectrophotometer variables, i.e. the melanin (I_M) and erythema (I_E) indices, were used to quantitatively assess the skin physical changes. Measurements were performed at 4-time intervals: before RT, at the end of RT and 1 and 6 months after the end of RT. Together with clinical covariates, melanin and erythema indices were correlated with skin toxicity, evaluated using the Radiation Therapy Oncology Group (RTOG) guidelines. Binary group classes were labeled according to a RTOG cut-off score of ≥ 2. The patient's dataset was randomly split into a training and testing set used for model development/validation and testing (75%/25% split). A 5-times repeated holdout cross-validation was performed. Three supervised machine learning models, including support vector machine (SVM), classification and regression tree analysis (CART) and logistic regression (LR), were employed for modeling and skin toxicity prediction purposes.

Results

Thirty-four (26.4%) patients presented with adverse skin effects (RTOG ≥2) at the end of treatment. The two spectrophotometric variables at the beginning of RT (I_M,T0 and I_E,T0), together with the volumes of breast (PTV2) and boost surgical cavity (PTV1), the body mass index (BMI) and the dose fractionation scheme (FRAC) were found significantly associated with the RTOG score groups (p<0.05) in univariate analysis. The diagnostic performances measured by the area-under-curve (AUC) were 0.816, 0.734, 0.714, 0.691 and 0.664 for IM, IE, PTV2, PTV1 and BMI, respectively. Classification performances reported precision, recall and F1-values greater than 0.8 for all models. The SVM classifier using the RBF kernel had the best performance, with accuracy, precision, recall and F-score equal to 89.8%, 88.7%, 98.6% and 93.3%, respectively. CART analysis classified patients with I_M,T0 ≥ 99 to be associated with RTOG ≥ 2 toxicity; subsequently, PTV1 and PTV2 played a significant role in increasing the classification rate. The CART model provided a very high diagnostic performance of AUC=0.959.

Conclusions

Spectrophotometry is an objective and reliable tool able to assess radiation induced skin tissue injury. Using a machine learning approach, we were able to predict grade RTOG ≥2 skin toxicity in patients undergoing breast RT. This approach may prove useful for treatment management aiming to improve patient quality of life.

Development and optimisation of a machine-learning prediction model for acute desquamation following breast radiotherapy in the multi-centre REQUITE cohort

Purpose

Some patients with breast cancer treated by surgery and radiation therapy experience clinically significant toxicity, which may adversely affect cosmesis and quality of life. There is a paucity of validated clinical prediction models for radiation toxicity. We used machine learning (ML) algorithms to develop and optimise a clinical prediction model for acute breast desquamation after whole breast external beam radiation therapy in the prospective multicenter REQUITE cohort study.

Methods and Materials

Using demographic and treatment-related features (m = 122) from patients (n = 2058) at 26 centers, we trained 8 ML algorithms with 10-fold cross-validation in a 50:50 random-split data set with class stratification to predict acute breast desquamation. Based on performance in the validation data set, the logistic model tree, random forest, and naïve Bayes models were taken forward to cost-sensitive learning optimisation.

Results

One hundred and ninety-two patients experienced acute desquamation. Resampling and cost-sensitive learning optimisation facilitated an improvement in classification performance. Based on maximising sensitivity (true positives), the “hero” model was the cost-sensitive random forest algorithm with a false-negative: false-positive misclassification penalty of 90:1 containing m = 114 predictive features. Model sensitivity and specificity were 0.77 and 0.66, respectively, with an area under the curve of 0.77 in the validation cohort.

Conclusions

ML algorithms with resampling and cost-sensitive learning generated clinically valid prediction models for acute desquamation using patient demographic and treatment features. Further external validation and inclusion of genomic markers in ML prediction models are worthwhile, to identify patients at increased risk of toxicity who may benefit from supportive intervention or even a change in treatment plan.

External Validation of a Predictive Model for Acute Skin Radiation Toxicity in the REQUITE Breast Cohort

Background: Acute skin toxicity is a common and usually transient side-effect of breast radiotherapy although, if sufficiently severe, it can affect breast cosmesis, aftercare costs and the patient's quality-of-life. The aim of this study was to develop predictive models for acute skin toxicity using published risk factors and externally validate the models in patients recruited into the prospective multi-center REQUITE (validating pREdictive models and biomarkers of radiotherapy toxicity to reduce side-effects and improve QUalITy of lifE in cancer survivors) study.

Methods: Patient and treatment-related risk factors significantly associated with acute breast radiation toxicity on multivariate analysis were identified in the literature. These predictors were used to develop risk models for acute erythema and acute desquamation (skin loss) in three Radiogenomics Consortium cohorts of patients treated by breast-conserving surgery and whole breast external beam radiotherapy (n = 2,031). The models were externally validated in the REQUITE breast cancer cohort (n = 2,057).

Results: The final risk model for acute erythema included BMI, breast size, hypo-fractionation, boost, tamoxifen use and smoking status. This model was validated in REQUITE with moderate discrimination (AUC 0.65), calibration and agreement between predicted and observed toxicity (Brier score 0.17). The risk model for acute desquamation, excluding the predictor tamoxifen use, failed to validate in the REQUITE cohort.

Conclusions: While most published prediction research in the field has focused on model development, this study reports successful external validation of a predictive model using clinical risk factors for acute erythema following radiotherapy after breast-conserving surgery. This model retained discriminatory power but will benefit from further re-calibration. A similar model to predict acute desquamation failed to validate in the REQUITE cohort. Future improvements and more accurate predictions are expected through the addition of genetic markers and application of other modeling and machine learning techniques.

Machine Learning-Based Models for Prediction of Toxicity Outcomes in Radiotherapy

In order to limit radiotherapy (RT)-related side effects, effective toxicity prediction and assessment schemes are essential. In recent years, the growing interest toward artificial intelligence and machine learning (ML) within the science community has led to the implementation of innovative tools in RT. Several researchers have demonstrated the high performance of ML-based models in predicting toxicity, but the application of these approaches in clinics is still lagging, partly due to their low interpretability. Therefore, an overview of contemporary research is needed in order to familiarize practitioners with common methods and strategies. Here, we present a review of ML-based models for predicting and classifying RT-induced complications from both a methodological and a clinical standpoint, focusing on the type of features considered, the ML methods used, and the main results achieved. Our work overviews published research in multiple cancer sites, including brain, breast, esophagus, gynecological, head and neck, liver, lung, and prostate cancers. The aim is to define the current state of the art and main achievements within the field for both researchers and clinicians.

REQUITE: A prospective multicentre cohort study of patients undergoing radiotherapy for breast, lung or prostate cancer

PURPOSE

REQUITE aimed to establish a resource for multi-national validation of models and biomarkers that predict risk of late toxicity following radiotherapy. The purpose of this article is to provide summary descriptive data.

METHODS

An international, prospective cohort study recruited cancer patients in 26 hospitals in eight countries between April 2014 and March 2017. Target recruitment was 5300 patients. Eligible patients had breast, prostate or lung cancer and planned potentially curable radiotherapy. Radiotherapy was prescribed according to local regimens, but centres used standardised data collection forms. Pre-treatment blood samples were collected. Patients were followed for a minimum of 12 (lung) or 24 (breast/prostate) months and summary descriptive statistics were generated.

RESULTS

The study recruited 2069 breast (99% of target), 1808 prostate (86%) and 561 lung (51%) cancer patients. The centralised, accessible database includes: physician- (47,025 forms) and patient- (54,901) reported outcomes; 11,563 breast photos; 17,107 DICOMs and 12,684 DVHs. Imputed genotype data are available for 4223 patients with European ancestry (1948 breast, 1728 prostate, 547 lung). Radiation-induced lymphocyte apoptosis (RILA) assay data are available for 1319 patients. DNA (n = 4409) and PAXgene tubes (n = 3039) are stored in the centralised biobank. Example prevalences of 2-year (1-year for lung) grade ≥2 CTCAE toxicities are 13% atrophy (breast), 3% rectal bleeding (prostate) and 27% dyspnoea (lung).

CONCLUSION

The comprehensive centralised database and linked biobank is a valuable resource for the radiotherapy community for validating predictive models and biomarkers.

PATIENT SUMMARY

Up to half of cancer patients undergo radiation therapy and irradiation of surrounding healthy tissue is unavoidable. Damage to healthy tissue can affect short- and long-term quality-of-life. Not all patients are equally sensitive to radiation "damage" but it is not possible at the moment to identify those who are. REQUITE was established with the aim of trying to understand more about how we could predict radiation sensitivity. The purpose of this paper is to provide an overview and summary of the data and material available. In the REQUITE study 4400 breast, prostate and lung cancer patients filled out questionnaires and donated blood. A large amount of data was collected in the same way. With all these data and samples a database and biobank were created that showed it is possible to collect this kind of information in a standardised way across countries. In the future, our database and linked biobank will be a resource for research and validation of clinical predictors and models of radiation sensitivity. REQUITE will also enable a better understanding of how many people suffer with radiotherapy toxicity.

High-dimensional prediction of binary outcomes in the presence of between-study heterogeneity

Many prediction methods have been proposed in the literature, but most of them ignore heterogeneity between populations. Either only data from a single study or population is available for model building and evaluation, or when data from multiple studies make up the training dataset, studies are pooled before model building. As a result, prediction models might perform less than expected when applied to new subjects from new study populations. We propose a linear method for building prediction models with high-dimensional data from multiple studies. Our method explicitly addresses between-population variability and tends to select predictors that are predictive in most of the study populations. We employ empirical Bayes estimators and hence avoid selection bias during the variable selection process. Simulation results demonstrate that the new method works better than other linear prediction methods that ignore the between-study variability. Our method is developed for classification into two groups.

Method for Automatic Selection of Parameters in Normal Tissue Complication Probability Modeling

PURPOSE

To present a fully automatic method to generate multiparameter normal tissue complication probability (NTCP) models and compare its results with those of a published model, using the same patient cohort.

METHODS AND MATERIALS

Data were analyzed from 345 rectal cancer patients treated with external radiation therapy to predict the risk of patients developing grade 1 or ≥2 cystitis. In total, 23 clinical factors were included in the analysis as candidate predictors of cystitis. Principal component analysis was used to decompose the bladder dose-volume histogram into 8 principal components, explaining more than 95% of the variance. The data set of clinical factors and principal components was divided into training (70%) and test (30%) data sets, with the training data set used by the algorithm to compute an NTCP model. The first step of the algorithm was to obtain a bootstrap sample, followed by multicollinearity reduction using the variance inflation factor and genetic algorithm optimization to determine an ordinal logistic regression model that minimizes the Bayesian information criterion. The process was repeated 100 times, and the model with the minimum Bayesian information criterion was recorded on each iteration. The most frequent model was selected as the final "automatically generated model" (AGM). The published model and AGM were fitted on the training data sets, and the risk of cystitis was calculated.

RESULTS

The 2 models had no significant differences in predictive performance, both for the training and test data sets (P value > .05) and found similar clinical and dosimetric factors as predictors. Both models exhibited good explanatory performance on the training data set (P values > .44), which was reduced on the test data sets (P values < .05).

CONCLUSIONS

The predictive value of the AGM is equivalent to that of the expert-derived published model. It demonstrates potential in saving time, tackling problems with a large number of parameters, and standardizing variable selection in NTCP modeling.

Stomach Dose-Volume Predicts Acute Gastrointestinal Toxicity in Chemoradiotherapy for Locally Advanced Pancreatic Cancer

AIMS

Gastrointestinal toxicity impedes dose escalation in chemoradiotherapy for hepatobiliary malignancies. Toxicity risk depends on clinical and radiotherapy metrics. We aimed to identify predictive factors using data from two prospective phase II clinical trials of locally advanced pancreatic cancer (LAPC).

MATERIALS AND METHODS

Ninety-one patients with available data from the ARCII (59.4 Gy in 33 fractions with gemcitabine, cisplatin and nelfinavir, n = 23) and SCALOP (50.4 Gy in 28 fractions with capecitabine or gemcitabine, n = 74) trials were studied. The independent variables analysed comprised age, sex, performance status, baseline symptoms, tumour size, weight loss, chemotherapy regimen and dose-volume histogram of stomach and duodenum in 5 Gy bins. The outcome measures used were Common Terminology Criteria of Adverse Events (CTCAE) grade and risk of CTCAE grade ≥2 acute upper gastrointestinal toxicity (anorexia, pain, nausea and/or vomiting). The risk of CTCAE grade ≥2 events was modelled using multivariable logistic regression and prediction of severity grade using ordinal regression.

RESULTS

CTCAE grade ≥2 symptoms occurred in 38 patients (42%). On univariate analysis, stomach V35-45Gy was predictive of risk (odds ratio 1.035, 95% confidence interval 1.007-1.063) and grade (1.023, 1.003-1.044) of toxicity. The area under the curve was 0.632 (0.516-0.747) with toxicity risk 33/66 (50%) above and 5/25 (20%) below the optimal discriminatory threshold (7.1 cm3). Using a threshold of 30 cm3, risk was 13/20 (65%) versus 25/71 (35%). The optimal multivariable logistic regression model incorporated patient sex, chemotherapy regimen and stomach V35-45Gy. Receiving gemcitabine rather than capecitabine (odds ratio 3.965, 95% confidence interval 1.274-12.342) and weight loss during induction chemotherapy (1.216, 1.043-1.419) were significant predictors for the SCALOP cohort, whereas age predicted toxicity risk in ARCII only (1.344, 1.015-1.780). Duodenum dose-volume did not predict toxicity risk or severity in any cohort.

CONCLUSIONS

In chemoradiotherapy for LAPC the volume of stomach irradiated to a moderately high dose (35-45 Gy) predicts the incidence and severity of acute toxicity. Other predictive factors can include age, sex, recent weight loss and concomitant chemotherapy agents.