Reproducing RECIST lesion selection via machine learning: Insights into intra and inter-radiologist variation

Background

The Response Evaluation Criteria in Solid Tumors (RECIST) aims to provide a standardized approach to assess treatment response in solid tumors. However, discrepancies in the selection of measurable and target lesions among radiologists using these criteria pose a significant limitation to their reproducibility and accuracy. This study aimed to understand the factors contributing to this variability.

Methods

Machine learning models were used to replicate, in parallel, the selection process of measurable and target lesions by two radiologists in a cohort of 40 patients from an internal pan-cancer dataset. The models were trained on lesion characteristics such as size, shape, texture, rank, and proximity to other lesions. Ablation experiments were conducted to evaluate the impact of lesion diameter, volume, and rank on the selection process.

Results

The models successfully reproduced the selection of measurable lesions, relying primarily on size-related features. Similarly, the models reproduced target lesion selection, relying mostly on lesion rank. Beyond these features, the importance placed by different radiologists on different visual characteristics can vary, specifically when choosing target lesions. Worth noting that substantial variability was still observed between radiologists in both measurable and target lesion selection.

Conclusions

Despite the successful replication of lesion selection, our results still revealed significant inter-radiologist disagreement. This underscores the necessity for more precise guidelines to standardize lesion selection processes and minimize reliance on individual interpretation and experience as a means to bridge existing ambiguities.

Overcoming data scarcity in radiomics/radiogenomics using synthetic radiomic features

PURPOSE

To evaluate the potential of synthetic radiomic data generation in addressing data scarcity in radiomics/radiogenomics models.

METHODS

This study was conducted on a retrospectively collected cohort of 386 colorectal cancer patients (n = 2570 lesions) for whom matched contrast-enhanced CT images and gene TP53 mutational status were available. The full cohort data was divided into a training cohort (n = 2055 lesions) and an independent and fixed test set (n = 515 lesions). Differently sized training sets were subsampled from the training cohort to measure the impact of sample size on model performance and assess the added value of synthetic radiomic augmentation at different sizes. Five different tabular synthetic data generation models were used to generate synthetic radiomic data based on "real-world" radiomics data extracted from this cohort. The quality and reproducibility of the generated synthetic radiomic data were assessed. Synthetic radiomics were then combined with "real-world" radiomic training data to evaluate their impact on the predictive model's performance.

RESULTS

A prediction model was generated using only "real-world" radiomic data, revealing the impact of data scarcity in this particular data set through a lack of predictive performance at low training sample numbers (n = 200, 400, 1000 lesions with average AUC = 0.52, 0.53, and 0.56 respectively, compared to 0.64 when using 2055 training lesions). Synthetic tabular data generation models created reproducible synthetic radiomic data with properties highly similar to "real-world" data (for n = 1000 lesions, average Chi-square = 0.932, average basic statistical correlation = 0.844). The integration of synthetic radiomic data consistently enhanced the performance of predictive models trained with small sample size sets (AUC enhanced by 9.6%, 11.3%, and 16.7% for models trained on n_samples = 200, 400, and 1000 lesions, respectively). In contrast, synthetic data generated from randomised/noisy radiomic data failed to enhance predictive performance underlining the requirement of true signal data to do so.

CONCLUSION

Synthetic radiomic data, when combined with real radiomics, could enhance the performance of predictive models. Tabular synthetic data generation might help to overcome limitations in medical AI stemming from data scarcity.

Artificial Intelligence-based Quantification of Pleural Plaque Volume and Association With Lung Function in Asbestos-exposed Patients

PURPOSE

Pleural plaques (PPs) are morphologic manifestations of long-term asbestos exposure. The relationship between PP and lung function is not well understood, whereas the time-consuming nature of PP delineation to obtain volume impedes research. To automate the laborious task of delineation, we aimed to develop automatic artificial intelligence (AI)-driven segmentation of PP. Moreover, we aimed to explore the relationship between pleural plaque volume (PPV) and pulmonary function tests.

MATERIALS AND METHODS

Radiologists manually delineated PPs retrospectively in computed tomography (CT) images of patients with occupational exposure to asbestos (May 2014 to November 2019). We trained an AI model with a no-new-UNet architecture. The Dice Similarity Coefficient quantified the overlap between AI and radiologists. The Spearman correlation coefficient ( r ) was used for the correlation between PPV and pulmonary function test metrics. When recorded, these were vital capacity (VC), forced vital capacity (FVC), and diffusing capacity for carbon monoxide (DLCO).

RESULTS

We trained the AI system on 422 CT scans in 5 folds, each time with a different fold (n = 84 to 85) as a test set. On these independent test sets combined, the correlation between the predicted volumes and the ground truth was r = 0.90, and the median overlap was 0.71 Dice Similarity Coefficient. We found weak to moderate correlations with PPV for VC (n = 80, r = -0.40) and FVC (n = 82, r = -0.38), but no correlation for DLCO (n = 84, r = -0.09). When the cohort was split on the median PPV, we observed statistically significantly lower VC ( P = 0.001) and FVC ( P = 0.04) values for the higher PPV patients, but not for DLCO ( P = 0.19).

CONCLUSION

We successfully developed an AI algorithm to automatically segment PP in CT images to enable fast volume extraction. Moreover, we have observed that PPV is associated with loss in VC and FVC.

Development and multicenter validation of a multiparametric imaging model to predict treatment response in rectal cancer

OBJECTIVES

To develop and validate a multiparametric model to predict neoadjuvant treatment response in rectal cancer at baseline using a heterogeneous multicenter MRI dataset.

METHODS

Baseline staging MRIs (T2W (T2-weighted)-MRI, diffusion-weighted imaging (DWI) / apparent diffusion coefficient (ADC)) of 509 patients (9 centres) treated with neoadjuvant chemoradiotherapy (CRT) were collected. Response was defined as (1) complete versus incomplete response, or (2) good (Mandard tumor regression grade (TRG) 1-2) versus poor response (TRG3-5). Prediction models were developed using combinations of the following variable groups: (1) Non-imaging: age/sex/tumor-location/tumor-morphology/CRT-surgery interval (2) Basic staging: cT-stage/cN-stage/mesorectal fascia involvement, derived from (2a) original staging reports, or (2b) expert re-evaluation (3) Advanced staging: variables from 2b combined with cTN-substaging/invasion depth/extramural vascular invasion/tumor length (4) Quantitative imaging: tumour volume + first-order histogram features (from T2W-MRI and DWI/ADC) Models were developed with data from 6 centers (n = 412) using logistic regression with the Least Absolute Shrinkage and Selector Operator (LASSO) feature selection, internally validated using repeated (n = 100) random hold-out validation, and externally validated using data from 3 centers (n = 97).

RESULTS

After external validation, the best model (including non-imaging and advanced staging variables) achieved an area under the curve of 0.60 (95%CI=0.48-0.72) to predict complete response and 0.65 (95%CI=0.53-0.76) to predict a good response. Quantitative variables did not improve model performance. Basic staging variables consistently achieved lower performance compared to advanced staging variables.

CONCLUSIONS

Overall model performance was moderate. Best results were obtained using advanced staging variables, highlighting the importance of good-quality staging according to current guidelines. Quantitative imaging features had no added value (in this heterogeneous dataset).

CLINICAL RELEVANCE STATEMENT

Predicting tumour response at baseline could aid in tailoring neoadjuvant therapies for rectal cancer. This study shows that image-based prediction models are promising, though are negatively affected by variations in staging quality and MRI acquisition, urging the need for harmonization.

KEY POINTS

This multicenter study combining clinical information and features derived from MRI rendered disappointing performance to predict response to neoadjuvant treatment in rectal cancer. Best results were obtained with the combination of clinical baseline information and state-of-the-art image-based staging variables, highlighting the importance of good quality staging according to current guidelines and staging templates. No added value was found for quantitative imaging features in this multicenter retrospective study. This is likely related to acquisition variations, which is a major problem for feature reproducibility and thus model generalizability.

Radiomic signatures from T2W and DWI MRI are predictive of tumour hypoxia in colorectal liver metastases

BACKGROUND

Tumour hypoxia is a negative predictive and prognostic biomarker in colorectal cancer typically assessed by invasive sampling methods, which suffer from many shortcomings. This retrospective proof-of-principle study explores the potential of MRI-derived imaging markers in predicting tumour hypoxia non-invasively in patients with colorectal liver metastases (CLM).

METHODS

A single-centre cohort of 146 CLMs from 112 patients were segmented on preoperative T2-weighted (T2W) images and diffusion-weighted imaging (DWI). HIF-1 alpha immunohistochemical staining index (high/low) was used as a reference standard. Radiomic features were extracted, and machine learning approaches were implemented to predict the degree of histopathological tumour hypoxia.

RESULTS

Radiomic signatures from DWI b200 (AUC = 0.79, 95% CI 0.61-0.93, p = 0.002) and ADC (AUC = 0.72, 95% CI 0.50-0.90, p = 0.019) were significantly predictive of tumour hypoxia. Morphological T2W TE75 (AUC = 0.64, 95% CI 0.42-0.82, p = 0.092) and functional DWI b0 (AUC = 0.66, 95% CI 0.46-0.84, p = 0.069) and b800 (AUC = 0.64, 95% CI 0.44-0.82, p = 0.071) images also provided predictive information. T2W TE300 (AUC = 0.57, 95% CI 0.33-0.78, p = 0.312) and b = 10 (AUC = 0.53, 95% CI 0.33-0.74, p = 0.415) images were not predictive of tumour hypoxia.

CONCLUSIONS

T2W and DWI sequences encode information predictive of tumour hypoxia. Prospective multicentre studies could help develop and validate robust non-invasive hypoxia-detection algorithms.

CRITICAL RELEVANCE STATEMENT

Hypoxia is a negative prognostic biomarker in colorectal cancer. Hypoxia is usually assessed by invasive sampling methods. This proof-of-principle retrospective study explores the role of AI-based MRI-derived imaging biomarkers in non-invasively predicting tumour hypoxia in patients with colorectal liver metastases (CLM).

Pelvic CT in addition to MRI to differentiate between rectal and sigmoid cancer on imaging using the sigmoid take-off as a landmark

BACKGROUND

The sigmoid take-off (STO) is a recently established landmark to discern rectal from sigmoid cancer on imaging. STO-assessment can be challenging on magnetic resonance imaging (MRI) due to varying axial planes.

PURPOSE

To establish the benefit of using computed tomography (CT; with consistent axial planes), in addition to MRI, to anatomically classify rectal versus sigmoid cancer using the STO.

MATERIAL AND METHODS

A senior and junior radiologist retrospectively classified 40 patients with rectal/rectosigmoid cancers using the STO, first on MRI-only (sagittal and oblique-axial views) and then using a combination of MRI and axial CT. Tumors were classified as rectal/rectosigmoid/sigmoid (according to published STO definitions) and then dichotomized into rectal versus sigmoid. Diagnostic confidence was documented using a 5-point scale.

RESULTS

Adding CT resulted in a change in anatomical tumor classification in 4/40 cases (10%) for the junior reader and in 6/40 cases (15%) for the senior reader. Diagnostic confidence increased significantly after adding CT for the junior reader (mean score 3.85 vs. 4.27; P < 0.001); confidence of the senior reader was not affected (4.28 vs. 4.25; P = 0.80). Inter-observer agreement was similarly good for MRI only (κ=0.77) and MRI + CT (κ=0.76). Readers reached consensus on the classification of rectal versus sigmoid cancer in 78%-85% of cases.

CONCLUSION

Availability of a consistent axial imaging plane - in the case of this study provided by CT - in addition to a standard MRI protocol with sagittal and oblique-axial imaging views can be helpful to more confidently localize tumors using the STO as a landmark, especially for more junior readers.

MRI anatomy of the rectum: key concepts important for rectal cancer staging and treatment planning

A good understanding of the MRI anatomy of the rectum and its surroundings is pivotal to ensure high-quality diagnostic evaluation and reporting of rectal cancer. With this pictorial review, we aim to provide an image-based overview of key anatomical concepts essential for treatment planning, response evaluation and post-operative assessment. These concepts include the cross-sectional anatomy of the rectal wall in relation to T-staging; differences in staging and treatment between anal and rectal cancer; landmarks used to define the upper and lower boundaries of the rectum; the anatomy of the pelvic floor and anal canal, the mesorectal fascia, peritoneum and peritoneal reflection; and guides to help discern different pelvic lymph node stations on MRI to properly stage regional and non-regional rectal lymph node metastases. Finally, this review will highlight key aspects of post-treatment anatomy, including the assessment of radiation-induced changes and the evaluation of the post-operative pelvis after different surgical resection and reconstruction techniques.

Identifying high-risk colon cancer on CT an a radiomics signature improve radiologist's performance for T staging?

PURPOSE

To assess the role of radiomics in detection of high-risk (pT3-4) colon cancer and develop a combined model that combines both radiomics and CT staging of colon cancer.

METHODS

We included 292 colon cancer patients who underwent pre-operative CT and primary surgical resection within 2 months. Three-dimensional segmentations and CT staging of primary colon tumors were done. From each 3D segmentation of colon tumor, radiomic features were automatically extracted. Logistic regression analysis was performed to identify associations between radiomic features and high-risk (pT3-4) colon tumors. A combined model that integrated both radiomics and CT staging was developed and their diagnostic performance was compared with that of conventional CT staging. Tenfold cross-validation was used to validate the performance of the model and CT staging.

RESULTS

The model that combined radiomic features and CT staging demonstrated a significantly better performance in detection of high-risk colon tumors in training set (AUC = 0.799, 95% CI: 0.720-0.839 for combined model and AUC = 0.697, 95% CI = 0.538-0.756 for CT staging only, p < 0.001 for difference). Cross-validation results also demonstrated significantly better detection performance of combined model (AUC = 0.727, 95% Confidence Interval (CI): 0.621-0.777 for combined model and AUC = 0.628, 95% CI = 0.558-0.689 for CT staging only, Boot CI = 0.099).

CONCLUSION

CT radiomic features of primary colon cancer, combined with CT staging, can improve the detection of high-risk colon cancer patients.

Current controversies in TNM for the radiological staging of rectal cancer and how to deal with them: results of a global online survey and multidisciplinary expert consensus

Objectives

To identify the main problem areas in the applicability of the current TNM staging system (8th ed.) for the radiological staging and reporting of rectal cancer and provide practice recommendations on how to handle them.

Methods

A global case-based online survey was conducted including 41 image-based rectal cancer cases focusing on various items included in the TNM system. Cases reaching < 80% agreement among survey respondents were identified as problem areas and discussed among an international expert panel, including 5 radiologists, 6 colorectal surgeons, 4 radiation oncologists, and 3 pathologists.

Results

Three hundred twenty-one respondents (from 32 countries) completed the survey. Sixteen problem areas were identified, related to cT staging in low-rectal cancers, definitions for cT4b and cM1a disease, definitions for mesorectal fascia (MRF) involvement, evaluation of lymph nodes versus tumor deposits, and staging of lateral lymph nodes. The expert panel recommended strategies on how to handle these, including advice on cT-stage categorization in case of involvement of different layers of the anal canal, specifications on which structures to include in the definition of cT4b disease, how to define MRF involvement by the primary tumor and other tumor-bearing structures, how to differentiate and report lymph nodes and tumor deposits on MRI, and how to anatomically localize and stage lateral lymph nodes.

Conclusions

The recommendations derived from this global survey and expert panel discussion may serve as a practice guide and support tool for radiologists (and other clinicians) involved in the staging of rectal cancer and may contribute to improved consistency in radiological staging and reporting.

Key Points

• Via a case-based online survey (incl. 321 respondents from 32 countries), we identified 16 problem areas related to the applicability of the TNM staging system for the radiological staging and reporting of rectal cancer.

• A multidisciplinary panel of experts recommended strategies on how to handle these problem areas, including advice on cT-stage categorization in case of involvement of different layers of the anal canal, specifications on which structures to include in the definition of cT4b disease, how to define mesorectal fascia involvement by the primary tumor and other tumor-bearing structures, how to differentiate and report lymph nodes and tumor deposits on MRI, and how to anatomically localize and stage lateral lymph nodes.

• These recommendations may serve as a practice guide and support tool for radiologists (and other clinicians) involved in the staging of rectal cancer and may contribute to improved consistency in radiological staging and reporting.

Sources of variation in multicenter rectal MRI data and their effect on radiomics feature reproducibility

OBJECTIVES

To investigate sources of variation in a multicenter rectal cancer MRI dataset focusing on hardware and image acquisition, segmentation methodology, and radiomics feature extraction software.

METHODS

T2W and DWI/ADC MRIs from 649 rectal cancer patients were retrospectively acquired in 9 centers. Fifty-two imaging features (14 first-order/6 shape/32 higher-order) were extracted from each scan using whole-volume (expert/non-expert) and single-slice segmentations using two different software packages (PyRadiomics/CapTk). Influence of hardware, acquisition, and patient-intrinsic factors (age/gender/cTN-stage) on ADC was assessed using linear regression. Feature reproducibility was assessed between segmentation methods and software packages using the intraclass correlation coefficient.

RESULTS

Image features differed significantly (p < 0.001) between centers with more substantial variations in ADC compared to T2W-MRI. In total, 64.3% of the variation in mean ADC was explained by differences in hardware and acquisition, compared to 0.4% by patient-intrinsic factors. Feature reproducibility between expert and non-expert segmentations was good to excellent (median ICC 0.89-0.90). Reproducibility for single-slice versus whole-volume segmentations was substantially poorer (median ICC 0.40-0.58). Between software packages, reproducibility was good to excellent (median ICC 0.99) for most features (first-order/shape/GLCM/GLRLM) but poor for higher-order (GLSZM/NGTDM) features (median ICC 0.00-0.41).

CONCLUSIONS

Significant variations are present in multicenter MRI data, particularly related to differences in hardware and acquisition, which will likely negatively influence subsequent analysis if not corrected for. Segmentation variations had a minor impact when using whole volume segmentations. Between software packages, higher-order features were less reproducible and caution is warranted when implementing these in prediction models.

KEY POINTS

• Features derived from T2W-MRI and in particular ADC differ significantly between centers when performing multicenter data analysis. • Variations in ADC are mainly (> 60%) caused by hardware and image acquisition differences and less so (< 1%) by patient- or tumor-intrinsic variations. • Features derived using different image segmentations (expert/non-expert) were reproducible, provided that whole-volume segmentations were used. When using different feature extraction software packages with similar settings, higher-order features were less reproducible.

Evolutions in rectal cancer MRI staging and risk stratification in The Netherlands

Purpose

To analyze how the MRI reporting of rectal cancer has evolved (following guideline updates) in The Netherlands.

Methods

Retrospective analysis of 712 patients (2011–2018) from 8 teaching hospitals in The Netherlands with available original radiological staging reports that were re-evaluated by a dedicated MR expert using updated guideline criteria. Original reports were classified as “free-text,” “semi-structured,” or “template” and completeness of reporting was documented. Patients were categorized as low versus high risk, first based on the original reports (high risk = cT3-4, cN+, and/or cMRF+) and then based on the expert re-evaluations (high risk = cT3cd-4, cN+, MRF+, and/or EMVI+). Evolutions over time were studied by splitting the inclusion period in 3 equal time periods.

Results

A significant increase in template reporting was observed (from 1.6 to 17.6–29.6%; p < 0.001), along with a significant increase in the reporting of cT-substage, number of N+ and extramesorectal nodes, MRF invasion and tumor-MRF distance, EMVI, anal sphincter involvement, and tumor morphology and circumference. Expert re-evaluation changed the risk classification from high to low risk in 18.0% of cases and from low to high risk in 1.7% (total 19.7%). In the majority (17.9%) of these cases, the changed risk classification was likely (at least in part) related to use of updated guideline criteria, which mainly led to a reduction in high-risk cT-stage and nodal downstaging.

Conclusion

Updated concepts of risk stratification have increasingly been adopted, accompanied by an increase in template reporting and improved completeness of reporting. Use of updated guideline criteria resulted in considerable downstaging (of mainly high-risk cT-stage and nodal stage).

Graphic abstract

The sigmoid take-off as a landmark to distinguish rectal from sigmoid tumours on MRI: Reproducibility, pitfalls and potential impact on treatment stratification

PURPOSE

The sigmoid take-off (STO) was recently introduced as a preferred landmark, agreed upon by expert consensus recommendation, to discern rectal from sigmoid cancer on imaging. Aim of this study was to assess the reproducibility of the STO, explore its potential treatment impact and identify its main interpretation pitfalls.

METHODS

Eleven international radiologists (with varying expertise) retrospectively assessed n = 155 patients with previously clinically staged upper rectal/rectosigmoid tumours and re-classified them using the STO as completely below (rectum), straddling the STO (rectosigmoid) or completely above (sigmoid), after which scores were dichotomized as rectum (below/straddling STO) and sigmoid (above STO), being the clinically most relevant distinction. A random subset of n = 48 was assessed likewise by 6 colorectal surgeons. .

RESULTS

Interobserver agreement (IOA) for the 3-category score ranged from κ0.19-0.82 (radiologists) and κ0.32-0.72 (surgeons), with highest scores for the most experienced radiologists (κ0.69-0.76). Of the 155 cases, 44 (28%) were re-classified by ≥ 80% of radiologists as sigmoid cancers; 36 of these originally received neoadjuvant treatment which in retrospect might have been omitted if the STO had been applied. Main interpretation pitfalls were related to anatomical variations, borderline cases near the STO and angulation of axial imaging planes.

CONCLUSIONS

Good agreement was reached for experienced radiologists. Despite considerable variation among less-expert readers, use of the STO could have changed treatment in ±1/4 of patients in our cohort. Identified interpretation pitfalls may serve as a basis for teaching and to further optimize MR protocols.

Is MRCP necessary to diagnose pancreas divisum?

Background

The purpose of this study is to compare the performance of three-dimensional magnetic resonance cholangiopancreatography (3D-MRCP) with non-MRCP T2-weighted magnetic resonance imaging (MRI) sequences for diagnosis of pancreas divisum (PD).

Methods

This is a retrospective study of 342 consecutive patients with abdominal MRI including 3D-MRCP. 3D-MRCP was a coronal respiration-navigated T2-weighted sequence with 1.5 mm slice thickness. Non-MRCP T2-weighted sequences were (1) a coronal inversion recovery sequence (TIRM) with 6 mm slice thickness and (2) a transverse single shot turbo spin echo sequence (HASTE) with 4 mm slice thickness. For 3D-MRCP, TIRM, and HASTE, presence of PD and assessment of evaluability were determined in a randomized manner. A consensus read by two radiologists using 3D-MRCP, non-MRCP T2-weighted sequences, and other available imaging sequences served as reference standard for diagnosis of PD. Statistical analysis included performance analysis of 3D-MRCP, TIRM, and HASTE and testing for noninferiority of non-MRCP T2-weighted sequences compared with 3D-MRCP.

Results

Thirty-three of 342 patients (9.7%) were diagnosed with PD using the reference standard. Sensitivity/specificity of 3D-MRCP for detecting PD were 81.2%/69.7% (p < 0.001). Sensitivity/specificity of TIRM and HASTE were 92.5%/93.9 and 98.1%/97.0%, respectively (p < 0.001 each). Grouped sensitivity/specificity of non-MRCP T2-weighted sequences were 99.8%/91.0%. Non-MRCP T2-weighted sequences were non-inferior to 3D-MRCP alone for diagnosis of PD. 20.2, 7.3%, and 2.3% of 3D-MRCP, TIRM, and HASTE, respectively, were not evaluable due to motion artifacts or insufficient duct depiction.

Conclusions

Non-MRCP T2-weighted MRI sequences offer high performance for diagnosis of PD and are noninferior to 3D-MRCP alone.

Trial registration

Not applicable.