The impact of inter-observer variation in delineation on robustness of radiomics features in non-small cell lung cancer

Mitigating Interobserver Variability in Radiomics with ComBat: A Feasibility Study

This study investigates Intraobserver Features Variability (IFV) in radiomics studies and assesses the effectiveness of the ComBat harmonization method in mitigating these effects. Methods: This study utilizes data from the NSCLC-Radiomics-Interobserver1 dataset, comprising CT scans of 22 Non-Small Cell Lung Cancer (NSCLC) patients, with multiple Gross Tumor Volume (GTV) delineations performed by five radiation oncologists. Segmentation was completed manually ("vis") or by autosegmentation with manual editing ("auto"). A total of 1229 radiomic features were extracted from each GTV, segmentation method, and oncologist. Features extracted included first order, shape, GLCM, GLRLM, GLSZM, and GLDM from original, wavelet-filtered, and LoG-filtered images. Results: Before implementing ComBat harmonization, 83% of features exhibited p-values below 0.05 in the "vis" approach; this percentage decreased to 34% post-harmonization. Similarly, for the "auto" approach, 75% of features demonstrated statistical significance prior to ComBat, but this figure declined to 33% after its application. Among a subset of three expert radiation oncologists, percentages changed from 77% to 25% for "vis" contouring and from 64% to 23% for "auto" contouring. This study demonstrates that ComBat harmonization could effectively reduce IFV, enhancing the feasibility of multicenter radiomics studies. It also highlights the significant impact of physician experience on radiomics analysis outcomes.

Post-radiotherapy stage III/IV non-small cell lung cancer radiomics research: a systematic review and comparison of CLEAR and RQS frameworks

BACKGROUND

Lung cancer, the second most common cancer, presents persistently dismal prognoses. Radiomics, a promising field, aims to provide novel imaging biomarkers to improve outcomes. However, clinical translation faces reproducibility challenges, despite efforts to address them with quality scoring tools.

OBJECTIVE

This study had two objectives: 1) identify radiomics biomarkers in post-radiotherapy stage III/IV nonsmall cell lung cancer (NSCLC) patients, 2) evaluate research quality using the CLEAR (CheckList_for_EvaluAtion_of_Radiomics_research), RQS (Radiomics_Quality_Score) frameworks, and formulate an amalgamated CLEAR-RQS tool to enhance scientific rigor.

MATERIALS AND METHODS

A systematic literature review (Jun-Aug 2023, MEDLINE/PubMed/SCOPUS) was conducted concerning stage III/IV NSCLC, radiotherapy, and radiomic features (RF). Extracted data included study design particulars, such as sample size, radiotherapy/CT technique, selected RFs, and endpoints. CLEAR and RQS were merged into a CLEAR-RQS checklist. Three readers appraised articles utilizing CLEAR, RQS, and CLEAR-RQS metrics.

RESULTS

Out of 871 articles, 11 met the inclusion/exclusion criteria. The Median cohort size was 91 (range: 10-337) with 9 studies being single-center. No common RF were identified. The merged CLEAR-RQS checklist comprised 61 items. Most unreported items were within CLEAR's "methods" and "open-source," and within RQS's "phantom-calibration," "registry-enrolled prospective-trial-design," and "cost-effective-analysis" sections. No study scored above 50% on RQS. Median CLEAR scores were 55.74% (32.33/58 points), and for RQS, 17.59% (6.3/36 points). CLEAR-RQS article ranking fell between CLEAR and RQS and aligned with CLEAR.

CONCLUSION

Radiomics research in post-radiotherapy stage III/IV NSCLC exhibits variability and frequently low-quality reporting. The formulated CLEAR-RQS checklist may facilitate education and holds promise for enhancing radiomics research quality.

CLINICAL RELEVANCE STATEMENT

Current radiomics research in the field of stage III/IV postradiotherapy NSCLC is heterogenous, lacking reproducibility, with no identified imaging biomarker. Radiomics research quality assessment tools may enhance scientific rigor and thereby facilitate radiomics translation into clinical practice.

KEY POINTS

There is heterogenous and low radiomics research quality in postradiotherapy stage III/IV nonsmall cell lung cancer. Barriers to reproducibility are small cohort size, nonvalidated studies, missing technical parameters, and lack of data, code, and model sharing. CLEAR (CheckList_for_EvaluAtion_of_Radiomics_research), RQS (Radiomics_Quality_Score), and the amalgamated CLEAR-RQS tool are useful frameworks for assessing radiomics research quality and may provide a valuable resource for educational purposes in the field of radiomics.

Interobserver delineation variability of computed tomography-based radiomic features of the parotid gland

PURPOSE

To assess the interobserver delineation variability of radiomic features of the parotid gland from computed tomography (CT) images and evaluate the correlation of these features for head and neck cancer (HNC) radiotherapy patients.

MATERIALS AND METHODS

Contrast-enhanced CT images of 20 HNC patients were utilized. The parotid glands were delineated by treating radiation oncologists (ROs), a selected RO and AccuContour auto-segmentation software. Dice similarity coefficients (DSCs) between each pair of observers were calculated. A total of 107 radiomic features were extracted, whose robustness to interobserver delineation was assessed using the intraclass correlation coefficient (ICC). Pearson correlation coefficients (r) were calculated to determine the relationship between the features. The influence of excluding unrobust features from normal tissue complication probability (NTCP) modeling was investigated for severe oral mucositis (grade ≥3).

RESULTS

The average DSC was 0.84 (95% confidence interval, 0.83-0.86). Most of the shape features demonstrated robustness (ICC ≥0.75), while the first-order and texture features were influenced by delineation variability. Among the three observers investigated, 42 features were sufficiently robust, out of which 36 features exhibited weak correlation (|r|<0.8). No significant difference in the robustness level was found when comparing manual segmentation by a single RO or automated segmentation with the actual clinical contour data made by treating ROs. Excluding unrobust features from the NTCP model for severe oral mucositis did not deteriorate the model performance.

CONCLUSION

Interobserver delineation variability had substantial impact on radiomic features of the parotid gland. Both manual and automated segmentation methods contributed similarly to this variation.

A clinically relevant computed tomography (CT) radiomics strategy for intracranial rodent brain tumour monitoring

Here, we establish a CT-radiomics based method for application in invasive, orthotopic rodent brain tumour models. Twenty four NOD/SCID mice were implanted with U87R-Luc2 GBM cells and longitudinally imaged via contrast enhanced (CE-CT) imaging. Pyradiomics was employed to extract CT-radiomic features from the tumour-implanted hemisphere and non-tumour-implanted hemisphere of acquired CT-scans. Inter-correlated features were removed (Spearman correlation > 0.85) and remaining features underwent predictive analysis (recursive feature elimination or Boruta algorithm). An area under the curve of the receiver operating characteristic curve was implemented to evaluate radiomic features for their capacity to predict defined outcomes. Firstly, we identified a subset of radiomic features which distinguish the tumour-implanted hemisphere and non- tumour-implanted hemisphere (i.e, tumour presence from normal tissue). Secondly, we successfully translate preclinical CT-radiomic pipelines to GBM patient CT scans (n = 10), identifying similar trends in tumour-specific feature intensities (E.g. 'glszm Zone Entropy'), thereby suggesting a mouse-to-human species conservation (a conservation of radiomic features across species). Thirdly, comparison of features across timepoints identify features which support preclinical tumour detection earlier than is possible by visual assessment of CT scans. This work establishes robust, preclinical CT-radiomic pipelines and describes the application of CE-CT for in-depth orthotopic brain tumour monitoring. Overall we provide evidence for the role of pre-clinical 'discovery' radiomics in the neuro-oncology space.

Prediction of 177Lu-DOTATATE Therapy Outcomes in Neuroendocrine Tumor Patients Using Semi-Automatic Tumor Delineation on 68Ga-DOTATATE PET/CT

BACKGROUND

Treatment of metastatic neuroendocrine tumors (NET) with 177Lu-DOTATATE peptide receptor radionuclide therapy (PRRT) results in favorable response only in a subset of patients. We investigated the prognostic value of quantitative pre-treatment semi-automatic 68Ga-DOTATATE PET/CT analysis in NET patients treated with PRRT.

METHODS

The medical records of 94 NET patients who received at least one cycle of PRRT at a single institution were retrospectively reviewed. On each pre-treatment 68Ga-DOTATATE PET/CT, the total tumor volume (TTV), maximum tumor standardized uptake value for the patient (SUVmax), and average uptake in the lesion with the lowest radiotracer uptake (SUVmin) were determined with a semi-automatic tumor delineation method. Progression-free survival (PFS) and overall survival (OS) among the patients were compared based on optimal cutoff values for the imaging parameters.

RESULTS

On Kaplan-Meier analysis and univariate Cox regression, significantly shorter PFS was observed in patients with lower SUVmax, lower SUVmin, and higher TTV. On multivariate Cox regression, lower SUVmin and higher TTV remained predictive of shorter PFS. Only higher TTV was found to be predictive of shorter OS on Kaplan-Meier and Cox regression analyses. In a post hoc Kaplan-Meier analysis, patients with at least one high-risk feature (low SUVmin or high TTV) showed shorter PFS and OS, which may be the most convenient parameter to measure in clinical practice.

CONCLUSIONS

The tumor volume and lowest lesion uptake on 68Ga-DOTATATE PET/CT can predict disease progression following PRRT in NET patients, with the former also predictive of overall survival. NET patients at risk for poor outcomes following PRRT can be identified with semi-automated quantitative analysis of 68Ga-DOTATATE PET/CT.

Assessment of Variabilities in Lung-Contouring Methods on CBCT Preclinical Radiomics Outputs

Simple Summary

This study is the first to evaluate the impact of contouring differences on radiomics analysis in preclinical CBCT scans. We found that the variation in quantitative image readouts was greater between segmentation tools than between observers.

Abstract

Radiomics image analysis has the potential to uncover disease characteristics for the development of predictive signatures and personalised radiotherapy treatment. Inter-observer and inter-software delineation variabilities are known to have downstream effects on radiomics features, reducing the reliability of the analysis. The purpose of this study was to investigate the impact of these variabilities on radiomics outputs from preclinical cone-beam computed tomography (CBCT) scans. Inter-observer variabilities were assessed using manual and semi-automated contours of mouse lungs (n = 16). Inter-software variabilities were determined between two tools (3D Slicer and ITK-SNAP). The contours were compared using Dice similarity coefficient (DSC) scores and the 95th percentile of the Hausdorff distance (HD_95p) metrics. The good reliability of the radiomics outputs was defined using intraclass correlation coefficients (ICC) and their 95% confidence intervals. The median DSC scores were high (0.82–0.94), and the HD_95p metrics were within the submillimetre range for all comparisons. the shape and NGTDM features were impacted the most. Manual contours had the most reliable features (73%), followed by semi-automated (66%) and inter-software (51%) variabilities. From a total of 842 features, 314 robust features overlapped across all contouring methodologies. In addition, our results have a 70% overlap with features identified from clinical inter-observer studies.

The role of artificial intelligence based on PET/CT radiomics in NSCLC: Disease management, opportunities, and challenges

Objectives

Lung cancer has been widely characterized through radiomics and artificial intelligence (AI). This review aims to summarize the published studies of AI based on positron emission tomography/computed tomography (PET/CT) radiomics in non-small-cell lung cancer (NSCLC).

Materials and methods

A comprehensive search of literature published between 2012 and 2022 was conducted on the PubMed database. There were no language or publication status restrictions on the search. About 127 articles in the search results were screened and gradually excluded according to the exclusion criteria. Finally, this review included 39 articles for analysis.

Results

Classification is conducted according to purposes and several studies were identified at each stage of disease:1) Cancer detection (n=8), 2) histology and stage of cancer (n=11), 3) metastases (n=6), 4) genotype (n=6), 5) treatment outcome and survival (n=8). There is a wide range of heterogeneity among studies due to differences in patient sources, evaluation criteria and workflow of radiomics. On the whole, most models show diagnostic performance comparable to or even better than experts, and the common problems are repeatability and clinical transformability.

Conclusion

AI-based PET/CT Radiomics play potential roles in NSCLC clinical management. However, there is still a long way to go before being translated into clinical application. Large-scale, multi-center, prospective research is the direction of future efforts, while we need to face the risk of repeatability of radiomics features and the limitation of access to large databases.

Clinical validation of a semi‐automated segmentation algorithm for target volume definition on planning CT and CBCT in stereotactic body radiotherapy ( SBRT ) for peripheral lung lesions

INTRODUCTION

Stereotactic body radiotherapy (SBRT) is an ablative method for lung malignancies. Here, the definition of the gross target volume (GTV) is subject to interobserver variation. In this study, we aimed to evaluate the interobserver variability during SBRT and its dosimetric impact, as well as to introduce a semi-automated delineation tool for both planning computer tomography (P-CT) and cone beam CT (CBCT) to help to standardise GTV delineation and adaptive volume-change registration.

METHODS

The interobserver variation of GTV manual contours from five physicians was analysed in 15 patients after lung SBRT on free breathing (FB) P-CT (n = 15) and CBCT (n = 90) before and after each fraction. The dosimetric impact from interobserver variations of GTV based on the original treatment plan was analysed. Next, the accuracy of an in-house easy-to-use semi-automated-segmentation algorithm for pulmonary lesions was compared with gold standard contours in FB P-CT and CBCT, as well as 4D P-CT of additional 10 patients.

RESULTS

The interobserver variability in manual contours resulted in violations of dose coverage of the planning target volume (PTV), which, in turn, resulted in compromised tumour control probability in contours from four physicians. The validation of the semi-automated delineation algorithm using thorax phantom led to a highly reliable accuracy in defining GTVs. Comparing the unsupervised auto-contours with the gold standard delineation revealed high equal high concordance for FB P-CT, 4D P-CT and CBCT, with a DSC of 0.83, 0.76 and 0.8, respectively. The supervised use of the semi-automated delineation tool improved its accuracy, with DSCs of 0.86, 0.86 and 0.8 for FB P-CT, 4D P-CT and CBCT, respectively. The use of the algorithm was associated with a significantly shorter working time. The semi-automated delineation tool can accurately register volume changes in CBCTs.

CONCLUSION

The segmentation algorithm provides a reliable, standardised and time-saving alternative for manual delineation in lung SBRT in P-CT and CBCT.