Observer variability in a phase II trial - assessing consistency in RECIST application

How Does Target Lesion Selection Affect RECIST? A Computer Simulation Study

OBJECTIVES

Response Evaluation Criteria in Solid Tumors (RECIST) is grounded on the assumption that target lesion selection is objective and representative of the change in total tumor burden (TTB) during therapy. A computer simulation model was designed to challenge this assumption, focusing on a particular aspect of subjectivity: target lesion selection.

MATERIALS AND METHODS

Disagreement among readers and the disagreement between individual reader measurements and TTB were analyzed as a function of the total number of lesions, affected organs, and lesion growth.

RESULTS

Disagreement rises when the number of lesions increases, when lesions are concentrated on a few organs, and when lesion growth borders the thresholds of progressive disease and partial response. There is an intrinsic methodological error in the estimation of TTB via RECIST 1.1, which depends on the number of lesions and their distributions. For example, for a fixed number of lesions at 5 and 15, distributed over a maximum of 4 organs, the error rates are observed to be 7.8% and 17.3%, respectively.

CONCLUSIONS

Our results demonstrate that RECIST can deliver an accurate estimate of TTB in localized disease, but fails in cases of distal metastases and multiple organ involvement. This is worsened by the "selection of the largest lesions," which introduces a bias that makes it hardly possible to perform an accurate estimate of the TTB. Including more (if not all) lesions in the quantitative analysis of tumor burden is desirable.

Evaluation of the capability and reproducibility of RECIST 1.1. measurements by technologists in breast cancer follow-up: a pilot study

The evaluation of tumor follow-up according to RECIST 1.1 has become essential in clinical practice given its role in therapeutic decision making. At the same time, radiologists are facing an increase in activity while facing a shortage. Radiographic technologists could contribute to the follow-up of these measures, but no studies have evaluated their ability to perform them. Ninety breast cancer patients were performed three CT follow-ups between September 2017 and August 2021. 270 follow-up treatment CT scans were analyzed including 445 target lesions. The rate of agreement of classifications RECIST 1.1 between five technologists and radiologists yielded moderate (k value between 0.47 and 0.52) and substantial (k value = 0.62 and k = 0.67) agreement values. 112 CT were classified as progressive disease (PD) by the radiologists, and 414 new lesions were identified. The analysis showed a percentage of strict agreement of progressive disease classification between reader-technologists and radiologists ranging from substantial to almost perfect agreement (range 73-97%). Analysis of intra-observer agreement was strong at almost perfect (k > 0.78) for 3 technologists. These results are encouraging regarding the ability of selected technologists to perform measurements according to RECIST 1.1 criteria by CT scan with good identification of disease progression.

Cetuximab plus irinotecan administered biweekly with reduced infusion time to heavily pretreated patients with metastatic colorectal cancer and related RAS and BRAF mutation status

Metastatic colorectal cancer (mCRC) is treated with cetuximab 250 mg/m2 administered weekly over 1 hour or biweekly (q2w) over 3.5 hours when combined with irinotecan. This prospective study investigated cetuximab 500 mg/m2 plus irinotecan 180 mg/m2 administered q2w over 1.5 hours independent of RAS or BRAF mutation status in mCRC patients in a third-line setting. The intention-to-treat population included 181 patients. No patients had complete response, 18% had partial responses (PR) and 48% stable disease (SD). For cetuximab, a relative dose intensity of ≥90% was reached in 78% and for irinotecan in 67% of the patients. Grade 3 to 4 toxicities were pain (17%), fatigue (9%), neutropenia (8%), diarrhea (8%), rash (8%), infection (7%) and hypersensitivity (3%). No deaths occurred. Next-generation sequencing in 96.7% of the patients revealed that 50.3% had RAS and BRAFV600E wild type (WT), with a mutation type (MT) in 45.1% of the RAS and 4.4% of the BRAFV600E genes. In patients with RAS-WT and RAS-MT tumors, a PR was obtained in 32% and 4% (P = .000003) and an SD in 43% and 53%, respectively, with a superior PFS (6.2 vs 3.7 months; hazard ratio [HR] 2.12, P = .00001) and OS (12.9 vs 8.8 months; HR 1.71, P = .0008). Treatment efficacy was poor in 7.4% of patients with an RAS mutation outside KRAS exon 2 and in 38% of patients with KRAS exon 2 mutations. Administration of cetuximab and irinotecan q2w, shortening treatment time from 3.5 to 1.5 hours, is recommended as standard therapy.

RECIST 1.1 and lesion selection: How to deal with ambiguity at baseline?

Response Evaluation Criteria In Solid Tumors (RECIST) is still the predominant criteria base for assessing tumor burden in oncology clinical trials. Despite several improvements that followed its first publication, RECIST continues to allow readers a lot of freedom in their evaluations. Notably in the selection of tumors at baseline. This subjectivity is the source of many suboptimal evaluations. When starting a baseline analysis, radiologists cannot always identify tumor malignancy with any certainty. Also, with RECIST, some findings can be deemed equivocal by radiologists with no confirmatory ground truth to rely on. In the specific case of Blinded Independent Central Review clinical trials with double reads using RECIST, the selection of equivocal tumors can have two major consequences: inter-reader variability and modified sensitivity of the therapeutic response. Apart from the main causes leading to the selection of an equivocal lesion, due to the uncertainty of the radiological characteristics or due to the censoring of on-site evaluations, several other situations can be described more precisely. These latter involve cases where an equivocal is selected as target or non-target lesions, the management of equivocal lymph nodes and the case of few target lesions. In all cases, awareness of the impact of selecting a non-malignant lesion will lead radiologists to make selections in the most rational way. Also, in clinical trials where the primary endpoint differs between phase 2 (response-related) and phase 3 (progression-related) trials, our impact analysis will help them to devise strategies for the management of equivocal lesions.

Intervention to Reduce Interobserver Variability in Computed Tomographic Measurement of Cancer Lesions Among Experienced Radiologists

While a growing number of research studies have reported the inter-observer variability in computed tomographic (CT) measurements, there are very few interventional studies performed. We aimed to assess whether a peer benchmarking intervention tool may have an influence on reducing interobserver variability in CT measurements and identify possible barriers to the intervention. In this retrospective study, 13 board-certified radiologists repeatedly reviewed 10 CT image sets of lung lesions and hepatic metastases during 3 noncontiguous time periods (T1, T2, T3). Each preselected case contained normal anatomy cephalad and caudal to the lesion of interest. Lesion size measurement under RECISTS 1.1 guidelines, choice of CT slice, and time spent on measurement were captured. Prior to their final measurements, the participants were exposed to the intervention designed to reduce the number of measurements deviating from the median. Chi-square test was performed to identify radiologist-dependent factors associated with the variability. The percent of deviating measurements during T1 and T2 were 20.0% and 23.1%, respectively. There was no statistically significant change in the number of deviating measurements upon the presentation of the intervention despite the decrease in percent from 23.1% to 17.7%. The identified barriers to the intervention include clinical disagreements among radiologists. Specifically, the inter-observer variability was associated with the controversy over the choice of CT image slice (P = 0.045) and selection of start-point, axis, and end-point (P = 0.011). Clinical disagreements rather than random errors were barriers to reducing interobserver variability in CT measurement among experienced radiologists. Future interventions could aim to resolve the disagreement in an interactive approach.

Interobserver Variation in Response Evaluation Criteria in Solid Tumors 1.1

PURPOSE

Response Evaluation Criteria in Solid Tumors (RECIST 1.1) is the gold standard for imaging response evaluation in cancer trials. We sought to evaluate consistency of applying RECIST 1.1 between 2 conventionally trained radiologists, designated as A and B; identify reasons for variation; and reconcile these differences for future studies.

METHODS

The study was approved as an institutional quality check exercise. Since no identifiable patient data was collected or used, a waiver of informed consent was granted. Imaging case report forms of a concluded multicentric breast cancer trial were retrospectively reviewed. Cohen's kappa was used to rate interobserver agreement in Response Evaluation Data (target response, nontarget response, new lesions, overall response). Significant variations were reassessed by a senior radiologist to extrapolate reasons for disagreement. Methods to improve agreement were similarly ascertained.

RESULTS

Sixty one cases with total of 82 data-pairs were evaluated (35 data-pairs in visit 5, 47 in visit 9). Both radiologists showed moderate agreement in target response (n = 82; ĸ = 0.477; 95% confidence interval [CI]: 0.314-0.640-), nontarget response (n = 82; ĸ = 0.578; 95% CI: 0.213-0.944) and overall response evaluation in both visits (n = 82; ĸ = 0.510; 95% CI: 0.344-0.676). Further assessment demonstrated "Prevalence effect" of Kappa in some cases which led to underestimation of agreement. Percent agreement of overall response was 74.39% while percent variation was 25.6%. Differences in interpreting RECIST 1.1 and in radiological image interpretation were the primary sources of variation. The commonest overall response was "Partial Response" (Rad A:45/82; Rad B:63/82).

CONCLUSION

Inspite of moderate interobserver agreement, qualitative interpretation differences in some cases increased interobserver variability. Protocols such as Adjudication, to reduce easily avoidable inconsistencies are or should be a part of the Standard Operating Procedure in imaging institutions. Based on our findings, a standard checklist has been developed to help reduce the interpretation error-margin for future studies. Such check-lists may improve interobserver agreement in the preadjudication phase thereby improving quality of results and reducing adjudication per case ratio.

CLINICAL RELEVANCE

Improving data reliability when using RECIST 1.1 will reflect in better cancer clinical trial outcomes. A checklist can be of use to imaging centers to assess and improve their own processes.

Discrepancies of assessments in a RECIST 1.1 phase II clinical trial - association between adjudication rate and variability in images and tumors selection

Background

In imaging-based clinical trials, it is common practice to perform double reads for each image, discrepant interpretations can result from these two different evaluations. In this study we analyzed discrepancies that occurred between local investigators (LI) and blinded independent central review (BICR) by comparing reader-selected imaging scans and lesions. Our goal was to identify the causes of discrepant declarations of progressive disease (PD) between LI and BICR in a clinical trial.

Methods

We retrospectively analyzed imaging data from a RECIST 1.1-based, multi-sites, phase II clinical trial of 179 patients with adult small cell lung cancer, treated with Cabazitaxel compared to Topotecan. Any discrepancies in the determination of PD between LI and BICR readers were reviewed by a third-party adjudicator. For each imaging time point and reader, we recorded the selected target lesions, non-target lesions, and new lesions. Odds ratios were calculated to measure the association between discrepant declarations of PD and the differences in reviewed imaging scans (e.g. same imaging modality but with different reconstruction parameters) and selected lesions. Reasons for discrepancies were analyzed.

Results

The average number of target lesions found by LI and BICR was respectively 2.9 and 3.4 per patient (p < 0.05), 18.4% of these target lesions were actually non-measurable. LI and BICR performed their evaluations based on different baseline imaging scans for 59% of the patients, they selected at least one different target lesion in 85% of patients. A total of 36.7% of patients required adjudication. Reasons of adjudication included differences in 1) reporting new lesions (53.7%), 2) the measured change of the tumor burden (18.5%), and 3) the progression of non-target lesions (11.2%). The rate of discrepancy was not associated with the selection of non-measurable target lesions or with the readers’ assessment of different images. Paradoxically, more discrepancies occurred when LI and BICR selected exactly the same target lesions at baseline compared to when readers selected not exactly the same lesions.

Conclusions

For a large proportion of evaluations, LI and BICR did not select the same imaging scans and target lesions but with a limited impact on the rate of discrepancy. The majority of discrepancies were explained by the difference in detecting new lesions.

Trial Registration

ARD12166 (https://clinicaltrials.gov/ct2/show/NCT01500720).

Inter-observer variability influences the Lugano classification when restaging lymphoma

Background

Lymphoma is an important and potentially curable oncological disease in South Africa. The staging and restaging of lymphoma have evolved over the years, with the latest international consensus guideline being the Lugano classification (LC). Prior to routine implementation of the LC, its robustness in the local setting should be determined.

Objectives

To determine the Inter-observer variability in response assignment when applying the LC in patients with lymphoma who were staged and restaged with computed tomography. In case of excessive discordance, specific mitigating measures will have to be taken before and during any proposed implementation of the LC.

Method

A total of 61 computed tomography scans in 21 patients were evaluated independently by four reviewers according to the LC, of which 21 scans were done at baseline, 21 at initial restaging and 19 at follow-up restaging. A retrospective comparative analysis was performed. Kappa values were calculated to determine agreement between observers.

Results

Only a moderate inter-observer agreement of 52% in the overall response classification was demonstrated. The most important sources of discrepancy were inconsistency in the assessment of target lesion regression to normal, determining the percentage change in the summed cross-sectional area of the target lesions and ascribing new lesions as either due to lymphoma or other causes.

Conclusion

Implementing the Lugano classification when restaging lymphoma is desirable to improve consistency and to conform to international guidelines. However, our study shows substantial inter-observer variability in response classification, potentially altering the treatment plan. Dedicated training and continuous quality control should, therefore, accompany the process.

Evaluation of inter-observer variability according to RECIST 1.1 and its influence on response classification in CT measurement of liver metastases

OBJECTIVE

The aim of this study is the evaluation of inter-observer variability in the measurement of liver metastases according to RECIST and its influence on response classification.

PATIENTS AND METHODS

A total of 100 radiologists measured liver target lesions, on pre- and post-chemotherapy CT scans of three patients. Each observer filled out a questionnaire about his personal and work features. The evaluations of a well experienced radiologist, considered as "the gold standard", were compared to those taken by the observers. The percentage of the observers in agreement with the reviewer, in terms of the response category and in terms of inter-observer variability, was calculated for each patient.

RESULTS

The percentage of the inter-observer agreement was elevated. Most of the observers in agreement with the reviewer were senior radiologists, while those who disagreed were junior radiologist, but this result did not reach a statistical significance. More than 30% of observers disagreed with the reviewer at least in one of the three cases.

CONCLUSIONS

RECIST measurements are reproducible on a large and heterogeneous population of radiologists. Age and expertise of the radiologist remain the most critical factors: this suggests a revision by well-experienced radiologists in clinical trials.

Childhood extracranial neoplasms: the role of imaging in drug development and clinical trials

Cancer is the leading cause of death in children older than 1 year of age and new drugs are necessary to improve outcomes. Imaging is crucial to the drug development process and assessment of therapeutic response. In adults, tumours are often assessed with CT using size criteria. Unfortunately, techniques established in adults are not necessarily applicable in children due to differing pathophysiology, ability to cooperate and increased susceptibility to ionising radiation. MRI, in particular quantitative MRI, has to date not been fully utilised in children with extracranial neoplasms. The specific challenges of imaging in children, the potential for functional imaging techniques to inform upon and their inclusion in clinical trials are discussed.

FOLFIRI and Cetuximab Every Second Week for First-Line Treatment of KRAS Wild-Type Metastatic Colorectal Cancer According to Phosphatase and Tensin Homolog Expression: A Phase II Study

BACKGROUND

Retrospective studies have suggested that phosphatase and tensin homolog (PTEN) expression might predict the efficacy of cetuximab in patients with KRAS wild-type metastatic colorectal cancer (mCRC). The present study was designed to prospectively evaluate the efficacy of first-line irinotecan, fluorouracil, and folinate (FOLFIRI) plus cetuximab every second week according to PTEN expression.

PATIENTS AND METHODS

Originally, patients with KRAS wild-type mCRC were randomly assigned to receive either FOLFIRI or cetuximab plus FOLFIRI (FOLFIRI-C). After a protocol amendment, the FOLFIRI arm was discontinued, and additional patients received FOLFIRI-C. Cox proportional hazard models were used to investigate the effect of PTEN and MET expression and BRAF and phosphatidylinositol-4,5-bisphosphate 3-kinase, catalytic subunit α mutations on progression-free survival (PFS) and overall survival (OS).

RESULTS

A total of 35 and 54 patients received FOLFIRI and FOLFIRI-C, respectively. For the patients assigned to FOLFIRI and FOLFIRI-C, the median OS was 17.7 and 23.3 months and the median PFS was 8.2 and 6.6 months, respectively. For patients receiving FOLFIRI-C, the loss of PTEN expression did not affect PFS or OS. Significant interactions for PFS were detected between the MET expression levels (P = .047) and BRAF mutation (P = .018) and treatment. On univariate analysis, BRAF mutation was significantly associated with shorter OS for patients receiving either FOLFIRI-C (P = .016) or FOLFIRI (P = .035). Multivariate analysis confirmed the independent prognostic value of BRAF mutation on OS and that of MET expression levels on PFS (P = .025) and OS (P = .028) but only in the patients receiving FOLFIRI alone. Adverse events with FOLFIRI-C were consistent with those expected from FOLFIRI plus weekly cetuximab.

CONCLUSION

Although prospective analysis of PTEN did not allow a validation of the prognostic value of this biomarker, an every second week cetuximab schedule, in addition to first-line FOLFIRI, was effective and well tolerated. The possible predictive value of MET expression levels warrants additional investigation.

Volume-based response evaluation with consensual lesion selection: a pilot study by using cloud solutions and comparison to RECIST 1.1

RATIONALE AND OBJECTIVES

Lesion volume is considered as a promising alternative to Response Evaluation Criteria in Solid Tumors (RECIST) to make tumor measurements more accurate and consistent, which would enable an earlier detection of temporal changes. In this article, we report the results of a pilot study aiming at evaluating the effects of a consensual lesion selection on volume-based response (VBR) assessments.

MATERIALS AND METHODS

Eleven patients with lung computed tomography scans acquired at three time points were selected from Reference Image Database to Evaluate Response to therapy in lung cancer (RIDER) and proprietary databases. Images were analyzed according to RECIST 1.1 and VBR criteria by three readers working in different geographic locations. Cloud solutions were used to connect readers and carry out a consensus process on the selection of lesions used for computing response. Because there are not currently accepted thresholds for computing VBR, we have applied a set of thresholds based on measurement variability (-35% and +55%). The benefit of this consensus was measured in terms of multiobserver agreement by using Fleiss kappa (κfleiss) and corresponding standard errors (SE).

RESULTS

VBR after consensual selection of target lesions allowed to obtain κfleiss = 0.85 (SE = 0.091), which increases up to 0.95 (SE = 0.092), if an extra consensus on new lesions is added. As a reference, the agreement when applying RECIST without consensus was κfleiss = 0.72 (SE = 0.088). These differences were found to be statistically significant according to a z-test.

CONCLUSIONS

An agreement on the selection of lesions allows reducing the inter-reader variability when computing VBR. Cloud solutions showed to be an interesting and feasible strategy for standardizing response evaluations, reducing variability, and increasing consistency of results in multicenter clinical trials.

Intraobserver and interobserver variability in computed tomography size and attenuation measurements in patients with renal cell carcinoma receiving antiangiogenic therapy: implications for alternative response criteria

BACKGROUND

Alternative response criteria have been proposed in patients with metastatic renal cell carcinoma (mRCC) who are receiving vascular endothelial growth factor (VEGF)-targeted therapy, including 10% tumor shrinkage as an indicator of response/outcome. However, to the authors' knowledge, intraobserver and interobserver measurement variability have not been defined in this setting. The objective of the current study was to determine intraobserver and interobserver agreement of computed tomography (CT) size and attenuation measurements to establish reproducible response indicators.

METHODS

Seventy-one patients with mRCC with 179 target lesions were enrolled in phase 2 and phase 3 trials of VEGF-targeted therapies and retrospectively studied with Institutional Review Board approval. Two radiologists independently measured the long axis diameter and mean attenuation of target lesions at baseline and on follow-up CT. Concordance correlation coefficients and Bland-Altman plots were used to assess intraobserver and interobserver agreement.

RESULTS

High concordance correlation coefficients (range, 0.8602-0.9984) were observed in all types of measurements. The 95% limits of agreement for the percentage change of the sum longest diameter was -7.30% to 7.86% for intraobserver variability, indicating that 10% tumor shrinkage represents a true change in tumor size when measured by a single observer. The 95% limits of interobserver variability were -16.3% to 15.4%. On multivariate analysis, the location of the lesion was found to significantly contribute to interobserver variability (P = .048). The 95% limits of intraobserver agreement for the percentage change in CT attenuation were -18.34% to 16.7%.

CONCLUSIONS

In patients with mRCC who are treated with VEGF inhibitors, 10% tumor shrinkage is a reproducible radiologic response indicator when baseline and follow-up studies are measured by a single radiologist. Lesion location contributes significantly to measurement variability and should be considered when selecting target lesions.

Comparison of EORTC criteria and PERCIST for PET/CT response evaluation of patients with metastatic colorectal cancer treated with irinotecan and cetuximab

The study aim was to compare European Organization for Research and Treatment of Cancer (EORTC) criteria with PET Response Criteria in Solid Tumors (PERCIST) for response evaluation of patients with metastatic colorectal cancer treated with a combination of the chemotherapeutic drug irinotecan and the monoclonal antibody cetuximab.

METHODS

From 2006 to 2009, patients with metastatic colorectal cancer were prospectively included in a phase II trial evaluating the combination of irinotecan and cetuximab every second week, as third-line treatment. (18)F-FDG PET/CT was performed between 1 and 14 d before the first treatment and after every fourth treatment cycle until progression was identified by CT with Response Evaluation Criteria in Solid Tumors (RECIST). Response evaluation with (18)F-FDG PET/CT was retrospectively performed according to both EORTC criteria and PERCIST, classifying the patients into 4 response categories: complete metabolic response (CMR), partial metabolic response (PMR), stable metabolic disease (SMD), and progressive metabolic disease (PMD). Individual best overall metabolic response (BOmR) was registered with both sets of criteria, as well as survival within response categories, and compared.

RESULTS

A total of 61 patients and 203 PET/CT scans were eligible for response evaluation. With EORTC criteria, 38 had PMR, 16 had SMD, and 7 had PMD as their BOmR. With PERCIST, 34 had PMR, 20 had SMD, and 7 had PMD as their BOmR. None of the patients had CMR. There was agreement between EORTC criteria and PERCIST in 87% of the patients, and the corresponding κ-coefficient was 0.76. Disagreements were confined to PMR and SMD. Median overall survival (OS) in months with EORTC criteria was 14.2 in the PMR group and 7.2 in the combined SMD + PMD group. With PERCIST, it was 14.5 in the PMR group and 7.9 in the SMD + PMD group.

CONCLUSION

Response evaluation with EORTC criteria and PERCIST gave similar responses and OS outcomes with good agreement on BOmR (κ-coefficient, 0.76) and similar significant differences in median OS between response groups. Compared with EORTC criteria, we find PERCIST unambiguous because of clear definitions and therefore more straightforward to use.

Comparison of response evaluation criteria in solid tumors with volumetric measurements for estimation of tumor burden in pancreatic adenocarcinoma and hepatocellular carcinoma

BACKGROUND

Response evaluation criteria in solid tumors (RECIST) is the accepted method for determining tumor progression. However, RECIST may not estimate disease burden accurately because the axial plane often does not produce the actual longest diameter. Volumetric measurements may be an alternative to better determine tumor size. Our aim was to compare volumetric measurements with RECIST in pancreatic ductal adenocarcinomas (PDA) and hepatocellular carcinomas (HCC).

METHODS

RECIST and volumetric measurements were determined in 9 patients with metastatic PDA and 17 patients with HCC who subsequently underwent liver transplantation. Gross pathologic measurements after hepatectomy also were analyzed for volumes.

RESULTS

Three-dimensional diameter in volumetric analysis was 38% and 36% higher than RECIST diameter in PDA and HCC, respectively (P < .01). However, RECIST yielded 78% and 23% larger estimated tumor volumes than volumetric analysis in PDA and HCC, respectively (P < .01). Gross pathologic volume in HCC showed a linear correlation with both volumetric analysis (r = .95; P < .01) and RECIST (r = .96; P < .01) but RECIST significantly overestimated gross pathologic volume by an average of 28% (P < .01) whereas volumetric analysis was similar to gross pathologic volume (P = .56). In categorizing treatment response in PDA, RECIST and volumetric analysis were in moderate agreement (κ = .49).

CONCLUSIONS

RECIST significantly may overestimate tumor burden compared with volumetric measurements in both PDA and HCC. Volumetric analysis may be the preferred method to detect tumor progression.