Radiologists and Clinical Trials: Part 1 The Truth About Reader Disagreements

RECIST 1.1 assessments variability: a systematic pictorial review of blinded double reads

Reader variability is intrinsic to radiologic oncology assessments, necessitating measures to enhance consistency and accuracy. RECIST 1.1 criteria play a crucial role in mitigating this variability by standardizing evaluations, aiming to establish an accepted "truth" confirmed by histology or patient survival. Clinical trials utilize Blind Independent Centralized Review (BICR) techniques to manage variability, employing double reads and adjudicators to address inter-observer discordance effectively. It is essential to dissect the root causes of variability in response assessments, with a specific focus on the factors influencing RECIST evaluations. We propose proactive measures for radiologists to address variability sources such as radiologist expertise, image quality, and accessibility of contextual information, which significantly impact interpretation and assessment precision. Adherence to standardization and RECIST guidelines is pivotal in diminishing variability and ensuring uniform results across studies. Variability factors, including lesion selection, new lesion appearance, and confirmation bias, can have profound implications on assessment accuracy and interpretation, underscoring the importance of identifying and addressing these factors. Delving into the causes of variability aids in enhancing the accuracy and consistency of response assessments in oncology, underscoring the role of standardized evaluation protocols and mitigating risk factors that contribute to variability. Access to contextual information is crucial. CRITICAL RELEVANCE STATEMENT: By understanding the causes of diagnosis variability, we can enhance the accuracy and consistency of response assessments in oncology, ultimately improving patient care and clinical outcomes. KEY POINTS: Baseline lesion selection and detection of new lesions play a major role in the occurrence of discordance. Image interpretation is influenced by contextual information, the lack of which can lead to diagnostic uncertainty. Radiologists must be trained in RECIST criteria to reduce errors and variability.

Low-contrast lesion detection in neck CT: a multireader study comparing deep learning, iterative, and filtered back projection reconstructions using realistic phantoms

BACKGROUND

Computed tomography (CT) reconstruction algorithms can improve image quality, especially deep learning reconstruction (DLR). We compared DLR, iterative reconstruction (IR), and filtered back projection (FBP) for lesion detection in neck CT.

METHODS

Nine patient-mimicking neck phantoms were examined with a 320-slice scanner at six doses: 0.5, 1, 1.6, 2.1, 3.1, and 5.2 mGy. Each of eight phantoms contained one circular lesion (diameter 1 cm; contrast -30 HU to the background) in the parapharyngeal space; one phantom had no lesions. Reconstruction was made using FBP, IR, and DLR. Thirteen readers were tasked with identifying and localizing lesions in 32 images with a lesion and 20 without lesions for each dose and reconstruction algorithm. Receiver operating characteristic (ROC) and localization ROC (LROC) analysis were performed.

RESULTS

DLR improved lesion detection with ROC area under the curve (AUC) 0.724 ± 0.023 (mean ± standard error of the mean) using DLR versus 0.696 ± 0.021 using IR (p = 0.037) and 0.671 ± 0.023 using FBP (p < 0.001). Likewise, DLR improved lesion localization, with LROC AUC 0.407 ± 0.039 versus 0.338 ± 0.041 using IR (p = 0.002) and 0.313 ± 0.044 using FBP (p < 0.001). Dose reduction to 0.5 mGy compromised lesion detection in FBP-reconstructed images compared to doses ≥ 2.1 mGy (p ≤ 0.024), while no effect was observed with DLR or IR (p ≥ 0.058).

CONCLUSION

DLR improved the detectability of lesions in neck CT imaging. Dose reduction to 0.5 mGy maintained lesion detectability when denoising reconstruction was used.

RELEVANCE STATEMENT

Deep learning enhances lesion detection in neck CT imaging compared to iterative reconstruction and filtered back projection, offering improved diagnostic performance and potential for x-ray dose reduction.

KEY POINTS

Low-contrast lesion detectability was assessed in anatomically realistic neck CT phantoms. Deep learning reconstruction (DLR) outperformed filtered back projection and iterative reconstruction. Dose has little impact on lesion detectability against anatomical background structures.

Can blood-based markers predict RECIST progression in non-small cell lung cancer treated with immunotherapy?

PURPOSE

In this study, we aimed to evaluate the potential of routine blood markers, serum tumour markers and their combination in predicting RECIST-defined progression in patients with stage IV non-small cell lung cancer (NSCLC) undergoing treatment with immune checkpoint inhibitors.

METHODS

We employed time-varying statistical models and machine learning classifiers in a Monte Carlo cross-validation approach to investigate the association between RECIST-defined progression and blood markers, serum tumour markers and their combination, in a retrospective cohort of 164 patients with NSCLC.

RESULTS

The performance of the routine blood markers in the prediction of progression free survival was moderate. Serum tumour markers and their combination with routine blood markers generally improved performance compared to routine blood markers alone. Elevated levels of C-reactive protein (CRP) and alkaline phosphatase (ALP) ranked as the top predictive routine blood markers, and CYFRA 21.1 was consistently among the most predictive serum tumour markers. Using these classifiers to predict overall survival yielded moderate to high performance, even when cases of death-defined progression were excluded. Performance varied across the treatment journey.

CONCLUSION

Routine blood tests, especially when combined with serum tumour markers, show moderate predictive value  of RECIST-defined progression in NSCLC patients receiving immune checkpoint inhibitors. The relationship between overall survival and RECIST-defined progression may be influenced by confounding factors.

Iodine density mapping for the diagnosis of acute bowel ischemia using fast kV-switching dual-energy CT

PURPOSE

To evaluate the diagnostic capabilities of a supplementary color ramped iodine density map compared to virtual monoenergetic images (VMIs) at 74 keV in the diagnosis of acute bowel ischemia (ABI).

METHODS

Data for this study were prospectively gathered and retrospectively evaluated. Patients referred to the Department of Diagnostic Radiology between October 2020 and August 2022 on the suspicion of ABI and underwent surgery < 12 h following fast kV-switching venous phase abdominal dual-energy CT (DECT) were consecutively included. Images were evaluated by two board-certified radiologists and two radiology residents. First round included only 74 keV VMIs resembling conventional 120 kVp images, and the second round included a supplementary iodine density map. Readers were asked to register presence of ABI as well as their confidence in their diagnosis based on a 5-point Likert scale. Sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) were calculated for each observer with the surgical findings as the gold-standard. McNemar's and Wilcoxon signed-rank test were used to compare registrations and diagnostic confidence across assessment rounds.

RESULTS

A total of 29 patients resulting in 31 DECT scans were included. Fourteen cases of ischemic/necrotic bowel were reported following surgery. Sensitivity and NPV were decreased with the use of supplementary iodine map images compared to 120 kVp-like images without supplementary iodine map images for three of four observers (round 1 range: 71.4-92.9% and 78.0-94.8%; round 2 range: 57.1-78.6% and 70.1-83.3%, respectively), while specificity and PPV were increased for three of four observers (round 1 range: 64.7-94.1% and 67.4-93.1%; round 2 range: 88.2-94.1% and 73.8-91.1%, respectively). However, no significant difference in ABI diagnosis or diagnostic confidence was found (p-value range: 0.07-1.00 and 0.23-0.58, respectively).

CONCLUSION

No significant difference for the diagnosis of ABI was found using supplementary iodine mapping. Our study may suggest a trend of increased specificity and decreased sensitivity, hence, the use of supplementary iodine mapping should be carefully considered.

Can we predict discordant RECIST 1.1 evaluations in double read clinical trials?

Background

In lung clinical trials with imaging, blinded independent central review with double reads is recommended to reduce evaluation bias and the Response Evaluation Criteria In Solid Tumor (RECIST) is still widely used. We retrospectively analyzed the inter-reader discrepancies rate over time, the risk factors for discrepancies related to baseline evaluations, and the potential of machine learning to predict inter-reader discrepancies.

Materials and methods

We retrospectively analyzed five BICR clinical trials for patients on immunotherapy or targeted therapy for lung cancer. Double reads of 1724 patients involving 17 radiologists were performed using RECIST 1.1. We evaluated the rate of discrepancies over time according to four endpoints: progressive disease declared (PDD), date of progressive disease (DOPD), best overall response (BOR), and date of the first response (DOFR). Risk factors associated with discrepancies were analyzed, two predictive models were evaluated.

Results

At the end of trials, the discrepancy rates between trials were not different. On average, the discrepancy rates were 21.0%, 41.0%, 28.8%, and 48.8% for PDD, DOPD, BOR, and DOFR, respectively. Over time, the discrepancy rate was higher for DOFR than DOPD, and the rates increased as the trial progressed, even after accrual was completed. It was rare for readers to not find any disease, for less than 7% of patients, at least one reader selected non-measurable disease only (NTL). Often the readers selected some of their target lesions (TLs) and NTLs in different organs, with ranges of 36.0-57.9% and 60.5-73.5% of patients, respectively. Rarely (4-8.1%) two readers selected all their TLs in different locations. Significant risk factors were different depending on the endpoint and the trial being considered. Prediction had a poor performance but the positive predictive value was higher than 80%. The best classification was obtained with BOR.

Conclusion

Predicting discordance rates necessitates having knowledge of patient accrual, patient survival, and the probability of discordances over time. In lung cancer trials, although risk factors for inter-reader discrepancies are known, they are weakly significant, the ability to predict discrepancies from baseline data is limited. To boost prediction accuracy, it would be necessary to enhance baseline-derived features or create new ones, considering other risk factors and looking into optimal reader associations.

The role of clinical imaging in oncology drug development: progress and new challenges

In 2008, the role of clinical imaging in oncology drug development was reviewed. The review outlined where imaging was being applied and considered the diverse demands across the phases of drug development. A limited set of imaging techniques was being used, largely based on structural measures of disease evaluated using established response criteria such as response evaluation criteria in solid tumours. Beyond structure, functional tissue imaging such as dynamic contrast-enhanced MRI and metabolic measures using [18F]flourodeoxyglucose positron emission tomography were being increasingly incorporated. Specific challenges related to the implementation of imaging were outlined including standardisation of scanning across study centres and consistency of analysis and reporting. More than a decade on the needs of modern drug development are reviewed, how imaging has evolved to support new drug development demands, the potential to translate state-of-the-art methods into routine tools and what is needed to enable the effective use of this broadening clinical trial toolset. In this review, we challenge the clinical and scientific imaging community to help refine existing clinical trial methods and innovate to deliver the next generation of techniques. Strong industry-academic partnerships and pre-competitive opportunities to co-ordinate efforts will ensure imaging technologies maintain a crucial role delivering innovative medicines to treat cancer.

Breaking down the RECIST 1.1 double read variability in lung trials: What do baseline assessments tell us?

Background

In clinical trials with imaging, Blinded Independent Central Review (BICR) with double reads ensures data blinding and reduces bias in drug evaluations. As double reads can cause discrepancies, evaluations require close monitoring which substantially increases clinical trial costs. We sought to document the variability of double reads at baseline, and variabilities across individual readers and lung trials.

Material and methods

We retrospectively analyzed data from five BICR clinical trials evaluating 1720 lung cancer patients treated with immunotherapy or targeted therapy. Fifteen radiologists were involved. The variability was analyzed using a set of 71 features derived from tumor selection, measurements, and disease location. We selected a subset of readers that evaluated ≥50 patients in ≥two trials, to compare individual reader’s selections. Finally, we evaluated inter-trial homogeneity using a subset of patients for whom both readers assessed the exact same disease locations. Significance level was 0.05. Multiple pair-wise comparisons of continuous variables and proportions were performed using one-way ANOVA and Marascuilo procedure, respectively.

Results

Across trials, on average per patient, target lesion (TL) number ranged 1.9 to 3.0, sum of tumor diameter (SOD) 57.1 to 91.9 mm. MeanSOD=83.7 mm. In four trials, MeanSOD of double reads was significantly different. Less than 10% of patients had TLs selected in completely different organs and 43.5% had at least one selected in different organs. Discrepancies in disease locations happened mainly in lymph nodes (20.1%) and bones (12.2%). Discrepancies in measurable disease happened mainly in lung (19.6%). Between individual readers, the MeanSOD and disease selection were significantly different (p&lt;0.001). In inter-trials comparisons, on average per patient, the number of selected TLs ranged 2.1 to 2.8, MeanSOD 61.0 to 92.4 mm. Trials were significantly different in MeanSOD (p&lt;0.0001) and average number of selected TLs (p=0.007). The proportion of patients having one of the top diseases was significantly different only between two trials for lung. Significant differences were observed for all other disease locations (p&lt;0.05).

Conclusions

We found significant double read variabilities at baseline, evidence of reading patterns and a means to compare trials. Clinical trial reliability is influenced by the interplay of readers, patients and trial design.

Inter-rater reliability of retrograde urethrograms

PURPOSE

Reliability of pre-operative testing is important for adequate surgical planning. For urethral stricture disease, preoperative planning frequently includes retrograde urethrogram (RUG). The radiographic interpretation of RUGs is often done by urologists themselves. We aimed to evaluate the reliability of RUG interpretation by urologists at our institution.

METHODS

We examined the RUGs of 193 patients. These were deidentified and interpreted by three urologists, two general urologists and one reconstructive urologist. These interpretations were compared in 2 ways. Each of the general urologists was compared to the "gold standard" reconstructive urologist interpretation, and the general urologists were additionally compared to each other. We used intraclass correlation coefficient (ICC) for numerical variables and Fleiss' Kappa or Cohen's Kappa statistic (κ) for categorical variables to rate inter-interpreter reliability and agreement among interpretations with regards to the quantitative variables of stricture length and caliber.

RESULTS

Level of agreement ranged from poor to moderate across all variables interpreted. Comparing general urologists to the gold standard yielded no better than moderate agreement, with the majority being poor to fair. Similarly, agreement amongst the general urologists did not reach above moderate, with the majority being poor to slight.

CONCLUSION

To our knowledge, this is the first analysis of inter-rater reliability of RUGs among practicing urologists. Our analysis showed clinically unacceptable reliability with regards to stricture length, location, caliber, and indicated procedures. This study suggests a need for standardized interpretation of RUGs and poses an opportunity for actionable improvement in management of strictures.

Application of the Lugano Classification for Initial Evaluation, Staging, and Response Assessment of Hodgkin and Non-Hodgkin Lymphoma: The PRoLoG Consensus Initiative (Part 2-Technical)

The aim of this initiative was to provide consensus recommendations from a consortium of academic and industry experts in the field of lymphoma and imaging for the consistent application of imaging assessment with the Lugano classification. Methods: Consensus was obtained through a series of meetings from July 2019 to October 2021 sponsored by the PINTaD (Pharma Imaging Network for Therapeutics and Diagnostics) as part of the ProLoG (PINTaD RespOnse criteria in Lymphoma wOrking Group) consensus initiative. Results: Consensus recommendations encompass all technical imaging aspects of the Lugano classification. Some technical considerations for PET/CT and diagnostic CT are clarified with regards to required imaging series and scan visits, as well as acquisition and reconstruction of PET images and influence of lesion size and background activity. Recommendations are given on the role of imaging and clinical reviewers as well as on training and monitoring. Finally, an example template of an imaging case report form is provided to support efficient collection of data with Lugano Classification. Conclusion: Consensus recommendations are made to comprehensively address technical and imaging areas of inconsistency and ambiguity in the classification encountered by end users. Such guidance should be used to support standardized acquisition and evaluation with the Lugano 2014.

Is Artificial Intelligence Replacing Our Radiology Stars? Not Yet!

Artificial intelligence (AI) is here to stay and will change health care as we know it. The availability of big data and the increasing numbers of AI algorithms approved by the US Food and Drug Administration together will help in improving the quality of care for patients and in overcoming human fatigue barriers. In oncology practice, patients and providers rely on the interpretation of radiologists when making clinical decisions; however, there is considerable variability among readers, and in particular for prostate imaging. AI represents an emerging solution to this problem, for which it can provide a much-needed form of standardization. The diagnostic performance of AI alone in comparison to a combination of an AI framework and radiologist assessment for evaluation of prostate imaging has yet to be explored. Here, we compare the performance of radiologists alone versus a combination of radiologists aided by a modern computer-aided diagnosis (CAD) AI system. We show that the radiologist-CAD combination demonstrates superior sensitivity and specificity in comparison to both radiologists alone and AI alone. Our findings demonstrate that a radiologist + AI combination could perform best for detection of prostate cancer lesions. A hybrid technology-human system could leverage the benefits of AI in improving radiologist performance while also reducing physician workload, minimizing burnout, and enhancing the quality of patient care.

Patient summary

Our report demonstrates the potential of artificial intelligence (AI) for improving the interpretation of prostate scans. A combination of AI and evaluation by a radiologist has the best performance in determining the severity of prostate cancer. A hybrid system that uses both AI and radiologists could maximize the quality of care for patients while reducing physician workload and burnout.

Pituitary incidentaloma

Pituitary incidentalomas (PI) are lesions of the pituitary region discovered fortuitously by imaging for reasons unrelated to pituitary disease. They range from small cysts to large invasive adenomas. All over the world, improvements in the quality and availability of radiological examinations are leading to an increase in the discovery of PI. In the last four decades, significant advances have been made in the understanding of PI. Autopsy studies have shown that about 10% of deceased individuals harbour a PI, most often a non-functioning microadenoma. In contrast, modern patient series showed that among PIs that come to endocrinological attention, a significant proportion are macroadenomas, and many patients suffer from asymptomatic or pauci-symptomatic endocrine or ophthalmologic disturbances. Other than adenomas, empty sella, Rathke's cleft cyst, craniopharyngioma and meningioma are the most frequent types of PIs. About 10% of micro-incidentalomas and 25% of macro-incidentalomas grow over time. Most cases can be managed conservatively by mere surveillance. Follow-up is necessary in all patients with macroadenoma, but uncertainty remains for microadenomas as to the extent of endocrinological work-up as well as the necessity and duration of follow-up. Visual and endocrine anomalies constitute the most common indications for surgery. When needed, surgery yields better outcome in PIs than in symptomatic pituitary lesions.

Radiologists and Clinical Trials: Part 2: Practical Statistical Methods for Understanding and Monitoring Independent Reader Performance

Though many clinical trials rely on medical image evaluations for primary or key secondary endpoints, the methods to monitor reader performance are all too often mired in the legacy use of adjudication rates. If misused, this simple metric can be misleading and sometimes entirely contradictory. Furthermore, attempts to overcome the limitations of adjudication rates using de novo or ad hoc methods often ignore well-established research conducted over the last half-century and can lead to inaccurate conclusions or variable interpretations. Underperforming readers can be missed, expert readers retrained, or worse, replaced. This paper aims to standardize reader performance evaluations using proven statistical methods. Additionally, these methods will describe how to discriminate between scenarios of concern and normal medical interpretation variability. Statistical methods are provided for inter-reader and intra-reader variability and bias, including the adjudicator's bias. Finally, we have compiled guidelines for calculating correct sample sizes, considerations for intra-reader memory recall, and applying alternative designs for independent readers.