Evaluation of inter-observer variability according to RECIST 1.1 and its influence on response classification in CT measurement of liver metastases

RECIST 1.1 assessments variability: a systematic pictorial review of blinded double reads

Reader variability is intrinsic to radiologic oncology assessments, necessitating measures to enhance consistency and accuracy. RECIST 1.1 criteria play a crucial role in mitigating this variability by standardizing evaluations, aiming to establish an accepted "truth" confirmed by histology or patient survival. Clinical trials utilize Blind Independent Centralized Review (BICR) techniques to manage variability, employing double reads and adjudicators to address inter-observer discordance effectively. It is essential to dissect the root causes of variability in response assessments, with a specific focus on the factors influencing RECIST evaluations. We propose proactive measures for radiologists to address variability sources such as radiologist expertise, image quality, and accessibility of contextual information, which significantly impact interpretation and assessment precision. Adherence to standardization and RECIST guidelines is pivotal in diminishing variability and ensuring uniform results across studies. Variability factors, including lesion selection, new lesion appearance, and confirmation bias, can have profound implications on assessment accuracy and interpretation, underscoring the importance of identifying and addressing these factors. Delving into the causes of variability aids in enhancing the accuracy and consistency of response assessments in oncology, underscoring the role of standardized evaluation protocols and mitigating risk factors that contribute to variability. Access to contextual information is crucial. CRITICAL RELEVANCE STATEMENT: By understanding the causes of diagnosis variability, we can enhance the accuracy and consistency of response assessments in oncology, ultimately improving patient care and clinical outcomes. KEY POINTS: Baseline lesion selection and detection of new lesions play a major role in the occurrence of discordance. Image interpretation is influenced by contextual information, the lack of which can lead to diagnostic uncertainty. Radiologists must be trained in RECIST criteria to reduce errors and variability.

Improving the Reliability of Peer Review Without a Gold Standard

Peer review plays a crucial role in accreditation and credentialing processes as it can identify outliers and foster a peer learning approach, facilitating error analysis and knowledge sharing. However, traditional peer review methods may fall short in effectively addressing the interpretive variability among reviewing and primary reading radiologists, hindering scalability and effectiveness. Reducing this variability is key to enhancing the reliability of results and instilling confidence in the review process. In this paper, we propose a novel statistical approach called "Bayesian Inter-Reviewer Agreement Rate" (BIRAR) that integrates radiologist variability. By doing so, BIRAR aims to enhance the accuracy and consistency of peer review assessments, providing physicians involved in quality improvement and peer learning programs with valuable and reliable insights. A computer simulation was designed to assign predefined interpretive error rates to hypothetical interpreting and peer-reviewing radiologists. The Monte Carlo simulation then sampled (100 samples per experiment) the data that would be generated by peer reviews. The performances of BIRAR and four other peer review methods for measuring interpretive error rates were then evaluated, including a method that uses a gold standard diagnosis. Application of the BIRAR method resulted in 93% and 79% higher relative accuracy and 43% and 66% lower relative variability, compared to "Single/Standard" and "Majority Panel" peer review methods, respectively. Accuracy was defined by the median difference of Monte Carlo simulations between measured and pre-defined "actual" interpretive error rates. Variability was defined by the 95% CI around the median difference of Monte Carlo simulations between measured and pre-defined "actual" interpretive error rates. BIRAR is a practical and scalable peer review method that produces more accurate and less variable assessments of interpretive quality by accounting for variability within the group's radiologists, implicitly applying a standard derived from the level of consensus within the group across various types of interpretive findings.

Repeatability of metabolic tumor burden and lesion glycolysis between clinical readers

The Metabolic Tumor Volume (MTV) and Tumor Lesion Glycolysis (TLG) has been shown to be independent prognostic predictors for clinical outcome in Diffuse Large B-cell Lymphoma (DLBCL). However, definitions of these measurements have not been standardized, leading to many sources of variation, operator evaluation continues to be one major source. In this study, we propose a reader reproducibility study to evaluate computation of TMV (& TLG) metrics based on differences in lesion delineation. In the first approach, reader manually corrected regional boundaries after automated detection performed across the lesions in a body scan (Reader M using a manual process, or manual). The other reader used a semi-automated method of lesion identification, without any boundary modification (Reader A using a semi- automated process, or auto). Parameters for active lesion were kept the same, derived from standard uptake values (SUVs) over a 41% threshold. We systematically contrasted MTV & TLG differences between expert readers (Reader M & A). We find that MTVs computed by Readers M and A were both concordant between them (concordant correlation coefficient of 0.96) and independently prognostic with a P-value of 0.0001 and 0.0002 respectively for overall survival after treatment. Additionally, we find TLG for these reader approaches showed concordance (CCC of 0.96) and was prognostic for over -all survival (p ≤ 0.0001 for both). In conclusion, the semi-automated approach (Reader A) provides acceptable quantification & prognosis of tumor burden (MTV) and TLG in comparison to expert reader assisted measurement (Reader M) on PET/CT scans.

Twenty Years On: RECIST as a Biomarker of Response in Solid Tumours an EORTC Imaging Group - ESOI Joint Paper

Response evaluation criteria in solid tumours (RECIST) v1.1 are currently the reference standard for evaluating efficacy of therapies in patients with solid tumours who are included in clinical trials, and they are widely used and accepted by regulatory agencies. This expert statement discusses the principles underlying RECIST, as well as their reproducibility and limitations. While the RECIST framework may not be perfect, the scientific bases for the anticancer drugs that have been approved using a RECIST-based surrogate endpoint remain valid. Importantly, changes in measurement have to meet thresholds defined by RECIST for response classification within thus partly circumventing the problems of measurement variability. The RECIST framework also applies to clinical patients in individual settings even though the relationship between tumour size changes and outcome from cohort studies is not necessarily translatable to individual cases. As reproducibility of RECIST measurements is impacted by reader experience, choice of target lesions and detection/interpretation of new lesions, it can result in patients changing response categories when measurements are near threshold values or if new lesions are missed or incorrectly interpreted. There are several situations where RECIST will fail to evaluate treatment-induced changes correctly; knowledge and understanding of these is crucial for correct interpretation. Also, some patterns of response/progression cannot be correctly documented by RECIST, particularly in relation to organ-site (e.g. bone without associated soft-tissue lesion) and treatment type (e.g. focal therapies). These require specialist reader experience and communication with oncologists to determine the actual impact of the therapy and best evaluation strategy. In such situations, alternative imaging markers for tumour response may be used but the sources of variability of individual imaging techniques need to be known and accounted for. Communication between imaging experts and oncologists regarding the level of confidence in a biomarker is essential for the correct interpretation of a biomarker and its application to clinical decision-making. Though measurement automation is desirable and potentially reduces the variability of results, associated technical difficulties must be overcome, and human adjudications may be required.

Optimization of the tumour response threshold in advanced gastroenteropancreatic neuroendocrine carcinomas treated with cisplatin/etoposide combined chemotherapy

OBJECTIVES

To identify early and more accurate imaging response criteria for computed tomography evaluation to define 'responders' in advanced gastroenteropancreatic neuroendocrine carcinoma (GEP-NEC) patients treated with cisplatin/etoposide combined chemotherapy.

MATERIALS AND METHODS

Thirty-seven patients with GEP-NEC treated with first-line cisplatin/etoposide (E/P) combined chemotherapy were enrolled in this study. Computed tomography scans of the chest, abdomen, and pelvis were performed at baseline, during the treatment course, and during follow-up. Tumour size was measured, and tumour response was evaluated by Response Evaluation Criteria in Solid Tumours (RECIST) 1.1. Receiver operating characteristic (ROC) analysis was carried out among the patients who progressed during follow-up. Thresholds from -55% to + 5% were tested by Kaplan-Meier analysis to define "responders" for significantly improved progression-free survival (PFS). The overall survival rate was compared between these two groups.

RESULTS

A reduction of 45% (vs. baseline) achieved the highest sensitivity (70%) and specificity (90%) by ROC analysis. This threshold divided patients into 15 responders and 22 nonresponders. Patients who were grouped as responders by the -45% threshold had a significantly longer PFS (11.06 months) than nonresponders (7.97 months, hazard ratio, 3.636; 95% confidence interval, 1.293-10.164). No significant difference was shown in overall survival between these two groups (29.1 vs. 21.4 months, P = 0.190).

CONCLUSION

A 45% reduction in target lesions may be considered to be a more reliable predictor than the RECIST 1.1 criteria in evaluating the outcome of GEP-NEC patients treated with E/P chemotherapy.

Intra- and inter-reader agreement of iRECIST and RECIST 1.1 criteria for the assessment of tumor response in patients receiving checkpoint inhibitor immunotherapy for lung cancer

OBJECTIVES

To investigate the inter- and intra-reader agreement of immune Response Evaluation Criteria in Solid Tumors (iRECIST) and Response Evaluation Criteria in Solid Tumors version 1.1 (RECIST 1.1) in patients with lung cancer treated with immunotherapy.

MATERIALS AND METHODS

This retrospective study included 85 patients with lung cancer treated with PD-1 blockade. Four radiologists evaluated computed topography (CT) scans before and after initiation of immunotherapy using iRECIST and RECIST 1.1. Weighted kappa (k) with equal weights was used to assess the intra-reader agreement between 2 repeated reads on overall response at all time points, best overall response, and the response at the time point of progression, as well as the intra-reader agreement between iRECIST and RECIST. The inter-reader agreement was calculated using Light's kappa.

RESULTS

Intra-reader agreement for overall response at all time points, best overall response, and time point of progression was substantial to almost perfect for both iRECIST and RECIST 1.1 (k = 0.651-0.983). Inter-reader agreement was substantial for iRECIST (κ = 0.657-0.742) while RECIST 1.1 was moderate to substantial (κ = 0.587-0.686). The level of inter-reader agreement was not higher on repeat read for iRECIST (κ = 0.677-0.709 and κ = 0.657-0.742 for first and second read, respectively) as well as for RECIST 1.1 (κ = 0.587-0.659 and κ = 0.633-0.686 for first and second read, respectively). Almost perfect agreement was observed between RECIST 1.1 and iRECIST at first (κ = 0.813-0.923) and second read (κ = 0.841-0.912).

CONCLUSION

The inter- and intra-reader agreement of iRECIST is high and similar to RECIST 1.1 in patients with lung cancer treated with immunotherapy.

The Value of CT 3D Reconstruction in the Classification and Nursing Effect Evaluation of Ankle Fracture

Aim

To study the application value of ankle fracture classification and diagnosis. In this paper, the clinical data of 100 cases of ankle fracture patients admitted from May 2020 to May 2021 were analyzed by CT 3D reconstruction. All patients received surgical treatment and underwent spiral CT 3D reconstruction and X-ray examination before surgery. The results showed that 20 cases (20.00%) of the 100 cases were PER, 24 cases (24%) of the 100 cases were PAB, 31 cases (31%) of the 100 cases were SER, and 25 cases (25%) of the 100 cases were SAB, respectively.

Conclusion

The diagnostic accuracy of CT 3D reconstruction for different types of ankle fracture is higher than that of X-ray, and the differences are statistically significant (P < 0.05). CT 3D reconstruction is applied in the early diagnosis of ankle fracture, which can accurately detect the classification of patients. It has important clinical application value and can be used as the first choice for the early classification diagnosis of ankle fracture.

Unlocking the Power of Artificial Intelligence and Big Data in Medicine

Data-driven science and its corollaries in machine learning and the wider field of artificial intelligence have the potential to drive important changes in medicine. However, medicine is not a science like any other: It is deeply and tightly bound with a large and wide network of legal, ethical, regulatory, economical, and societal dependencies. As a consequence, the scientific and technological progresses in handling information and its further processing and cross-linking for decision support and predictive systems must be accompanied by parallel changes in the global environment, with numerous stakeholders, including citizen and society. What can be seen at the first glance as a barrier and a mechanism slowing down the progression of data science must, however, be considered an important asset. Only global adoption can transform the potential of big data and artificial intelligence into an effective breakthroughs in handling health and medicine. This requires science and society, scientists and citizens, to progress together.