1
|
Iannessi A, Beaumont H, Ojango C, Bertrand AS, Liu Y. RECIST 1.1 assessments variability: a systematic pictorial review of blinded double reads. Insights Imaging 2024; 15:199. [PMID: 39112819 PMCID: PMC11306910 DOI: 10.1186/s13244-024-01774-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/28/2024] [Accepted: 07/07/2024] [Indexed: 08/10/2024] Open
Abstract
Reader variability is intrinsic to radiologic oncology assessments, necessitating measures to enhance consistency and accuracy. RECIST 1.1 criteria play a crucial role in mitigating this variability by standardizing evaluations, aiming to establish an accepted "truth" confirmed by histology or patient survival. Clinical trials utilize Blind Independent Centralized Review (BICR) techniques to manage variability, employing double reads and adjudicators to address inter-observer discordance effectively. It is essential to dissect the root causes of variability in response assessments, with a specific focus on the factors influencing RECIST evaluations. We propose proactive measures for radiologists to address variability sources such as radiologist expertise, image quality, and accessibility of contextual information, which significantly impact interpretation and assessment precision. Adherence to standardization and RECIST guidelines is pivotal in diminishing variability and ensuring uniform results across studies. Variability factors, including lesion selection, new lesion appearance, and confirmation bias, can have profound implications on assessment accuracy and interpretation, underscoring the importance of identifying and addressing these factors. Delving into the causes of variability aids in enhancing the accuracy and consistency of response assessments in oncology, underscoring the role of standardized evaluation protocols and mitigating risk factors that contribute to variability. Access to contextual information is crucial. CRITICAL RELEVANCE STATEMENT: By understanding the causes of diagnosis variability, we can enhance the accuracy and consistency of response assessments in oncology, ultimately improving patient care and clinical outcomes. KEY POINTS: Baseline lesion selection and detection of new lesions play a major role in the occurrence of discordance. Image interpretation is influenced by contextual information, the lack of which can lead to diagnostic uncertainty. Radiologists must be trained in RECIST criteria to reduce errors and variability.
Collapse
Affiliation(s)
- Antoine Iannessi
- Cancer Center Antoine Lacassagne 33 Av. de Valombrose, 06100, Nice, France
- Median Technologies SA 1800 Route des Crêtes, 06560, Valbonne, France
| | - Hubert Beaumont
- Median Technologies SA 1800 Route des Crêtes, 06560, Valbonne, France.
| | - Christine Ojango
- Median Technologies SA 1800 Route des Crêtes, 06560, Valbonne, France
| | - Anne-Sophie Bertrand
- Imaging Center Beaulieu-sur-mer 18 Bd Eugène Gauthier, 06310, Beaulieu-sur-Mer, France
| | - Yan Liu
- Median Technologies SA 1800 Route des Crêtes, 06560, Valbonne, France
| |
Collapse
|
2
|
Äijö T, Elgort D, Becker M, Herzog R, Brown RKJ, Odry BL, Vianu R. Improving the Reliability of Peer Review Without a Gold Standard. JOURNAL OF IMAGING INFORMATICS IN MEDICINE 2024; 37:489-503. [PMID: 38316666 PMCID: PMC11031531 DOI: 10.1007/s10278-024-00971-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/04/2023] [Revised: 10/29/2023] [Accepted: 11/27/2023] [Indexed: 02/07/2024]
Abstract
Peer review plays a crucial role in accreditation and credentialing processes as it can identify outliers and foster a peer learning approach, facilitating error analysis and knowledge sharing. However, traditional peer review methods may fall short in effectively addressing the interpretive variability among reviewing and primary reading radiologists, hindering scalability and effectiveness. Reducing this variability is key to enhancing the reliability of results and instilling confidence in the review process. In this paper, we propose a novel statistical approach called "Bayesian Inter-Reviewer Agreement Rate" (BIRAR) that integrates radiologist variability. By doing so, BIRAR aims to enhance the accuracy and consistency of peer review assessments, providing physicians involved in quality improvement and peer learning programs with valuable and reliable insights. A computer simulation was designed to assign predefined interpretive error rates to hypothetical interpreting and peer-reviewing radiologists. The Monte Carlo simulation then sampled (100 samples per experiment) the data that would be generated by peer reviews. The performances of BIRAR and four other peer review methods for measuring interpretive error rates were then evaluated, including a method that uses a gold standard diagnosis. Application of the BIRAR method resulted in 93% and 79% higher relative accuracy and 43% and 66% lower relative variability, compared to "Single/Standard" and "Majority Panel" peer review methods, respectively. Accuracy was defined by the median difference of Monte Carlo simulations between measured and pre-defined "actual" interpretive error rates. Variability was defined by the 95% CI around the median difference of Monte Carlo simulations between measured and pre-defined "actual" interpretive error rates. BIRAR is a practical and scalable peer review method that produces more accurate and less variable assessments of interpretive quality by accounting for variability within the group's radiologists, implicitly applying a standard derived from the level of consensus within the group across various types of interpretive findings.
Collapse
Affiliation(s)
| | - Daniel Elgort
- Covera Health, New York, NY, USA
- Present Address: Aster Insights, Tampa, FL, USA
| | - Murray Becker
- Covera Health, New York, NY, USA
- Rutgers Robert Wood Johnson Medical School, New Brunswick, NJ, USA
| | | | - Richard K J Brown
- Department of Radiology, University of Michigan (Michigan Medicine), Ann Arbor, MI, USA
| | | | | |
Collapse
|
3
|
Choi JW, Dean EA, Lu H, Thompson Z, Qi J, Krivenko G, Jain MD, Locke FL, Balagurunathan Y. Repeatability of metabolic tumor burden and lesion glycolysis between clinical readers. Front Immunol 2023; 14:994520. [PMID: 36875072 PMCID: PMC9975754 DOI: 10.3389/fimmu.2023.994520] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/14/2022] [Accepted: 01/10/2023] [Indexed: 02/17/2023] Open
Abstract
The Metabolic Tumor Volume (MTV) and Tumor Lesion Glycolysis (TLG) has been shown to be independent prognostic predictors for clinical outcome in Diffuse Large B-cell Lymphoma (DLBCL). However, definitions of these measurements have not been standardized, leading to many sources of variation, operator evaluation continues to be one major source. In this study, we propose a reader reproducibility study to evaluate computation of TMV (& TLG) metrics based on differences in lesion delineation. In the first approach, reader manually corrected regional boundaries after automated detection performed across the lesions in a body scan (Reader M using a manual process, or manual). The other reader used a semi-automated method of lesion identification, without any boundary modification (Reader A using a semi- automated process, or auto). Parameters for active lesion were kept the same, derived from standard uptake values (SUVs) over a 41% threshold. We systematically contrasted MTV & TLG differences between expert readers (Reader M & A). We find that MTVs computed by Readers M and A were both concordant between them (concordant correlation coefficient of 0.96) and independently prognostic with a P-value of 0.0001 and 0.0002 respectively for overall survival after treatment. Additionally, we find TLG for these reader approaches showed concordance (CCC of 0.96) and was prognostic for over -all survival (p ≤ 0.0001 for both). In conclusion, the semi-automated approach (Reader A) provides acceptable quantification & prognosis of tumor burden (MTV) and TLG in comparison to expert reader assisted measurement (Reader M) on PET/CT scans.
Collapse
Affiliation(s)
- Jung W Choi
- Department of Diagnostic Imaging and Interventional Radiology, H Lee Moffitt Cancer Center, Tampa, FL, United States
| | - Erin A Dean
- Blood and Marrow Transplant and Cellular Immunotherapy, H. Lee. Moffitt Cancer Center, Tampa, FL, United States.,Division of Hematology and Oncology, University of Florida, Gainesville, FL, , United States
| | - Hong Lu
- Cancer Physiology, H. Lee. Moffitt Cancer Center, Tampa, FL, United States.,Tianjin Medical University Cancer Institute and Hospital, Tianjin, China
| | - Zachary Thompson
- Biostatistics & Bioinformatics, H. Lee. Moffitt Cancer Center, Tampa, FL, United States
| | - Jin Qi
- Cancer Physiology, H. Lee. Moffitt Cancer Center, Tampa, FL, United States
| | - Gabe Krivenko
- Blood and Marrow Transplant and Cellular Immunotherapy, H. Lee. Moffitt Cancer Center, Tampa, FL, United States
| | - Michael D Jain
- Blood and Marrow Transplant and Cellular Immunotherapy, H. Lee. Moffitt Cancer Center, Tampa, FL, United States
| | - Frederick L Locke
- Blood and Marrow Transplant and Cellular Immunotherapy, H. Lee. Moffitt Cancer Center, Tampa, FL, United States
| | | |
Collapse
|
4
|
Fournier L, de Geus-Oei LF, Regge D, Oprea-Lager DE, D’Anastasi M, Bidaut L, Bäuerle T, Lopci E, Cappello G, Lecouvet F, Mayerhoefer M, Kunz WG, Verhoeff JJC, Caruso D, Smits M, Hoffmann RT, Gourtsoyianni S, Beets-Tan R, Neri E, deSouza NM, Deroose CM, Caramella C. Twenty Years On: RECIST as a Biomarker of Response in Solid Tumours an EORTC Imaging Group - ESOI Joint Paper. Front Oncol 2022; 11:800547. [PMID: 35083155 PMCID: PMC8784734 DOI: 10.3389/fonc.2021.800547] [Citation(s) in RCA: 8] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/23/2021] [Accepted: 11/30/2021] [Indexed: 12/15/2022] Open
Abstract
Response evaluation criteria in solid tumours (RECIST) v1.1 are currently the reference standard for evaluating efficacy of therapies in patients with solid tumours who are included in clinical trials, and they are widely used and accepted by regulatory agencies. This expert statement discusses the principles underlying RECIST, as well as their reproducibility and limitations. While the RECIST framework may not be perfect, the scientific bases for the anticancer drugs that have been approved using a RECIST-based surrogate endpoint remain valid. Importantly, changes in measurement have to meet thresholds defined by RECIST for response classification within thus partly circumventing the problems of measurement variability. The RECIST framework also applies to clinical patients in individual settings even though the relationship between tumour size changes and outcome from cohort studies is not necessarily translatable to individual cases. As reproducibility of RECIST measurements is impacted by reader experience, choice of target lesions and detection/interpretation of new lesions, it can result in patients changing response categories when measurements are near threshold values or if new lesions are missed or incorrectly interpreted. There are several situations where RECIST will fail to evaluate treatment-induced changes correctly; knowledge and understanding of these is crucial for correct interpretation. Also, some patterns of response/progression cannot be correctly documented by RECIST, particularly in relation to organ-site (e.g. bone without associated soft-tissue lesion) and treatment type (e.g. focal therapies). These require specialist reader experience and communication with oncologists to determine the actual impact of the therapy and best evaluation strategy. In such situations, alternative imaging markers for tumour response may be used but the sources of variability of individual imaging techniques need to be known and accounted for. Communication between imaging experts and oncologists regarding the level of confidence in a biomarker is essential for the correct interpretation of a biomarker and its application to clinical decision-making. Though measurement automation is desirable and potentially reduces the variability of results, associated technical difficulties must be overcome, and human adjudications may be required.
Collapse
Affiliation(s)
- Laure Fournier
- Imaging Group, European Organisation of Research and Treatment in Cancer (EORTC), Brussels, Belgium
- European Society of Oncologic Imaging (ESOI), European Society of Radiology, Vienna, Austria
- Université de Paris, Assistance Publique–Hôpitaux de Paris (AP-HP), Hopital europeen Georges Pompidou, Department of Radiology, Paris Cardiovascular Research Center (PARCC) Unité Mixte de Recherche (UMRS) 970, Institut national de la santé et de la recherche médicale (INSERM), Paris, France
| | - Lioe-Fee de Geus-Oei
- Imaging Group, European Organisation of Research and Treatment in Cancer (EORTC), Brussels, Belgium
- Department of Radiology, Leiden University Medical Center, Leiden, Netherlands
- Biomedical Photonic Imaging Group, University of Twente, Enschede, Netherlands
| | - Daniele Regge
- European Society of Oncologic Imaging (ESOI), European Society of Radiology, Vienna, Austria
- Department of Surgical Sciences, University of Turin, Turin, Italy
- Radiology Unit, Candiolo Cancer Institute, Fondazione del Piemonte per l’Oncologia-Istituto Di Ricovero e Cura a Carattere Scientifico (FPO-IRCCS), Turin, Italy
| | - Daniela-Elena Oprea-Lager
- Imaging Group, European Organisation of Research and Treatment in Cancer (EORTC), Brussels, Belgium
- Department of Radiology & Nuclear Medicine, Cancer Centre Amsterdam, Amsterdam University Medical Centers [Vrije Universiteit (VU) University], Amsterdam, Netherlands
| | - Melvin D’Anastasi
- European Society of Oncologic Imaging (ESOI), European Society of Radiology, Vienna, Austria
- Medical Imaging Department, Mater Dei Hospital, University of Malta, Msida, Malta
| | - Luc Bidaut
- Imaging Group, European Organisation of Research and Treatment in Cancer (EORTC), Brussels, Belgium
- College of Science, University of Lincoln, Lincoln, United Kingdom
| | - Tobias Bäuerle
- European Society of Oncologic Imaging (ESOI), European Society of Radiology, Vienna, Austria
- Institute of Radiology, University Hospital Erlangen, Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU), Erlangen, Germany
| | - Egesta Lopci
- Imaging Group, European Organisation of Research and Treatment in Cancer (EORTC), Brussels, Belgium
- Nuclear Medicine Unit, Istituto Di Ricovero e Cura a Carattere Scientifico (IRCCS) – Humanitas Research Hospital, Milan, Italy
| | - Giovanni Cappello
- Department of Surgical Sciences, University of Turin, Turin, Italy
- Radiology Unit, Candiolo Cancer Institute, Fondazione del Piemonte per l’Oncologia-Istituto Di Ricovero e Cura a Carattere Scientifico (FPO-IRCCS), Turin, Italy
| | - Frederic Lecouvet
- Imaging Group, European Organisation of Research and Treatment in Cancer (EORTC), Brussels, Belgium
- Department of Radiology, Institut de Recherche Expérimentale et Clinique (IREC), Cliniques Universitaires Saint Luc, Université Catholique de Louvain (UCLouvain), Brussels, Belgium
| | - Marius Mayerhoefer
- European Society of Oncologic Imaging (ESOI), European Society of Radiology, Vienna, Austria
- Department of Radiology, Memorial Sloan Kettering Cancer Center, New York, NY, United States
- Department of Biomedical Imaging and Image-guided Therapy, Medical University of Vienna, Vienna, Austria
| | - Wolfgang G. Kunz
- Imaging Group, European Organisation of Research and Treatment in Cancer (EORTC), Brussels, Belgium
- European Society of Oncologic Imaging (ESOI), European Society of Radiology, Vienna, Austria
- Department of Radiology, University Hospital, Ludwig Maximilian University (LMU) Munich, Munich, Germany
| | - Joost J. C. Verhoeff
- Imaging Group, European Organisation of Research and Treatment in Cancer (EORTC), Brussels, Belgium
- Department of Radiotherapy, University Medical Center Utrecht, Utrecht University, Utrecht, Netherlands
| | - Damiano Caruso
- European Society of Oncologic Imaging (ESOI), European Society of Radiology, Vienna, Austria
- Department of Medical-Surgical Sciences and Translational Medicine, Sapienza University of Rome, Rome, Italy
| | - Marion Smits
- Imaging Group, European Organisation of Research and Treatment in Cancer (EORTC), Brussels, Belgium
- Department of Radiology & Nuclear Medicine, Erasmus MC, University Medical Centre Rotterdam, Rotterdam, Netherlands
- Brain Tumour Centre, Erasmus Medical Centre (MC) Cancer Institute, Rotterdam, Netherlands
| | - Ralf-Thorsten Hoffmann
- European Society of Oncologic Imaging (ESOI), European Society of Radiology, Vienna, Austria
- Institute and Policlinic for Diagnostic and Interventional Radiology, University Hospital, Carl-Gustav-Carus Technical University Dresden, Dresden, Germany
| | - Sofia Gourtsoyianni
- European Society of Oncologic Imaging (ESOI), European Society of Radiology, Vienna, Austria
- Department of Radiology, School of Medicine, National and Kapodistrian University of Athens, Areteion Hospital, Athens, Greece
| | - Regina Beets-Tan
- European Society of Oncologic Imaging (ESOI), European Society of Radiology, Vienna, Austria
- Department of Radiology, The Netherlands Cancer Institute, Amsterdam, Netherlands
- School For Oncology and Developmental Biology (GROW) School for Oncology and Developmental Biology, Maastricht University, Maastricht, Netherlands
| | - Emanuele Neri
- European Society of Oncologic Imaging (ESOI), European Society of Radiology, Vienna, Austria
- Diagnostic and Interventional Radiology, Department of Translational Research and of New Surgical and Medical Technologies, University of Pisa, Pisa, Italy
| | - Nandita M. deSouza
- Imaging Group, European Organisation of Research and Treatment in Cancer (EORTC), Brussels, Belgium
- Division of Radiotherapy and Imaging, The Institute of Cancer Research and Royal Marsden National Health Service (NHS) Foundation Trust, London, United Kingdom
- European Imaging Biomarkers Alliance (EIBALL), European Society of Radiology, Vienna, Austria
- Quantitative Imaging Biomarkers Alliance, Radiological Society of North America, Oak Brook, IL, United States
| | - Christophe M. Deroose
- Imaging Group, European Organisation of Research and Treatment in Cancer (EORTC), Brussels, Belgium
- Nuclear Medicine, University Hospitals Leuven, Leuven, Belgium
- Nuclear Medicine & Molecular Imaging, Department of Imaging and Pathology, Katholieke Universiteit (KU) Leuven, Leuven, Belgium
| | - Caroline Caramella
- Imaging Group, European Organisation of Research and Treatment in Cancer (EORTC), Brussels, Belgium
- Radiology Department, Hôpital Marie Lannelongue, Groupe Hospitalier Paris Saint Joseph Centre International des Cancers Thoraciques, Université Paris-Saclay, Le Plessis-Robinson, France
| |
Collapse
|
6
|
Huicochea Castellanos S, Pagano A, Plodkowski AJ, Girshman J, Hellmann MD, Rizvi H, Flynn J, Zheng J, Capanu M, Halpenny DF, Ginsberg MS. Intra- and inter-reader agreement of iRECIST and RECIST 1.1 criteria for the assessment of tumor response in patients receiving checkpoint inhibitor immunotherapy for lung cancer. Lung Cancer 2021; 161:60-67. [PMID: 34536733 DOI: 10.1016/j.lungcan.2021.08.020] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/28/2021] [Revised: 08/17/2021] [Accepted: 08/31/2021] [Indexed: 11/29/2022]
Abstract
OBJECTIVES To investigate the inter- and intra-reader agreement of immune Response Evaluation Criteria in Solid Tumors (iRECIST) and Response Evaluation Criteria in Solid Tumors version 1.1 (RECIST 1.1) in patients with lung cancer treated with immunotherapy. MATERIALS AND METHODS This retrospective study included 85 patients with lung cancer treated with PD-1 blockade. Four radiologists evaluated computed topography (CT) scans before and after initiation of immunotherapy using iRECIST and RECIST 1.1. Weighted kappa (k) with equal weights was used to assess the intra-reader agreement between 2 repeated reads on overall response at all time points, best overall response, and the response at the time point of progression, as well as the intra-reader agreement between iRECIST and RECIST. The inter-reader agreement was calculated using Light's kappa. RESULTS Intra-reader agreement for overall response at all time points, best overall response, and time point of progression was substantial to almost perfect for both iRECIST and RECIST 1.1 (k = 0.651-0.983). Inter-reader agreement was substantial for iRECIST (κ = 0.657-0.742) while RECIST 1.1 was moderate to substantial (κ = 0.587-0.686). The level of inter-reader agreement was not higher on repeat read for iRECIST (κ = 0.677-0.709 and κ = 0.657-0.742 for first and second read, respectively) as well as for RECIST 1.1 (κ = 0.587-0.659 and κ = 0.633-0.686 for first and second read, respectively). Almost perfect agreement was observed between RECIST 1.1 and iRECIST at first (κ = 0.813-0.923) and second read (κ = 0.841-0.912). CONCLUSION The inter- and intra-reader agreement of iRECIST is high and similar to RECIST 1.1 in patients with lung cancer treated with immunotherapy.
Collapse
Affiliation(s)
- Sandra Huicochea Castellanos
- Department of Radiology, Weill Cornell Medicine-New York Presbyterian Hospital, New York, NY 10065, United States
| | - Andrew Pagano
- Department of Radiology, Memorial Sloan Kettering Cancer Center, New York, NY 10065, United States.
| | - Andrew J Plodkowski
- Department of Radiology, Memorial Sloan Kettering Cancer Center, New York, NY 10065, United States
| | - Jeffrey Girshman
- Department of Radiology, Memorial Sloan Kettering Cancer Center, New York, NY 10065, United States
| | - Matthew D Hellmann
- Department of Medicine, Memorial Sloan Kettering Cancer Center, New York, NY 10065, United States
| | - Hira Rizvi
- Department of Medicine, Memorial Sloan Kettering Cancer Center, New York, NY 10065, United States
| | - Jessica Flynn
- Department of Epidemiology and Biostatistics, Memorial Sloan Kettering Cancer Center, New York, NY 10065, United States
| | - Junting Zheng
- Department of Epidemiology and Biostatistics, Memorial Sloan Kettering Cancer Center, New York, NY 10065, United States
| | - Marinela Capanu
- Department of Epidemiology and Biostatistics, Memorial Sloan Kettering Cancer Center, New York, NY 10065, United States
| | - Darragh F Halpenny
- Department of Radiology, Tallaght University Hospital, Dublin D24 NR04, Ireland
| | - Michelle S Ginsberg
- Department of Radiology, Memorial Sloan Kettering Cancer Center, New York, NY 10065, United States
| |
Collapse
|