1
|
Iannessi A, Beaumont H, Ojango C, Bertrand AS, Liu Y. RECIST 1.1 assessments variability: a systematic pictorial review of blinded double reads. Insights Imaging 2024; 15:199. [PMID: 39112819 PMCID: PMC11306910 DOI: 10.1186/s13244-024-01774-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/28/2024] [Accepted: 07/07/2024] [Indexed: 08/10/2024] Open
Abstract
Reader variability is intrinsic to radiologic oncology assessments, necessitating measures to enhance consistency and accuracy. RECIST 1.1 criteria play a crucial role in mitigating this variability by standardizing evaluations, aiming to establish an accepted "truth" confirmed by histology or patient survival. Clinical trials utilize Blind Independent Centralized Review (BICR) techniques to manage variability, employing double reads and adjudicators to address inter-observer discordance effectively. It is essential to dissect the root causes of variability in response assessments, with a specific focus on the factors influencing RECIST evaluations. We propose proactive measures for radiologists to address variability sources such as radiologist expertise, image quality, and accessibility of contextual information, which significantly impact interpretation and assessment precision. Adherence to standardization and RECIST guidelines is pivotal in diminishing variability and ensuring uniform results across studies. Variability factors, including lesion selection, new lesion appearance, and confirmation bias, can have profound implications on assessment accuracy and interpretation, underscoring the importance of identifying and addressing these factors. Delving into the causes of variability aids in enhancing the accuracy and consistency of response assessments in oncology, underscoring the role of standardized evaluation protocols and mitigating risk factors that contribute to variability. Access to contextual information is crucial. CRITICAL RELEVANCE STATEMENT: By understanding the causes of diagnosis variability, we can enhance the accuracy and consistency of response assessments in oncology, ultimately improving patient care and clinical outcomes. KEY POINTS: Baseline lesion selection and detection of new lesions play a major role in the occurrence of discordance. Image interpretation is influenced by contextual information, the lack of which can lead to diagnostic uncertainty. Radiologists must be trained in RECIST criteria to reduce errors and variability.
Collapse
Affiliation(s)
- Antoine Iannessi
- Cancer Center Antoine Lacassagne 33 Av. de Valombrose, 06100, Nice, France
- Median Technologies SA 1800 Route des Crêtes, 06560, Valbonne, France
| | - Hubert Beaumont
- Median Technologies SA 1800 Route des Crêtes, 06560, Valbonne, France.
| | - Christine Ojango
- Median Technologies SA 1800 Route des Crêtes, 06560, Valbonne, France
| | - Anne-Sophie Bertrand
- Imaging Center Beaulieu-sur-mer 18 Bd Eugène Gauthier, 06310, Beaulieu-sur-Mer, France
| | - Yan Liu
- Median Technologies SA 1800 Route des Crêtes, 06560, Valbonne, France
| |
Collapse
|
2
|
Bellmann Q, Peng Y, Genske U, Yan L, Wagner M, Jahnke P. Low-contrast lesion detection in neck CT: a multireader study comparing deep learning, iterative, and filtered back projection reconstructions using realistic phantoms. Eur Radiol Exp 2024; 8:84. [PMID: 39046565 PMCID: PMC11269546 DOI: 10.1186/s41747-024-00486-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/20/2024] [Accepted: 06/18/2024] [Indexed: 07/25/2024] Open
Abstract
BACKGROUND Computed tomography (CT) reconstruction algorithms can improve image quality, especially deep learning reconstruction (DLR). We compared DLR, iterative reconstruction (IR), and filtered back projection (FBP) for lesion detection in neck CT. METHODS Nine patient-mimicking neck phantoms were examined with a 320-slice scanner at six doses: 0.5, 1, 1.6, 2.1, 3.1, and 5.2 mGy. Each of eight phantoms contained one circular lesion (diameter 1 cm; contrast -30 HU to the background) in the parapharyngeal space; one phantom had no lesions. Reconstruction was made using FBP, IR, and DLR. Thirteen readers were tasked with identifying and localizing lesions in 32 images with a lesion and 20 without lesions for each dose and reconstruction algorithm. Receiver operating characteristic (ROC) and localization ROC (LROC) analysis were performed. RESULTS DLR improved lesion detection with ROC area under the curve (AUC) 0.724 ± 0.023 (mean ± standard error of the mean) using DLR versus 0.696 ± 0.021 using IR (p = 0.037) and 0.671 ± 0.023 using FBP (p < 0.001). Likewise, DLR improved lesion localization, with LROC AUC 0.407 ± 0.039 versus 0.338 ± 0.041 using IR (p = 0.002) and 0.313 ± 0.044 using FBP (p < 0.001). Dose reduction to 0.5 mGy compromised lesion detection in FBP-reconstructed images compared to doses ≥ 2.1 mGy (p ≤ 0.024), while no effect was observed with DLR or IR (p ≥ 0.058). CONCLUSION DLR improved the detectability of lesions in neck CT imaging. Dose reduction to 0.5 mGy maintained lesion detectability when denoising reconstruction was used. RELEVANCE STATEMENT Deep learning enhances lesion detection in neck CT imaging compared to iterative reconstruction and filtered back projection, offering improved diagnostic performance and potential for x-ray dose reduction. KEY POINTS Low-contrast lesion detectability was assessed in anatomically realistic neck CT phantoms. Deep learning reconstruction (DLR) outperformed filtered back projection and iterative reconstruction. Dose has little impact on lesion detectability against anatomical background structures.
Collapse
Affiliation(s)
- Quirin Bellmann
- Department of Radiology, Charité-Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin, Humboldt-Universität zu Berlin, and Berlin Institute of Health, Charitéplatz 1, 10117, Berlin, Germany
| | - Yang Peng
- Department of Radiology, Charité-Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin, Humboldt-Universität zu Berlin, and Berlin Institute of Health, Charitéplatz 1, 10117, Berlin, Germany
- Department of Radiology, Tongji Hospital, Tongji Medical College, Huazhong University of Science and Technology, 1095 Jiefang Avenue, Wuhan, 430030, Hubei Province, China
| | - Ulrich Genske
- Department of Radiology, Charité-Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin, Humboldt-Universität zu Berlin, and Berlin Institute of Health, Charitéplatz 1, 10117, Berlin, Germany
| | - Li Yan
- Department of Radiology, Charité-Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin, Humboldt-Universität zu Berlin, and Berlin Institute of Health, Charitéplatz 1, 10117, Berlin, Germany
| | - Moritz Wagner
- Department of Radiology, Charité-Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin, Humboldt-Universität zu Berlin, and Berlin Institute of Health, Charitéplatz 1, 10117, Berlin, Germany
| | - Paul Jahnke
- Department of Radiology, Charité-Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin, Humboldt-Universität zu Berlin, and Berlin Institute of Health, Charitéplatz 1, 10117, Berlin, Germany.
- Berlin Institute of Health (BIH), Anna-Louisa-Karsch-Str. 2, 10178, Berlin, Germany.
| |
Collapse
|
3
|
Yeghaian M, Tareco Bucho TM, de Bruin M, Schmitz A, Bodalal Z, Smit EF, Beets-Tan RGH, van den Broek D, Trebeschi S. Can blood-based markers predict RECIST progression in non-small cell lung cancer treated with immunotherapy? J Cancer Res Clin Oncol 2024; 150:329. [PMID: 38922374 PMCID: PMC11208229 DOI: 10.1007/s00432-024-05814-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/01/2023] [Accepted: 05/21/2024] [Indexed: 06/27/2024]
Abstract
PURPOSE In this study, we aimed to evaluate the potential of routine blood markers, serum tumour markers and their combination in predicting RECIST-defined progression in patients with stage IV non-small cell lung cancer (NSCLC) undergoing treatment with immune checkpoint inhibitors. METHODS We employed time-varying statistical models and machine learning classifiers in a Monte Carlo cross-validation approach to investigate the association between RECIST-defined progression and blood markers, serum tumour markers and their combination, in a retrospective cohort of 164 patients with NSCLC. RESULTS The performance of the routine blood markers in the prediction of progression free survival was moderate. Serum tumour markers and their combination with routine blood markers generally improved performance compared to routine blood markers alone. Elevated levels of C-reactive protein (CRP) and alkaline phosphatase (ALP) ranked as the top predictive routine blood markers, and CYFRA 21.1 was consistently among the most predictive serum tumour markers. Using these classifiers to predict overall survival yielded moderate to high performance, even when cases of death-defined progression were excluded. Performance varied across the treatment journey. CONCLUSION Routine blood tests, especially when combined with serum tumour markers, show moderate predictive value of RECIST-defined progression in NSCLC patients receiving immune checkpoint inhibitors. The relationship between overall survival and RECIST-defined progression may be influenced by confounding factors.
Collapse
Affiliation(s)
- Melda Yeghaian
- GROW Research Institute for Oncology and Reproduction, Maastricht University, Maastricht, The Netherlands
- Radiology Department, Netherlands Cancer Institute, Amsterdam, The Netherlands
| | - Teresa M Tareco Bucho
- GROW Research Institute for Oncology and Reproduction, Maastricht University, Maastricht, The Netherlands
- Radiology Department, Netherlands Cancer Institute, Amsterdam, The Netherlands
| | - Melissa de Bruin
- Radiology Department, Netherlands Cancer Institute, Amsterdam, The Netherlands
| | - Alexander Schmitz
- Radiology Department, Netherlands Cancer Institute, Amsterdam, The Netherlands
| | - Zuhir Bodalal
- GROW Research Institute for Oncology and Reproduction, Maastricht University, Maastricht, The Netherlands
- Radiology Department, Netherlands Cancer Institute, Amsterdam, The Netherlands
| | - Egbert F Smit
- Pulmonology Department, Leiden University Medical Center, Leiden, The Netherlands
| | - Regina G H Beets-Tan
- GROW Research Institute for Oncology and Reproduction, Maastricht University, Maastricht, The Netherlands
- Radiology Department, Netherlands Cancer Institute, Amsterdam, The Netherlands
- Faculty of Health Sciences, University of Southern Denmark, Odense, Denmark
| | - Daan van den Broek
- Department of Laboratory Medicine, Netherlands Cancer Institute, Amsterdam, The Netherlands
| | - Stefano Trebeschi
- GROW Research Institute for Oncology and Reproduction, Maastricht University, Maastricht, The Netherlands.
- Radiology Department, Netherlands Cancer Institute, Amsterdam, The Netherlands.
| |
Collapse
|
4
|
Xu JJ, Ulriksen PS, Jawad S, Rohde YZ, Sejer M, Achiam MP, Resch TA, Lönn L, Hansen KL. Iodine density mapping for the diagnosis of acute bowel ischemia using fast kV-switching dual-energy CT. Abdom Radiol (NY) 2024; 49:312-319. [PMID: 37978076 PMCID: PMC10789852 DOI: 10.1007/s00261-023-04097-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/13/2023] [Revised: 10/11/2023] [Accepted: 10/18/2023] [Indexed: 11/19/2023]
Abstract
PURPOSE To evaluate the diagnostic capabilities of a supplementary color ramped iodine density map compared to virtual monoenergetic images (VMIs) at 74 keV in the diagnosis of acute bowel ischemia (ABI). METHODS Data for this study were prospectively gathered and retrospectively evaluated. Patients referred to the Department of Diagnostic Radiology between October 2020 and August 2022 on the suspicion of ABI and underwent surgery < 12 h following fast kV-switching venous phase abdominal dual-energy CT (DECT) were consecutively included. Images were evaluated by two board-certified radiologists and two radiology residents. First round included only 74 keV VMIs resembling conventional 120 kVp images, and the second round included a supplementary iodine density map. Readers were asked to register presence of ABI as well as their confidence in their diagnosis based on a 5-point Likert scale. Sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) were calculated for each observer with the surgical findings as the gold-standard. McNemar's and Wilcoxon signed-rank test were used to compare registrations and diagnostic confidence across assessment rounds. RESULTS A total of 29 patients resulting in 31 DECT scans were included. Fourteen cases of ischemic/necrotic bowel were reported following surgery. Sensitivity and NPV were decreased with the use of supplementary iodine map images compared to 120 kVp-like images without supplementary iodine map images for three of four observers (round 1 range: 71.4-92.9% and 78.0-94.8%; round 2 range: 57.1-78.6% and 70.1-83.3%, respectively), while specificity and PPV were increased for three of four observers (round 1 range: 64.7-94.1% and 67.4-93.1%; round 2 range: 88.2-94.1% and 73.8-91.1%, respectively). However, no significant difference in ABI diagnosis or diagnostic confidence was found (p-value range: 0.07-1.00 and 0.23-0.58, respectively). CONCLUSION No significant difference for the diagnosis of ABI was found using supplementary iodine mapping. Our study may suggest a trend of increased specificity and decreased sensitivity, hence, the use of supplementary iodine mapping should be carefully considered.
Collapse
Affiliation(s)
- Jack Junchi Xu
- Department of Diagnostic Radiology, Copenhagen University Hospital, Rigshospitalet, 2100, Copenhagen, Denmark.
- Department of Clinical Medicine, University of Copenhagen, 2100, Copenhagen, Denmark.
| | - Peter Sommer Ulriksen
- Department of Diagnostic Radiology, Copenhagen University Hospital, Rigshospitalet, 2100, Copenhagen, Denmark
- Department of Clinical Medicine, University of Copenhagen, 2100, Copenhagen, Denmark
| | - Samir Jawad
- Department of Diagnostic Radiology, Copenhagen University Hospital, Rigshospitalet, 2100, Copenhagen, Denmark
| | - Yecatarina Zincuk Rohde
- Department of Diagnostic Radiology, Copenhagen University Hospital, Rigshospitalet, 2100, Copenhagen, Denmark
| | - Morten Sejer
- Department of Diagnostic Radiology, Copenhagen University Hospital, Rigshospitalet, 2100, Copenhagen, Denmark
| | - Michael Patrick Achiam
- Department of Clinical Medicine, University of Copenhagen, 2100, Copenhagen, Denmark
- Department of Surgery and Transplantation, Copenhagen University Hospital, Rigshospitalet, 2100, Copenhagen, Denmark
| | - Timothy Andrew Resch
- Department of Clinical Medicine, University of Copenhagen, 2100, Copenhagen, Denmark
- Department of Vascular Surgery, Copenhagen University Hospital, Rigshospitalet, 2100, Copenhagen, Denmark
| | - Lars Lönn
- Department of Diagnostic Radiology, Copenhagen University Hospital, Rigshospitalet, 2100, Copenhagen, Denmark
- Department of Clinical Medicine, University of Copenhagen, 2100, Copenhagen, Denmark
| | - Kristoffer Lindskov Hansen
- Department of Diagnostic Radiology, Copenhagen University Hospital, Rigshospitalet, 2100, Copenhagen, Denmark
- Department of Clinical Medicine, University of Copenhagen, 2100, Copenhagen, Denmark
| |
Collapse
|
5
|
Beaumont H, Iannessi A. Can we predict discordant RECIST 1.1 evaluations in double read clinical trials? Front Oncol 2023; 13:1239570. [PMID: 37869080 PMCID: PMC10585359 DOI: 10.3389/fonc.2023.1239570] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/13/2023] [Accepted: 09/05/2023] [Indexed: 10/24/2023] Open
Abstract
Background In lung clinical trials with imaging, blinded independent central review with double reads is recommended to reduce evaluation bias and the Response Evaluation Criteria In Solid Tumor (RECIST) is still widely used. We retrospectively analyzed the inter-reader discrepancies rate over time, the risk factors for discrepancies related to baseline evaluations, and the potential of machine learning to predict inter-reader discrepancies. Materials and methods We retrospectively analyzed five BICR clinical trials for patients on immunotherapy or targeted therapy for lung cancer. Double reads of 1724 patients involving 17 radiologists were performed using RECIST 1.1. We evaluated the rate of discrepancies over time according to four endpoints: progressive disease declared (PDD), date of progressive disease (DOPD), best overall response (BOR), and date of the first response (DOFR). Risk factors associated with discrepancies were analyzed, two predictive models were evaluated. Results At the end of trials, the discrepancy rates between trials were not different. On average, the discrepancy rates were 21.0%, 41.0%, 28.8%, and 48.8% for PDD, DOPD, BOR, and DOFR, respectively. Over time, the discrepancy rate was higher for DOFR than DOPD, and the rates increased as the trial progressed, even after accrual was completed. It was rare for readers to not find any disease, for less than 7% of patients, at least one reader selected non-measurable disease only (NTL). Often the readers selected some of their target lesions (TLs) and NTLs in different organs, with ranges of 36.0-57.9% and 60.5-73.5% of patients, respectively. Rarely (4-8.1%) two readers selected all their TLs in different locations. Significant risk factors were different depending on the endpoint and the trial being considered. Prediction had a poor performance but the positive predictive value was higher than 80%. The best classification was obtained with BOR. Conclusion Predicting discordance rates necessitates having knowledge of patient accrual, patient survival, and the probability of discordances over time. In lung cancer trials, although risk factors for inter-reader discrepancies are known, they are weakly significant, the ability to predict discrepancies from baseline data is limited. To boost prediction accuracy, it would be necessary to enhance baseline-derived features or create new ones, considering other risk factors and looking into optimal reader associations.
Collapse
|
6
|
Murphy PS, Galette P, van der Aart J, Janiczek RL, Patel N, Brown AP. The role of clinical imaging in oncology drug development: progress and new challenges. Br J Radiol 2023; 96:20211126. [PMID: 37393537 PMCID: PMC10546429 DOI: 10.1259/bjr.20211126] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/08/2021] [Revised: 02/14/2023] [Accepted: 06/06/2023] [Indexed: 07/03/2023] Open
Abstract
In 2008, the role of clinical imaging in oncology drug development was reviewed. The review outlined where imaging was being applied and considered the diverse demands across the phases of drug development. A limited set of imaging techniques was being used, largely based on structural measures of disease evaluated using established response criteria such as response evaluation criteria in solid tumours. Beyond structure, functional tissue imaging such as dynamic contrast-enhanced MRI and metabolic measures using [18F]flourodeoxyglucose positron emission tomography were being increasingly incorporated. Specific challenges related to the implementation of imaging were outlined including standardisation of scanning across study centres and consistency of analysis and reporting. More than a decade on the needs of modern drug development are reviewed, how imaging has evolved to support new drug development demands, the potential to translate state-of-the-art methods into routine tools and what is needed to enable the effective use of this broadening clinical trial toolset. In this review, we challenge the clinical and scientific imaging community to help refine existing clinical trial methods and innovate to deliver the next generation of techniques. Strong industry-academic partnerships and pre-competitive opportunities to co-ordinate efforts will ensure imaging technologies maintain a crucial role delivering innovative medicines to treat cancer.
Collapse
Affiliation(s)
| | - Paul Galette
- Telix Pharmaceuticals (US) Inc, Fishers, United States
| | | | | | | | - Andrew P. Brown
- Vale Imaging Consultancy Solutions, Harston, Cambridge, United Kingdom
| |
Collapse
|
7
|
Iannessi A, Beaumont H. Breaking down the RECIST 1.1 double read variability in lung trials: What do baseline assessments tell us? Front Oncol 2023; 13:988784. [PMID: 37007064 PMCID: PMC10060958 DOI: 10.3389/fonc.2023.988784] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/07/2022] [Accepted: 02/03/2023] [Indexed: 03/18/2023] Open
Abstract
BackgroundIn clinical trials with imaging, Blinded Independent Central Review (BICR) with double reads ensures data blinding and reduces bias in drug evaluations. As double reads can cause discrepancies, evaluations require close monitoring which substantially increases clinical trial costs. We sought to document the variability of double reads at baseline, and variabilities across individual readers and lung trials.Material and methodsWe retrospectively analyzed data from five BICR clinical trials evaluating 1720 lung cancer patients treated with immunotherapy or targeted therapy. Fifteen radiologists were involved. The variability was analyzed using a set of 71 features derived from tumor selection, measurements, and disease location. We selected a subset of readers that evaluated ≥50 patients in ≥two trials, to compare individual reader’s selections. Finally, we evaluated inter-trial homogeneity using a subset of patients for whom both readers assessed the exact same disease locations. Significance level was 0.05. Multiple pair-wise comparisons of continuous variables and proportions were performed using one-way ANOVA and Marascuilo procedure, respectively.ResultsAcross trials, on average per patient, target lesion (TL) number ranged 1.9 to 3.0, sum of tumor diameter (SOD) 57.1 to 91.9 mm. MeanSOD=83.7 mm. In four trials, MeanSOD of double reads was significantly different. Less than 10% of patients had TLs selected in completely different organs and 43.5% had at least one selected in different organs. Discrepancies in disease locations happened mainly in lymph nodes (20.1%) and bones (12.2%). Discrepancies in measurable disease happened mainly in lung (19.6%). Between individual readers, the MeanSOD and disease selection were significantly different (p<0.001). In inter-trials comparisons, on average per patient, the number of selected TLs ranged 2.1 to 2.8, MeanSOD 61.0 to 92.4 mm. Trials were significantly different in MeanSOD (p<0.0001) and average number of selected TLs (p=0.007). The proportion of patients having one of the top diseases was significantly different only between two trials for lung. Significant differences were observed for all other disease locations (p<0.05).ConclusionsWe found significant double read variabilities at baseline, evidence of reading patterns and a means to compare trials. Clinical trial reliability is influenced by the interplay of readers, patients and trial design.
Collapse
|
8
|
Inter-rater reliability of retrograde urethrograms. World J Urol 2023; 41:1163-1167. [PMID: 36800013 DOI: 10.1007/s00345-023-04323-0] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/05/2022] [Accepted: 01/30/2023] [Indexed: 02/18/2023] Open
Abstract
PURPOSE Reliability of pre-operative testing is important for adequate surgical planning. For urethral stricture disease, preoperative planning frequently includes retrograde urethrogram (RUG). The radiographic interpretation of RUGs is often done by urologists themselves. We aimed to evaluate the reliability of RUG interpretation by urologists at our institution. METHODS We examined the RUGs of 193 patients. These were deidentified and interpreted by three urologists, two general urologists and one reconstructive urologist. These interpretations were compared in 2 ways. Each of the general urologists was compared to the "gold standard" reconstructive urologist interpretation, and the general urologists were additionally compared to each other. We used intraclass correlation coefficient (ICC) for numerical variables and Fleiss' Kappa or Cohen's Kappa statistic (κ) for categorical variables to rate inter-interpreter reliability and agreement among interpretations with regards to the quantitative variables of stricture length and caliber. RESULTS Level of agreement ranged from poor to moderate across all variables interpreted. Comparing general urologists to the gold standard yielded no better than moderate agreement, with the majority being poor to fair. Similarly, agreement amongst the general urologists did not reach above moderate, with the majority being poor to slight. CONCLUSION To our knowledge, this is the first analysis of inter-rater reliability of RUGs among practicing urologists. Our analysis showed clinically unacceptable reliability with regards to stricture length, location, caliber, and indicated procedures. This study suggests a need for standardized interpretation of RUGs and poses an opportunity for actionable improvement in management of strictures.
Collapse
|
9
|
Ricard F, Barrington S, Korn R, Brueggenwerth G, Trotman J, Cheson B, Salles G, Schwartz L, Goldmacher G, Jarecha R, Narang J, Broussais F, Galette P, Liu M, Bajpai S, Perlman E, Gillis J, Smalberg I, Terve P, Zahlmann G, Schmid A. Application of the Lugano Classification for Initial Evaluation, Staging, and Response Assessment of Hodgkin and Non-Hodgkin Lymphoma: The PRoLoG Consensus Initiative (Part 2-Technical). J Nucl Med 2023; 64:239-243. [PMID: 35835581 PMCID: PMC9902846 DOI: 10.2967/jnumed.122.264124] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/11/2022] [Revised: 07/07/2022] [Accepted: 07/07/2022] [Indexed: 02/04/2023] Open
Abstract
The aim of this initiative was to provide consensus recommendations from a consortium of academic and industry experts in the field of lymphoma and imaging for the consistent application of imaging assessment with the Lugano classification. Methods: Consensus was obtained through a series of meetings from July 2019 to October 2021 sponsored by the PINTaD (Pharma Imaging Network for Therapeutics and Diagnostics) as part of the ProLoG (PINTaD RespOnse criteria in Lymphoma wOrking Group) consensus initiative. Results: Consensus recommendations encompass all technical imaging aspects of the Lugano classification. Some technical considerations for PET/CT and diagnostic CT are clarified with regards to required imaging series and scan visits, as well as acquisition and reconstruction of PET images and influence of lesion size and background activity. Recommendations are given on the role of imaging and clinical reviewers as well as on training and monitoring. Finally, an example template of an imaging case report form is provided to support efficient collection of data with Lugano Classification. Conclusion: Consensus recommendations are made to comprehensively address technical and imaging areas of inconsistency and ambiguity in the classification encountered by end users. Such guidance should be used to support standardized acquisition and evaluation with the Lugano 2014.
Collapse
Affiliation(s)
| | - Sally Barrington
- King’s College London and Guy’s and St. Thomas’ PET Centre, School of Biomedical Engineering and Imaging Sciences, King’s College London, King’s Health Partners, London, United Kingdom
| | - Ron Korn
- Adjunct Faculty TGEN/City of Hope and Imaging Endpoints Core Lab, Scottsdale, Arizona
| | | | - Judith Trotman
- Concord Repatriation General Hospital, University of Sydney, Concord, Australia
| | - Bruce Cheson
- Lymphoma Research Foundation, New York, New York
| | - Gilles Salles
- Department of Medicine, Memorial Sloan Kettering Cancer Center and Weil Cornell Medicine, New York, New York
| | - Larry Schwartz
- Department of Radiology, Columbia University College of Physicians and Surgeons and Radiologist-in-Chief, The New York Presbyterian Hospital, Columbia, New York
| | | | | | - Jayant Narang
- Takeda Pharmaceutical Company Ltd, Cambridge, Massachusetts
| | - Florence Broussais
- Lymphoma Study Association Research Center LYSARC, Pierre Benite, France Calyx International, Billerica, Massachusetts
| | | | - Min Liu
- Autolus Therapeutics, London, United Kingdom
| | | | - Eric Perlman
- Perlman Advisory Group LLC, Boynton Beach, Florida
| | | | - Ira Smalberg
- Saint John’s Cancer Institute and Tower Imaging Medical Group, Sherman Oaks, California
| | - Pierre Terve
- KEOSYS Medical Imaging, Saint Herblain, France; and
| | - Gudrun Zahlmann
- QIBA/RSNA, Radiological Society of North America, Oak Brook, Illinois
| | - Annette Schmid
- Takeda Pharmaceutical Company Ltd, Cambridge, Massachusetts
| |
Collapse
|
10
|
Is Artificial Intelligence Replacing Our Radiology Stars? Not Yet! EUR UROL SUPPL 2022; 48:14-16. [PMID: 36588775 PMCID: PMC9794880 DOI: 10.1016/j.euros.2022.09.024] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 09/06/2022] [Indexed: 12/23/2022] Open
Abstract
Artificial intelligence (AI) is here to stay and will change health care as we know it. The availability of big data and the increasing numbers of AI algorithms approved by the US Food and Drug Administration together will help in improving the quality of care for patients and in overcoming human fatigue barriers. In oncology practice, patients and providers rely on the interpretation of radiologists when making clinical decisions; however, there is considerable variability among readers, and in particular for prostate imaging. AI represents an emerging solution to this problem, for which it can provide a much-needed form of standardization. The diagnostic performance of AI alone in comparison to a combination of an AI framework and radiologist assessment for evaluation of prostate imaging has yet to be explored. Here, we compare the performance of radiologists alone versus a combination of radiologists aided by a modern computer-aided diagnosis (CAD) AI system. We show that the radiologist-CAD combination demonstrates superior sensitivity and specificity in comparison to both radiologists alone and AI alone. Our findings demonstrate that a radiologist + AI combination could perform best for detection of prostate cancer lesions. A hybrid technology-human system could leverage the benefits of AI in improving radiologist performance while also reducing physician workload, minimizing burnout, and enhancing the quality of patient care. Patient summary Our report demonstrates the potential of artificial intelligence (AI) for improving the interpretation of prostate scans. A combination of AI and evaluation by a radiologist has the best performance in determining the severity of prostate cancer. A hybrid system that uses both AI and radiologists could maximize the quality of care for patients while reducing physician workload and burnout.
Collapse
|
11
|
Abstract
Pituitary incidentalomas (PI) are lesions of the pituitary region discovered fortuitously by imaging for reasons unrelated to pituitary disease. They range from small cysts to large invasive adenomas. All over the world, improvements in the quality and availability of radiological examinations are leading to an increase in the discovery of PI. In the last four decades, significant advances have been made in the understanding of PI. Autopsy studies have shown that about 10% of deceased individuals harbour a PI, most often a non-functioning microadenoma. In contrast, modern patient series showed that among PIs that come to endocrinological attention, a significant proportion are macroadenomas, and many patients suffer from asymptomatic or pauci-symptomatic endocrine or ophthalmologic disturbances. Other than adenomas, empty sella, Rathke's cleft cyst, craniopharyngioma and meningioma are the most frequent types of PIs. About 10% of micro-incidentalomas and 25% of macro-incidentalomas grow over time. Most cases can be managed conservatively by mere surveillance. Follow-up is necessary in all patients with macroadenoma, but uncertainty remains for microadenomas as to the extent of endocrinological work-up as well as the necessity and duration of follow-up. Visual and endocrine anomalies constitute the most common indications for surgery. When needed, surgery yields better outcome in PIs than in symptomatic pituitary lesions.
Collapse
|
12
|
Raunig DL, Schmid AM, Miller CG, Walovitch RC, O'Connor M, Noever K, Hristova I, O'Neal M, Brueggenwerth G, Ford RR. Radiologists and Clinical Trials: Part 2: Practical Statistical Methods for Understanding and Monitoring Independent Reader Performance. Ther Innov Regul Sci 2021; 55:1122-1138. [PMID: 34244987 DOI: 10.1007/s43441-021-00317-5] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/11/2020] [Accepted: 06/18/2021] [Indexed: 01/02/2023]
Abstract
Though many clinical trials rely on medical image evaluations for primary or key secondary endpoints, the methods to monitor reader performance are all too often mired in the legacy use of adjudication rates. If misused, this simple metric can be misleading and sometimes entirely contradictory. Furthermore, attempts to overcome the limitations of adjudication rates using de novo or ad hoc methods often ignore well-established research conducted over the last half-century and can lead to inaccurate conclusions or variable interpretations. Underperforming readers can be missed, expert readers retrained, or worse, replaced. This paper aims to standardize reader performance evaluations using proven statistical methods. Additionally, these methods will describe how to discriminate between scenarios of concern and normal medical interpretation variability. Statistical methods are provided for inter-reader and intra-reader variability and bias, including the adjudicator's bias. Finally, we have compiled guidelines for calculating correct sample sizes, considerations for intra-reader memory recall, and applying alternative designs for independent readers.
Collapse
Affiliation(s)
- David L Raunig
- Takeda, 300 Massachusetts Ave, Cambridge, MA, 02139, USA.
| | | | | | | | | | | | | | | | | | | |
Collapse
|