1
|
Topff L, Steltenpool S, Ranschaert ER, Ramanauskas N, Menezes R, Visser JJ, Beets-Tan RGH, Hartkamp NS. Artificial intelligence-assisted double reading of chest radiographs to detect clinically relevant missed findings: a two-centre evaluation. Eur Radiol 2024:10.1007/s00330-024-10676-w. [PMID: 38466390 DOI: 10.1007/s00330-024-10676-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/21/2023] [Revised: 01/21/2024] [Accepted: 02/01/2024] [Indexed: 03/13/2024]
Abstract
OBJECTIVES To evaluate an artificial intelligence (AI)-assisted double reading system for detecting clinically relevant missed findings on routinely reported chest radiographs. METHODS A retrospective study was performed in two institutions, a secondary care hospital and tertiary referral oncology centre. Commercially available AI software performed a comparative analysis of chest radiographs and radiologists' authorised reports using a deep learning and natural language processing algorithm, respectively. The AI-detected discrepant findings between images and reports were assessed for clinical relevance by an external radiologist, as part of the commercial service provided by the AI vendor. The selected missed findings were subsequently returned to the institution's radiologist for final review. RESULTS In total, 25,104 chest radiographs of 21,039 patients (mean age 61.1 years ± 16.2 [SD]; 10,436 men) were included. The AI software detected discrepancies between imaging and reports in 21.1% (5289 of 25,104). After review by the external radiologist, 0.9% (47 of 5289) of cases were deemed to contain clinically relevant missed findings. The institution's radiologists confirmed 35 of 47 missed findings (74.5%) as clinically relevant (0.1% of all cases). Missed findings consisted of lung nodules (71.4%, 25 of 35), pneumothoraces (17.1%, 6 of 35) and consolidations (11.4%, 4 of 35). CONCLUSION The AI-assisted double reading system was able to identify missed findings on chest radiographs after report authorisation. The approach required an external radiologist to review the AI-detected discrepancies. The number of clinically relevant missed findings by radiologists was very low. CLINICAL RELEVANCE STATEMENT The AI-assisted double reader workflow was shown to detect diagnostic errors and could be applied as a quality assurance tool. Although clinically relevant missed findings were rare, there is potential impact given the common use of chest radiography. KEY POINTS • A commercially available double reading system supported by artificial intelligence was evaluated to detect reporting errors in chest radiographs (n=25,104) from two institutions. • Clinically relevant missed findings were found in 0.1% of chest radiographs and consisted of unreported lung nodules, pneumothoraces and consolidations. • Applying AI software as a secondary reader after report authorisation can assist in reducing diagnostic errors without interrupting the radiologist's reading workflow. However, the number of AI-detected discrepancies was considerable and required review by a radiologist to assess their relevance.
Collapse
Affiliation(s)
- Laurens Topff
- Department of Radiology, Netherlands Cancer Institute, Amsterdam, The Netherlands.
- GROW School for Oncology and Reproduction, Maastricht University, Maastricht, The Netherlands.
| | - Sanne Steltenpool
- Department of Radiology and Nuclear Medicine, Erasmus MC, University Medical Center Rotterdam, Rotterdam, The Netherlands
- Department of Radiology, Elisabeth-TweeSteden Hospital, Tilburg, The Netherlands
| | - Erik R Ranschaert
- Department of Radiology, St. Nikolaus Hospital, Eupen, Belgium
- Ghent University, Ghent, Belgium
| | - Naglis Ramanauskas
- Oxipit UAB, Vilnius, Lithuania
- Department of Radiology, Nuclear Medicine and Medical Physics, Institute of Biomedical Sciences, Faculty of Medicine, Vilnius University, Vilnius, Lithuania
| | - Renee Menezes
- Biostatistics Centre, Department of Psychosocial Research and Epidemiology, Netherlands Cancer Institute, Amsterdam, The Netherlands
| | - Jacob J Visser
- Department of Radiology and Nuclear Medicine, Erasmus MC, University Medical Center Rotterdam, Rotterdam, The Netherlands
| | - Regina G H Beets-Tan
- Department of Radiology, Netherlands Cancer Institute, Amsterdam, The Netherlands
- GROW School for Oncology and Reproduction, Maastricht University, Maastricht, The Netherlands
| | - Nolan S Hartkamp
- Department of Radiology, Elisabeth-TweeSteden Hospital, Tilburg, The Netherlands
| |
Collapse
|
2
|
Miró Catalina Q, Vidal-Alaball J, Fuster-Casanovas A, Escalé-Besa A, Ruiz Comellas A, Solé-Casals J. Real-world testing of an artificial intelligence algorithm for the analysis of chest X-rays in primary care settings. Sci Rep 2024; 14:5199. [PMID: 38431731 PMCID: PMC10908781 DOI: 10.1038/s41598-024-55792-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/30/2023] [Accepted: 02/27/2024] [Indexed: 03/05/2024] Open
Abstract
Interpreting chest X-rays is a complex task, and artificial intelligence algorithms for this purpose are currently being developed. It is important to perform external validations of these algorithms in order to implement them. This study therefore aims to externally validate an AI algorithm's diagnoses in real clinical practice, comparing them to a radiologist's diagnoses. The aim is also to identify diagnoses the algorithm may not have been trained for. A prospective observational study for the external validation of the AI algorithm in a region of Catalonia, comparing the AI algorithm's diagnosis with that of the reference radiologist, considered the gold standard. The external validation was performed with a sample of 278 images and reports, 51.8% of which showed no radiological abnormalities according to the radiologist's report. Analysing the validity of the AI algorithm, the average accuracy was 0.95 (95% CI 0.92; 0.98), the sensitivity was 0.48 (95% CI 0.30; 0.66) and the specificity was 0.98 (95% CI 0.97; 0.99). The conditions where the algorithm was most sensitive were external, upper abdominal and cardiac and/or valvular implants. On the other hand, the conditions where the algorithm was less sensitive were in the mediastinum, vessels and bone. The algorithm has been validated in the primary care setting and has proven to be useful when identifying images with or without conditions. However, in order to be a valuable tool to help and support experts, it requires additional real-world training to enhance its diagnostic capabilities for some of the conditions analysed. Our study emphasizes the need for continuous improvement to ensure the algorithm's effectiveness in primary care.
Collapse
Affiliation(s)
- Queralt Miró Catalina
- Unitat de Suport a la Recerca de la Catalunya Central, Fundació Institut Universitari per a la Recerca a l'Atenció Primària de Salut Jordi Gol i Gurina, Sant Fruitós de Bages, Spain
- Health Promotion in Rural Areas Research Group, Gerència d'Atenció Primària i a la Comunitat de la Catalunya Central, Institut Català de la Salut, Carrer Pica d'Estats, 13-15, 08272, Sant Fruitós de Bages, Barcelona, Spain
- Faculty of Science Technology and Engineering, University of Vic-Central University of Catalonia, Vic, Spain
| | - Josep Vidal-Alaball
- Unitat de Suport a la Recerca de la Catalunya Central, Fundació Institut Universitari per a la Recerca a l'Atenció Primària de Salut Jordi Gol i Gurina, Sant Fruitós de Bages, Spain.
- Health Promotion in Rural Areas Research Group, Gerència d'Atenció Primària i a la Comunitat de la Catalunya Central, Institut Català de la Salut, Carrer Pica d'Estats, 13-15, 08272, Sant Fruitós de Bages, Barcelona, Spain.
- Faculty of Medicine, University of Vic-Central University of Catalonia, Vic, Spain.
| | - Aïna Fuster-Casanovas
- Unitat de Suport a la Recerca de la Catalunya Central, Fundació Institut Universitari per a la Recerca a l'Atenció Primària de Salut Jordi Gol i Gurina, Sant Fruitós de Bages, Spain
- Health Promotion in Rural Areas Research Group, Gerència d'Atenció Primària i a la Comunitat de la Catalunya Central, Institut Català de la Salut, Carrer Pica d'Estats, 13-15, 08272, Sant Fruitós de Bages, Barcelona, Spain
| | - Anna Escalé-Besa
- Unitat de Suport a la Recerca de la Catalunya Central, Fundació Institut Universitari per a la Recerca a l'Atenció Primària de Salut Jordi Gol i Gurina, Sant Fruitós de Bages, Spain
- Health Promotion in Rural Areas Research Group, Gerència d'Atenció Primària i a la Comunitat de la Catalunya Central, Institut Català de la Salut, Carrer Pica d'Estats, 13-15, 08272, Sant Fruitós de Bages, Barcelona, Spain
- Faculty of Medicine, University of Vic-Central University of Catalonia, Vic, Spain
| | - Anna Ruiz Comellas
- Unitat de Suport a la Recerca de la Catalunya Central, Fundació Institut Universitari per a la Recerca a l'Atenció Primària de Salut Jordi Gol i Gurina, Sant Fruitós de Bages, Spain
- Health Promotion in Rural Areas Research Group, Gerència d'Atenció Primària i a la Comunitat de la Catalunya Central, Institut Català de la Salut, Carrer Pica d'Estats, 13-15, 08272, Sant Fruitós de Bages, Barcelona, Spain
- Faculty of Medicine, University of Vic-Central University of Catalonia, Vic, Spain
| | - Jordi Solé-Casals
- Data and Signal Processing Group, Faculty of Science, Technology and Engineering, University of Vic-Central University of Catalonia, Vic, Spain.
- Department of Psychiatry, University of Cambridge, Cambridge, UK.
| |
Collapse
|
3
|
Grenier PA, Brun AL, Mellot F. [The contribution of artificial intelligence (AI) subsequent to the processing of thoracic imaging]. Rev Mal Respir 2024; 41:110-126. [PMID: 38129269 DOI: 10.1016/j.rmr.2023.12.001] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/24/2023] [Accepted: 11/27/2023] [Indexed: 12/23/2023]
Abstract
The contribution of artificial intelligence (AI) to medical imaging is currently the object of widespread experimentation. The development of deep learning (DL) methods, particularly convolution neural networks (CNNs), has led to performance gains often superior to those achieved by conventional methods such as machine learning. Radiomics is an approach aimed at extracting quantitative data not accessible to the human eye from images expressing a disease. The data subsequently feed machine learning models and produce diagnostic or prognostic probabilities. As for the multiple applications of AI methods in thoracic imaging, they are undergoing evaluation. Chest radiography is a practically ideal field for the development of DL algorithms able to automatically interpret X-rays. Current algorithms can detect up to 14 different abnormalities present either in isolation or in combination. Chest CT is another area offering numerous AI applications. Various algorithms have been specifically formed and validated for the detection and characterization of pulmonary nodules and pulmonary embolism, as well as segmentation and quantitative analysis of the extent of diffuse lung diseases (emphysema, infectious pneumonias, interstitial lung disease). In addition, the analysis of medical images can be associated with clinical, biological, and functional data (multi-omics analysis), the objective being to construct predictive approaches regarding disease prognosis and response to treatment.
Collapse
Affiliation(s)
- P A Grenier
- Délégation à la recherche clinique et l'innovation, hôpital Foch, Suresnes, France.
| | - A L Brun
- Service de radiologie, hôpital Foch, Suresnes, France
| | - F Mellot
- Service de radiologie, hôpital Foch, Suresnes, France
| |
Collapse
|
4
|
Vidal-Alaball J, Panadés Zafra R, Escalé-Besa A, Martinez-Millana A. The artificial intelligence revolution in primary care: Challenges, dilemmas and opportunities. Aten Primaria 2024; 56:102820. [PMID: 38056048 PMCID: PMC10714322 DOI: 10.1016/j.aprim.2023.102820] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/06/2023] [Accepted: 11/08/2023] [Indexed: 12/08/2023] Open
Abstract
Artificial intelligence (AI) can be a valuable tool for primary care (PC), as, among other things, it can help healthcare professionals improve diagnostic accuracy, chronic disease management and the overall efficiency of the care they provide. It is important to emphasise that AI should not be seen as a replacement tool, but as an aid to PC professionals. Although AI is capable of processing large amounts of data and generating accurate predictions, it cannot replace the skill and expertise of professionals in clinical decision making. AI still requires the interpretation and clinical judgement of a trained healthcare professional and cannot provide the empathy and emotional support often required in healthcare.
Collapse
Affiliation(s)
- Josep Vidal-Alaball
- Unitat de Suport a la Recerca de la Catalunya Central, Fundació Institut Universitari per a la Recerca a l'Atenció Primària de Salut Jordi Gol i Gurina, Sant Fruitós de Bages, Barcelona, Spain; Grup de Recerca Promoció de la Salut en l'Àmbit Rural, Gerència d'Atenció Primària i a la Comunitat de Catalunya Central, Institut Català de la Salut, Sant Fruitós de Bages, Barcelona, Spain; Facultat de Medicina, Universitat de Vic-Universitat Central de Catalunya, Vic, Barcelona, Spain; Grup de Salut Digital CAMFIC, Barcelona, Spain
| | - Robert Panadés Zafra
- Grup de Recerca Promoció de la Salut en l'Àmbit Rural, Gerència d'Atenció Primària i a la Comunitat de Catalunya Central, Institut Català de la Salut, Sant Fruitós de Bages, Barcelona, Spain; Grup de Salut Digital CAMFIC, Barcelona, Spain; Equip d'Atenció Primària d'Anoia Rural, Gerència d'Atenció Primària i a la Comunitat de Catalunya Central, Institut Català de la Salut, Jorba i Copons, Barcelona, Spain
| | - Anna Escalé-Besa
- Grup de Recerca Promoció de la Salut en l'Àmbit Rural, Gerència d'Atenció Primària i a la Comunitat de Catalunya Central, Institut Català de la Salut, Sant Fruitós de Bages, Barcelona, Spain; Grup de Salut Digital CAMFIC, Barcelona, Spain; Equip d'Atenció Primària Navàs-Balsareny, Gerència d'Atenció Primària i a la Comunitat de Catalunya Central, Institut Català de la Salut, Navàs, Barcelona, Spain.
| | - Antonio Martinez-Millana
- Grup de Salut Digital CAMFIC, Barcelona, Spain; Instituto Universitario de Investigación de Aplicaciones de las Tecnologías de la Información y de las Comunicaciones Avanzadas, Universitat Politècnica de València, Valencia, Spain
| |
Collapse
|
5
|
Tripathi S, Tabari A, Mansur A, Dabbara H, Bridge CP, Daye D. From Machine Learning to Patient Outcomes: A Comprehensive Review of AI in Pancreatic Cancer. Diagnostics (Basel) 2024; 14:174. [PMID: 38248051 PMCID: PMC10814554 DOI: 10.3390/diagnostics14020174] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/19/2023] [Revised: 12/28/2023] [Accepted: 12/29/2023] [Indexed: 01/23/2024] Open
Abstract
Pancreatic cancer is a highly aggressive and difficult-to-detect cancer with a poor prognosis. Late diagnosis is common due to a lack of early symptoms, specific markers, and the challenging location of the pancreas. Imaging technologies have improved diagnosis, but there is still room for improvement in standardizing guidelines. Biopsies and histopathological analysis are challenging due to tumor heterogeneity. Artificial Intelligence (AI) revolutionizes healthcare by improving diagnosis, treatment, and patient care. AI algorithms can analyze medical images with precision, aiding in early disease detection. AI also plays a role in personalized medicine by analyzing patient data to tailor treatment plans. It streamlines administrative tasks, such as medical coding and documentation, and provides patient assistance through AI chatbots. However, challenges include data privacy, security, and ethical considerations. This review article focuses on the potential of AI in transforming pancreatic cancer care, offering improved diagnostics, personalized treatments, and operational efficiency, leading to better patient outcomes.
Collapse
Affiliation(s)
- Satvik Tripathi
- Department of Radiology, Massachusetts General Hospital, Boston, MA 02114, USA; (S.T.); (A.T.); (A.M.); (C.P.B.)
- Athinoula A. Martinos Center for Biomedical Imaging, Charlestown, MA 02129, USA
- Harvard Medical School, Boston, MA 02115, USA
| | - Azadeh Tabari
- Department of Radiology, Massachusetts General Hospital, Boston, MA 02114, USA; (S.T.); (A.T.); (A.M.); (C.P.B.)
- Harvard Medical School, Boston, MA 02115, USA
| | - Arian Mansur
- Department of Radiology, Massachusetts General Hospital, Boston, MA 02114, USA; (S.T.); (A.T.); (A.M.); (C.P.B.)
- Harvard Medical School, Boston, MA 02115, USA
| | - Harika Dabbara
- Boston University Chobanian & Avedisian School of Medicine, Boston, MA 02118, USA;
| | - Christopher P. Bridge
- Department of Radiology, Massachusetts General Hospital, Boston, MA 02114, USA; (S.T.); (A.T.); (A.M.); (C.P.B.)
- Athinoula A. Martinos Center for Biomedical Imaging, Charlestown, MA 02129, USA
- Harvard Medical School, Boston, MA 02115, USA
| | - Dania Daye
- Department of Radiology, Massachusetts General Hospital, Boston, MA 02114, USA; (S.T.); (A.T.); (A.M.); (C.P.B.)
- Athinoula A. Martinos Center for Biomedical Imaging, Charlestown, MA 02129, USA
- Harvard Medical School, Boston, MA 02115, USA
| |
Collapse
|
6
|
Bennani S, Regnard NE, Ventre J, Lassalle L, Nguyen T, Ducarouge A, Dargent L, Guillo E, Gouhier E, Zaimi SH, Canniff E, Malandrin C, Khafagy P, Koulakian H, Revel MP, Chassagnon G. Using AI to Improve Radiologist Performance in Detection of Abnormalities on Chest Radiographs. Radiology 2023; 309:e230860. [PMID: 38085079 DOI: 10.1148/radiol.230860] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/18/2023]
Abstract
Background Chest radiography remains the most common radiologic examination, and interpretation of its results can be difficult. Purpose To explore the potential benefit of artificial intelligence (AI) assistance in the detection of thoracic abnormalities on chest radiographs by evaluating the performance of radiologists with different levels of expertise, with and without AI assistance. Materials and Methods Patients who underwent both chest radiography and thoracic CT within 72 hours between January 2010 and December 2020 in a French public hospital were screened retrospectively. Radiographs were randomly included until reaching 500 radiographs, with about 50% of radiographs having abnormal findings. A senior thoracic radiologist annotated the radiographs for five abnormalities (pneumothorax, pleural effusion, consolidation, mediastinal and hilar mass, lung nodule) based on the corresponding CT results (ground truth). A total of 12 readers (four thoracic radiologists, four general radiologists, four radiology residents) read half the radiographs without AI and half the radiographs with AI (ChestView; Gleamer). Changes in sensitivity and specificity were measured using paired t tests. Results The study included 500 patients (mean age, 54 years ± 19 [SD]; 261 female, 239 male), with 522 abnormalities visible on 241 radiographs. On average, for all readers, AI use resulted in an absolute increase in sensitivity of 26% (95% CI: 20, 32), 14% (95% CI: 11, 17), 12% (95% CI: 10, 14), 8.5% (95% CI: 6, 11), and 5.9% (95% CI: 4, 8) for pneumothorax, consolidation, nodule, pleural effusion, and mediastinal and hilar mass, respectively (P < .001). Specificity increased with AI assistance (3.9% [95% CI: 3.2, 4.6], 3.7% [95% CI: 3, 4.4], 2.9% [95% CI: 2.3, 3.5], and 2.1% [95% CI: 1.6, 2.6] for pleural effusion, mediastinal and hilar mass, consolidation, and nodule, respectively), except in the diagnosis of pneumothorax (-0.2%; 95% CI: -0.36, -0.04; P = .01). The mean reading time was 81 seconds without AI versus 56 seconds with AI (31% decrease, P < .001). Conclusion AI-assisted chest radiography interpretation resulted in absolute increases in sensitivity for all radiologists of various levels of expertise and reduced the reading times; specificity increased with AI, except in the diagnosis of pneumothorax. © RSNA, 2023 Supplemental material is available for this article.
Collapse
Affiliation(s)
- Souhail Bennani
- From the Department of Thoracic Imaging, Cochin Hospital, AP-HP, 27 Rue du Faubourg Saint-Jacques, Paris 75014, France (S.B., L.D., E. Guillo, E. Gouhier, S.H.Z., E.C., M.P.R., G.C.); Gleamer, Paris, France (S.B., N.E.R., J.V., L.L., T.N., A.D.); Réseau d'Imagerie Sud Francilien, Lieusant, France (N.E.R., L.L., C.M.); Department of Pediatric Radiology, Armand Trousseau Hospital, AP-HP, Paris, France (T.N.); HFR Fribourg, Fribourg, Switzerland (P.K.); and Centre d'Imagerie Médicale de l'Ouest Parisien, Paris, France (H.K.)
| | - Nor-Eddine Regnard
- From the Department of Thoracic Imaging, Cochin Hospital, AP-HP, 27 Rue du Faubourg Saint-Jacques, Paris 75014, France (S.B., L.D., E. Guillo, E. Gouhier, S.H.Z., E.C., M.P.R., G.C.); Gleamer, Paris, France (S.B., N.E.R., J.V., L.L., T.N., A.D.); Réseau d'Imagerie Sud Francilien, Lieusant, France (N.E.R., L.L., C.M.); Department of Pediatric Radiology, Armand Trousseau Hospital, AP-HP, Paris, France (T.N.); HFR Fribourg, Fribourg, Switzerland (P.K.); and Centre d'Imagerie Médicale de l'Ouest Parisien, Paris, France (H.K.)
| | - Jeanne Ventre
- From the Department of Thoracic Imaging, Cochin Hospital, AP-HP, 27 Rue du Faubourg Saint-Jacques, Paris 75014, France (S.B., L.D., E. Guillo, E. Gouhier, S.H.Z., E.C., M.P.R., G.C.); Gleamer, Paris, France (S.B., N.E.R., J.V., L.L., T.N., A.D.); Réseau d'Imagerie Sud Francilien, Lieusant, France (N.E.R., L.L., C.M.); Department of Pediatric Radiology, Armand Trousseau Hospital, AP-HP, Paris, France (T.N.); HFR Fribourg, Fribourg, Switzerland (P.K.); and Centre d'Imagerie Médicale de l'Ouest Parisien, Paris, France (H.K.)
| | - Louis Lassalle
- From the Department of Thoracic Imaging, Cochin Hospital, AP-HP, 27 Rue du Faubourg Saint-Jacques, Paris 75014, France (S.B., L.D., E. Guillo, E. Gouhier, S.H.Z., E.C., M.P.R., G.C.); Gleamer, Paris, France (S.B., N.E.R., J.V., L.L., T.N., A.D.); Réseau d'Imagerie Sud Francilien, Lieusant, France (N.E.R., L.L., C.M.); Department of Pediatric Radiology, Armand Trousseau Hospital, AP-HP, Paris, France (T.N.); HFR Fribourg, Fribourg, Switzerland (P.K.); and Centre d'Imagerie Médicale de l'Ouest Parisien, Paris, France (H.K.)
| | - Toan Nguyen
- From the Department of Thoracic Imaging, Cochin Hospital, AP-HP, 27 Rue du Faubourg Saint-Jacques, Paris 75014, France (S.B., L.D., E. Guillo, E. Gouhier, S.H.Z., E.C., M.P.R., G.C.); Gleamer, Paris, France (S.B., N.E.R., J.V., L.L., T.N., A.D.); Réseau d'Imagerie Sud Francilien, Lieusant, France (N.E.R., L.L., C.M.); Department of Pediatric Radiology, Armand Trousseau Hospital, AP-HP, Paris, France (T.N.); HFR Fribourg, Fribourg, Switzerland (P.K.); and Centre d'Imagerie Médicale de l'Ouest Parisien, Paris, France (H.K.)
| | - Alexis Ducarouge
- From the Department of Thoracic Imaging, Cochin Hospital, AP-HP, 27 Rue du Faubourg Saint-Jacques, Paris 75014, France (S.B., L.D., E. Guillo, E. Gouhier, S.H.Z., E.C., M.P.R., G.C.); Gleamer, Paris, France (S.B., N.E.R., J.V., L.L., T.N., A.D.); Réseau d'Imagerie Sud Francilien, Lieusant, France (N.E.R., L.L., C.M.); Department of Pediatric Radiology, Armand Trousseau Hospital, AP-HP, Paris, France (T.N.); HFR Fribourg, Fribourg, Switzerland (P.K.); and Centre d'Imagerie Médicale de l'Ouest Parisien, Paris, France (H.K.)
| | - Lucas Dargent
- From the Department of Thoracic Imaging, Cochin Hospital, AP-HP, 27 Rue du Faubourg Saint-Jacques, Paris 75014, France (S.B., L.D., E. Guillo, E. Gouhier, S.H.Z., E.C., M.P.R., G.C.); Gleamer, Paris, France (S.B., N.E.R., J.V., L.L., T.N., A.D.); Réseau d'Imagerie Sud Francilien, Lieusant, France (N.E.R., L.L., C.M.); Department of Pediatric Radiology, Armand Trousseau Hospital, AP-HP, Paris, France (T.N.); HFR Fribourg, Fribourg, Switzerland (P.K.); and Centre d'Imagerie Médicale de l'Ouest Parisien, Paris, France (H.K.)
| | - Enora Guillo
- From the Department of Thoracic Imaging, Cochin Hospital, AP-HP, 27 Rue du Faubourg Saint-Jacques, Paris 75014, France (S.B., L.D., E. Guillo, E. Gouhier, S.H.Z., E.C., M.P.R., G.C.); Gleamer, Paris, France (S.B., N.E.R., J.V., L.L., T.N., A.D.); Réseau d'Imagerie Sud Francilien, Lieusant, France (N.E.R., L.L., C.M.); Department of Pediatric Radiology, Armand Trousseau Hospital, AP-HP, Paris, France (T.N.); HFR Fribourg, Fribourg, Switzerland (P.K.); and Centre d'Imagerie Médicale de l'Ouest Parisien, Paris, France (H.K.)
| | - Elodie Gouhier
- From the Department of Thoracic Imaging, Cochin Hospital, AP-HP, 27 Rue du Faubourg Saint-Jacques, Paris 75014, France (S.B., L.D., E. Guillo, E. Gouhier, S.H.Z., E.C., M.P.R., G.C.); Gleamer, Paris, France (S.B., N.E.R., J.V., L.L., T.N., A.D.); Réseau d'Imagerie Sud Francilien, Lieusant, France (N.E.R., L.L., C.M.); Department of Pediatric Radiology, Armand Trousseau Hospital, AP-HP, Paris, France (T.N.); HFR Fribourg, Fribourg, Switzerland (P.K.); and Centre d'Imagerie Médicale de l'Ouest Parisien, Paris, France (H.K.)
| | - Sophie-Hélène Zaimi
- From the Department of Thoracic Imaging, Cochin Hospital, AP-HP, 27 Rue du Faubourg Saint-Jacques, Paris 75014, France (S.B., L.D., E. Guillo, E. Gouhier, S.H.Z., E.C., M.P.R., G.C.); Gleamer, Paris, France (S.B., N.E.R., J.V., L.L., T.N., A.D.); Réseau d'Imagerie Sud Francilien, Lieusant, France (N.E.R., L.L., C.M.); Department of Pediatric Radiology, Armand Trousseau Hospital, AP-HP, Paris, France (T.N.); HFR Fribourg, Fribourg, Switzerland (P.K.); and Centre d'Imagerie Médicale de l'Ouest Parisien, Paris, France (H.K.)
| | - Emma Canniff
- From the Department of Thoracic Imaging, Cochin Hospital, AP-HP, 27 Rue du Faubourg Saint-Jacques, Paris 75014, France (S.B., L.D., E. Guillo, E. Gouhier, S.H.Z., E.C., M.P.R., G.C.); Gleamer, Paris, France (S.B., N.E.R., J.V., L.L., T.N., A.D.); Réseau d'Imagerie Sud Francilien, Lieusant, France (N.E.R., L.L., C.M.); Department of Pediatric Radiology, Armand Trousseau Hospital, AP-HP, Paris, France (T.N.); HFR Fribourg, Fribourg, Switzerland (P.K.); and Centre d'Imagerie Médicale de l'Ouest Parisien, Paris, France (H.K.)
| | - Cécile Malandrin
- From the Department of Thoracic Imaging, Cochin Hospital, AP-HP, 27 Rue du Faubourg Saint-Jacques, Paris 75014, France (S.B., L.D., E. Guillo, E. Gouhier, S.H.Z., E.C., M.P.R., G.C.); Gleamer, Paris, France (S.B., N.E.R., J.V., L.L., T.N., A.D.); Réseau d'Imagerie Sud Francilien, Lieusant, France (N.E.R., L.L., C.M.); Department of Pediatric Radiology, Armand Trousseau Hospital, AP-HP, Paris, France (T.N.); HFR Fribourg, Fribourg, Switzerland (P.K.); and Centre d'Imagerie Médicale de l'Ouest Parisien, Paris, France (H.K.)
| | - Philippe Khafagy
- From the Department of Thoracic Imaging, Cochin Hospital, AP-HP, 27 Rue du Faubourg Saint-Jacques, Paris 75014, France (S.B., L.D., E. Guillo, E. Gouhier, S.H.Z., E.C., M.P.R., G.C.); Gleamer, Paris, France (S.B., N.E.R., J.V., L.L., T.N., A.D.); Réseau d'Imagerie Sud Francilien, Lieusant, France (N.E.R., L.L., C.M.); Department of Pediatric Radiology, Armand Trousseau Hospital, AP-HP, Paris, France (T.N.); HFR Fribourg, Fribourg, Switzerland (P.K.); and Centre d'Imagerie Médicale de l'Ouest Parisien, Paris, France (H.K.)
| | - Hasmik Koulakian
- From the Department of Thoracic Imaging, Cochin Hospital, AP-HP, 27 Rue du Faubourg Saint-Jacques, Paris 75014, France (S.B., L.D., E. Guillo, E. Gouhier, S.H.Z., E.C., M.P.R., G.C.); Gleamer, Paris, France (S.B., N.E.R., J.V., L.L., T.N., A.D.); Réseau d'Imagerie Sud Francilien, Lieusant, France (N.E.R., L.L., C.M.); Department of Pediatric Radiology, Armand Trousseau Hospital, AP-HP, Paris, France (T.N.); HFR Fribourg, Fribourg, Switzerland (P.K.); and Centre d'Imagerie Médicale de l'Ouest Parisien, Paris, France (H.K.)
| | - Marie-Pierre Revel
- From the Department of Thoracic Imaging, Cochin Hospital, AP-HP, 27 Rue du Faubourg Saint-Jacques, Paris 75014, France (S.B., L.D., E. Guillo, E. Gouhier, S.H.Z., E.C., M.P.R., G.C.); Gleamer, Paris, France (S.B., N.E.R., J.V., L.L., T.N., A.D.); Réseau d'Imagerie Sud Francilien, Lieusant, France (N.E.R., L.L., C.M.); Department of Pediatric Radiology, Armand Trousseau Hospital, AP-HP, Paris, France (T.N.); HFR Fribourg, Fribourg, Switzerland (P.K.); and Centre d'Imagerie Médicale de l'Ouest Parisien, Paris, France (H.K.)
| | - Guillaume Chassagnon
- From the Department of Thoracic Imaging, Cochin Hospital, AP-HP, 27 Rue du Faubourg Saint-Jacques, Paris 75014, France (S.B., L.D., E. Guillo, E. Gouhier, S.H.Z., E.C., M.P.R., G.C.); Gleamer, Paris, France (S.B., N.E.R., J.V., L.L., T.N., A.D.); Réseau d'Imagerie Sud Francilien, Lieusant, France (N.E.R., L.L., C.M.); Department of Pediatric Radiology, Armand Trousseau Hospital, AP-HP, Paris, France (T.N.); HFR Fribourg, Fribourg, Switzerland (P.K.); and Centre d'Imagerie Médicale de l'Ouest Parisien, Paris, France (H.K.)
| |
Collapse
|
7
|
Ng AY, Oberije CJG, Ambrózay É, Szabó E, Serfőző O, Karpati E, Fox G, Glocker B, Morris EA, Forrai G, Kecskemethy PD. Prospective implementation of AI-assisted screen reading to improve early detection of breast cancer. Nat Med 2023; 29:3044-3049. [PMID: 37973948 PMCID: PMC10719086 DOI: 10.1038/s41591-023-02625-9] [Citation(s) in RCA: 5] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/08/2022] [Accepted: 10/04/2023] [Indexed: 11/19/2023]
Abstract
Artificial intelligence (AI) has the potential to improve breast cancer screening; however, prospective evidence of the safe implementation of AI into real clinical practice is limited. A commercially available AI system was implemented as an additional reader to standard double reading to flag cases for further arbitration review among screened women. Performance was assessed prospectively in three phases: a single-center pilot rollout, a wider multicenter pilot rollout and a full live rollout. The results showed that, compared to double reading, implementing the AI-assisted additional-reader process could achieve 0.7-1.6 additional cancer detection per 1,000 cases, with 0.16-0.30% additional recalls, 0-0.23% unnecessary recalls and a 0.1-1.9% increase in positive predictive value (PPV) after 7-11% additional human reads of AI-flagged cases (equating to 4-6% additional overall reading workload). The majority of cancerous cases detected by the AI-assisted additional-reader process were invasive (83.3%) and small-sized (≤10 mm, 47.0%). This evaluation suggests that using AI as an additional reader can improve the early detection of breast cancer with relevant prognostic features, with minimal to no unnecessary recalls. Although the AI-assisted additional-reader workflow requires additional reads, the higher PPV suggests that it can increase screening effectiveness.
Collapse
Affiliation(s)
- Annie Y Ng
- Kheiron Medical Technologies, London, UK.
| | | | | | | | | | | | | | - Ben Glocker
- Kheiron Medical Technologies, London, UK
- Department of Computing, Imperial College London, London, UK
| | | | | | | |
Collapse
|
8
|
Lind Plesner L, Müller FC, Brejnebøl MW, Laustrup LC, Rasmussen F, Nielsen OW, Boesen M, Brun Andersen M. Commercially Available Chest Radiograph AI Tools for Detecting Airspace Disease, Pneumothorax, and Pleural Effusion. Radiology 2023; 308:e231236. [PMID: 37750768 DOI: 10.1148/radiol.231236] [Citation(s) in RCA: 14] [Impact Index Per Article: 14.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 09/27/2023]
Abstract
Background Commercially available artificial intelligence (AI) tools can assist radiologists in interpreting chest radiographs, but their real-life diagnostic accuracy remains unclear. Purpose To evaluate the diagnostic accuracy of four commercially available AI tools for detection of airspace disease, pneumothorax, and pleural effusion on chest radiographs. Materials and Methods This retrospective study included consecutive adult patients who underwent chest radiography at one of four Danish hospitals in January 2020. Two thoracic radiologists (or three, in cases of disagreement) who had access to all previous and future imaging labeled chest radiographs independently for the reference standard. Area under the receiver operating characteristic curve, sensitivity, and specificity were calculated. Sensitivity and specificity were additionally stratified according to the severity of findings, number of findings on chest radiographs, and radiographic projection. The χ2 and McNemar tests were used for comparisons. Results The data set comprised 2040 patients (median age, 72 years [IQR, 58-81 years]; 1033 female), of whom 669 (32.8%) had target findings. The AI tools demonstrated areas under the receiver operating characteristic curve ranging 0.83-0.88 for airspace disease, 0.89-0.97 for pneumothorax, and 0.94-0.97 for pleural effusion. Sensitivities ranged 72%-91% for airspace disease, 63%-90% for pneumothorax, and 62%-95% for pleural effusion. Negative predictive values ranged 92%-100% for all target findings. In airspace disease, pneumothorax, and pleural effusion, specificity was high for chest radiographs with normal or single findings (range, 85%-96%, 99%-100%, and 95%-100%, respectively) and markedly lower for chest radiographs with four or more findings (range, 27%-69%, 96%-99%, 65%-92%, respectively) (P < .001). AI sensitivity was lower for vague airspace disease (range, 33%-61%) and small pneumothorax or pleural effusion (range, 9%-94%) compared with larger findings (range, 81%-100%; P value range, > .99 to < .001). Conclusion Current-generation AI tools showed moderate to high sensitivity for detecting airspace disease, pneumothorax, and pleural effusion on chest radiographs. However, they produced more false-positive findings than radiology reports, and their performance decreased for smaller-sized target findings and when multiple findings were present. © RSNA, 2023 Supplemental material is available for this article. See also the editorial by Yanagawa and Tomiyama in this issue.
Collapse
Affiliation(s)
- Louis Lind Plesner
- From the Department of Radiology, Herlev and Gentofte Hospital, Borgmester Ib, Juuls vej 1 Herlev, Copenhagen 2730, Denmark (L.L.P., F.C.M., M.W.B., L.C.L., M.B.A.); Faculty of Health Sciences, University of Copenhagen, Copenhagen, Denmark (L.L.P., M.W.B., O.W.N., M.B., M.B.A.); Radiological Artificial Intelligence Testcenter, RAIT.dk, Capital Region of Denmark (L.L.P., F.C.M., M.W.B., M.B., M.B.A.); Departments of Radiology (M.W.B., M.B.) and Cardiology (O.W.N.), Bispebjerg and Frederiksberg Hospital, Copenhagen, Denmark; and Department of Radiology, Aarhus University Hospital, Aarhus, Denmark (F.R.)
| | - Felix C Müller
- From the Department of Radiology, Herlev and Gentofte Hospital, Borgmester Ib, Juuls vej 1 Herlev, Copenhagen 2730, Denmark (L.L.P., F.C.M., M.W.B., L.C.L., M.B.A.); Faculty of Health Sciences, University of Copenhagen, Copenhagen, Denmark (L.L.P., M.W.B., O.W.N., M.B., M.B.A.); Radiological Artificial Intelligence Testcenter, RAIT.dk, Capital Region of Denmark (L.L.P., F.C.M., M.W.B., M.B., M.B.A.); Departments of Radiology (M.W.B., M.B.) and Cardiology (O.W.N.), Bispebjerg and Frederiksberg Hospital, Copenhagen, Denmark; and Department of Radiology, Aarhus University Hospital, Aarhus, Denmark (F.R.)
| | - Mathias W Brejnebøl
- From the Department of Radiology, Herlev and Gentofte Hospital, Borgmester Ib, Juuls vej 1 Herlev, Copenhagen 2730, Denmark (L.L.P., F.C.M., M.W.B., L.C.L., M.B.A.); Faculty of Health Sciences, University of Copenhagen, Copenhagen, Denmark (L.L.P., M.W.B., O.W.N., M.B., M.B.A.); Radiological Artificial Intelligence Testcenter, RAIT.dk, Capital Region of Denmark (L.L.P., F.C.M., M.W.B., M.B., M.B.A.); Departments of Radiology (M.W.B., M.B.) and Cardiology (O.W.N.), Bispebjerg and Frederiksberg Hospital, Copenhagen, Denmark; and Department of Radiology, Aarhus University Hospital, Aarhus, Denmark (F.R.)
| | - Lene C Laustrup
- From the Department of Radiology, Herlev and Gentofte Hospital, Borgmester Ib, Juuls vej 1 Herlev, Copenhagen 2730, Denmark (L.L.P., F.C.M., M.W.B., L.C.L., M.B.A.); Faculty of Health Sciences, University of Copenhagen, Copenhagen, Denmark (L.L.P., M.W.B., O.W.N., M.B., M.B.A.); Radiological Artificial Intelligence Testcenter, RAIT.dk, Capital Region of Denmark (L.L.P., F.C.M., M.W.B., M.B., M.B.A.); Departments of Radiology (M.W.B., M.B.) and Cardiology (O.W.N.), Bispebjerg and Frederiksberg Hospital, Copenhagen, Denmark; and Department of Radiology, Aarhus University Hospital, Aarhus, Denmark (F.R.)
| | - Finn Rasmussen
- From the Department of Radiology, Herlev and Gentofte Hospital, Borgmester Ib, Juuls vej 1 Herlev, Copenhagen 2730, Denmark (L.L.P., F.C.M., M.W.B., L.C.L., M.B.A.); Faculty of Health Sciences, University of Copenhagen, Copenhagen, Denmark (L.L.P., M.W.B., O.W.N., M.B., M.B.A.); Radiological Artificial Intelligence Testcenter, RAIT.dk, Capital Region of Denmark (L.L.P., F.C.M., M.W.B., M.B., M.B.A.); Departments of Radiology (M.W.B., M.B.) and Cardiology (O.W.N.), Bispebjerg and Frederiksberg Hospital, Copenhagen, Denmark; and Department of Radiology, Aarhus University Hospital, Aarhus, Denmark (F.R.)
| | - Olav W Nielsen
- From the Department of Radiology, Herlev and Gentofte Hospital, Borgmester Ib, Juuls vej 1 Herlev, Copenhagen 2730, Denmark (L.L.P., F.C.M., M.W.B., L.C.L., M.B.A.); Faculty of Health Sciences, University of Copenhagen, Copenhagen, Denmark (L.L.P., M.W.B., O.W.N., M.B., M.B.A.); Radiological Artificial Intelligence Testcenter, RAIT.dk, Capital Region of Denmark (L.L.P., F.C.M., M.W.B., M.B., M.B.A.); Departments of Radiology (M.W.B., M.B.) and Cardiology (O.W.N.), Bispebjerg and Frederiksberg Hospital, Copenhagen, Denmark; and Department of Radiology, Aarhus University Hospital, Aarhus, Denmark (F.R.)
| | - Mikael Boesen
- From the Department of Radiology, Herlev and Gentofte Hospital, Borgmester Ib, Juuls vej 1 Herlev, Copenhagen 2730, Denmark (L.L.P., F.C.M., M.W.B., L.C.L., M.B.A.); Faculty of Health Sciences, University of Copenhagen, Copenhagen, Denmark (L.L.P., M.W.B., O.W.N., M.B., M.B.A.); Radiological Artificial Intelligence Testcenter, RAIT.dk, Capital Region of Denmark (L.L.P., F.C.M., M.W.B., M.B., M.B.A.); Departments of Radiology (M.W.B., M.B.) and Cardiology (O.W.N.), Bispebjerg and Frederiksberg Hospital, Copenhagen, Denmark; and Department of Radiology, Aarhus University Hospital, Aarhus, Denmark (F.R.)
| | - Michael Brun Andersen
- From the Department of Radiology, Herlev and Gentofte Hospital, Borgmester Ib, Juuls vej 1 Herlev, Copenhagen 2730, Denmark (L.L.P., F.C.M., M.W.B., L.C.L., M.B.A.); Faculty of Health Sciences, University of Copenhagen, Copenhagen, Denmark (L.L.P., M.W.B., O.W.N., M.B., M.B.A.); Radiological Artificial Intelligence Testcenter, RAIT.dk, Capital Region of Denmark (L.L.P., F.C.M., M.W.B., M.B., M.B.A.); Departments of Radiology (M.W.B., M.B.) and Cardiology (O.W.N.), Bispebjerg and Frederiksberg Hospital, Copenhagen, Denmark; and Department of Radiology, Aarhus University Hospital, Aarhus, Denmark (F.R.)
| |
Collapse
|
9
|
Park SH, Sul AR, Ko Y, Jang HY, Lee JG. Radiologist's Guide to Evaluating Publications of Clinical Research on AI: How We Do It. Radiology 2023; 308:e230288. [PMID: 37750772 DOI: 10.1148/radiol.230288] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 09/27/2023]
Abstract
Literacy in research studies of artificial intelligence (AI) has become an important skill for radiologists. It is required to make a proper assessment of the validity, reproducibility, and clinical applicability of AI studies. However, AI studies are generally perceived to be more difficult for clinician readers to evaluate than traditional clinical research studies. This special report-as an effective, concise guide for readers-aims to assist clinical radiologists in critically evaluating different types of clinical research articles involving AI. It does not intend to be a comprehensive checklist or methodological summary for complete clinical evaluation of AI or a reporting guideline. Ten key items for readers to check are described, regarding study purpose, function and clinical context of AI, training data, data preprocessing, AI modeling techniques, test data, AI performance, helpfulness and value of AI, interpretability of AI, and code sharing. The important aspects of each item are explained for readers to consider when reading publications on AI clinical research. Evaluating each item can help radiologists assess the validity, reproducibility, and clinical applicability of clinical research articles involving AI.
Collapse
Affiliation(s)
- Seong Ho Park
- From the Department of Radiology and Research Institute of Radiology, Asan Medical Center, University of Ulsan College of Medicine, 88 Olympic-ro 43-gil, Songpa-gu, Seoul 05505, South Korea (S.H.P., Y.K., H.Y.J.); Division of Healthcare Research Outcomes Research, National Evidence-based Healthcare Collaborating Agency, Seoul, South Korea (A.R.S.); and Biomedical Engineering Research Center, Asan Institute for Life Sciences, University of Ulsan College of Medicine, Seoul, South Korea (J.G.L.)
| | - Ah-Ram Sul
- From the Department of Radiology and Research Institute of Radiology, Asan Medical Center, University of Ulsan College of Medicine, 88 Olympic-ro 43-gil, Songpa-gu, Seoul 05505, South Korea (S.H.P., Y.K., H.Y.J.); Division of Healthcare Research Outcomes Research, National Evidence-based Healthcare Collaborating Agency, Seoul, South Korea (A.R.S.); and Biomedical Engineering Research Center, Asan Institute for Life Sciences, University of Ulsan College of Medicine, Seoul, South Korea (J.G.L.)
| | - Yousun Ko
- From the Department of Radiology and Research Institute of Radiology, Asan Medical Center, University of Ulsan College of Medicine, 88 Olympic-ro 43-gil, Songpa-gu, Seoul 05505, South Korea (S.H.P., Y.K., H.Y.J.); Division of Healthcare Research Outcomes Research, National Evidence-based Healthcare Collaborating Agency, Seoul, South Korea (A.R.S.); and Biomedical Engineering Research Center, Asan Institute for Life Sciences, University of Ulsan College of Medicine, Seoul, South Korea (J.G.L.)
| | - Hye Young Jang
- From the Department of Radiology and Research Institute of Radiology, Asan Medical Center, University of Ulsan College of Medicine, 88 Olympic-ro 43-gil, Songpa-gu, Seoul 05505, South Korea (S.H.P., Y.K., H.Y.J.); Division of Healthcare Research Outcomes Research, National Evidence-based Healthcare Collaborating Agency, Seoul, South Korea (A.R.S.); and Biomedical Engineering Research Center, Asan Institute for Life Sciences, University of Ulsan College of Medicine, Seoul, South Korea (J.G.L.)
| | - June-Goo Lee
- From the Department of Radiology and Research Institute of Radiology, Asan Medical Center, University of Ulsan College of Medicine, 88 Olympic-ro 43-gil, Songpa-gu, Seoul 05505, South Korea (S.H.P., Y.K., H.Y.J.); Division of Healthcare Research Outcomes Research, National Evidence-based Healthcare Collaborating Agency, Seoul, South Korea (A.R.S.); and Biomedical Engineering Research Center, Asan Institute for Life Sciences, University of Ulsan College of Medicine, Seoul, South Korea (J.G.L.)
| |
Collapse
|