1
|
Takamatsu A, Ueno M, Yoshida K, Kobayashi T, Kobayashi S, Gabata T. Performance of artificial intelligence-based software for the automatic detection of lung lesions on chest radiographs of patients with suspected lung cancer. Jpn J Radiol 2024; 42:291-299. [PMID: 38032419 PMCID: PMC10899395 DOI: 10.1007/s11604-023-01503-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/22/2023] [Accepted: 10/11/2023] [Indexed: 12/01/2023]
Abstract
PURPOSE This study aimed to evaluate the performance of the commercially available artificial intelligence-based software CXR-AID for the automatic detection of pulmonary nodules on the chest radiographs of patients suspected of having lung cancer. MATERIALS AND METHODS This retrospective study included 399 patients with clinically suspected lung cancer who underwent CT and chest radiography within 1 month between June 2020 and May 2022. The candidate areas on chest radiographs identified by CXR-AID were categorized into target (properly detected areas) and non-target (improperly detected areas) areas. The non-target areas were further divided into non-target normal areas (false positives for normal structures) and non-target abnormal areas. The visibility score, characteristics and location of the nodules, presence of overlapping structures, and background lung score and presence of pulmonary disease were manually evaluated and compared between the nodules detected or undetected by CXR-AID. The probability indices calculated by CXR-AID were compared between the target and non-target areas. RESULTS Among the 450 nodules detected in 399 patients, 331 nodules detected in 313 patients were visible on chest radiographs during manual evaluation. CXR-AID detected 264 of these 331 nodules with a sensitivity of 0.80. The detection sensitivity increased significantly with the visibility score. No significant correlation was observed between the background lung score and sensitivity. The non-target area per image was 0.85, and the probability index of the non-target area was lower than that of the target area. The non-target normal area per image was 0.24. Larger and more solid nodules exhibited higher sensitivities, while nodules with overlapping structures demonstrated lower detection sensitivities. CONCLUSION The nodule detection sensitivity of CXR-AID on chest radiographs was 0.80, and the non-target and non-target normal areas per image were 0.85 and 0.24, respectively. Larger, solid nodules without overlapping structures were detected more readily by CXR-AID.
Collapse
Affiliation(s)
- Atsushi Takamatsu
- Department of Radiology, Kanazawa University Graduate School of Medical Sciences, 13-1 Takaramachi, Kanazawa, Ishikawa, 920-8641, Japan
| | - Midori Ueno
- Department of Radiology, Kanazawa University Graduate School of Medical Sciences, 13-1 Takaramachi, Kanazawa, Ishikawa, 920-8641, Japan
| | - Kotaro Yoshida
- Department of Radiology, Kanazawa University Graduate School of Medical Sciences, 13-1 Takaramachi, Kanazawa, Ishikawa, 920-8641, Japan.
| | - Takeshi Kobayashi
- Department of Diagnostic and Interventional Radiology, Ishikawa Prefectural Central Hospital, Kanazawa, Ishikawa, 920-8530, Japan
| | - Satoshi Kobayashi
- Department of Radiology, Kanazawa University Graduate School of Medical Sciences, 13-1 Takaramachi, Kanazawa, Ishikawa, 920-8641, Japan
| | - Toshifumi Gabata
- Department of Radiology, Kanazawa University Graduate School of Medical Sciences, 13-1 Takaramachi, Kanazawa, Ishikawa, 920-8641, Japan
| |
Collapse
|
2
|
Sabry AH, I. Dallal Bashi O, Nik Ali N, Mahmood Al Kubaisi Y. Lung disease recognition methods using audio-based analysis with machine learning. Heliyon 2024; 10:e26218. [PMID: 38420389 PMCID: PMC10900411 DOI: 10.1016/j.heliyon.2024.e26218] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/05/2022] [Revised: 12/11/2023] [Accepted: 02/08/2024] [Indexed: 03/02/2024] Open
Abstract
The use of computer-based automated approaches and improvements in lung sound recording techniques have made lung sound-based diagnostics even better and devoid of subjectivity errors. Using a computer to evaluate lung sound features more thoroughly with the use of analyzing changes in lung sound behavior, recording measurements, suppressing the presence of noise contaminations, and graphical representations are all made possible by computer-based lung sound analysis. This paper starts with a discussion of the need for this research area, providing an overview of the field and the motivations behind it. Following that, it details the survey methodology used in this work. It presents a discussion on the elements of sound-based lung disease classification using machine learning algorithms. This includes commonly prior considered datasets, feature extraction techniques, pre-processing methods, artifact removal methods, lung-heart sound separation, deep learning algorithms, and wavelet transform of lung audio signals. The study introduces studies that review lung screening including a summary table of these references and discusses the literature gaps in the existing studies. It is concluded that the use of sound-based machine learning in the classification of respiratory diseases has promising results. While we believe this material will prove valuable to physicians and researchers exploring sound-signal-based machine learning, large-scale investigations remain essential to solidify the findings and foster wider adoption within the medical community.
Collapse
Affiliation(s)
- Ahmad H. Sabry
- Department of Medical Instrumentation Engineering Techniques, Shatt Al-Arab University College, Basra, Iraq
| | - Omar I. Dallal Bashi
- Medical Technical Institute, Northern Technical University, 95G2+P34, Mosul, 41002, Iraq
| | - N.H. Nik Ali
- School of Electrical Engineering, College of Engineering, Universiti Teknologi MARA, 40450 Shah Alam, Selangor, Malaysia
| | - Yasir Mahmood Al Kubaisi
- Department of Sustainability Management, Dubai Academic Health Corporation, Dubai, 4545, United Arab Emirates
| |
Collapse
|
3
|
Yoon SH, Park S, Jang S, Kim J, Lee KW, Lee W, Lee S, Yun G, Lee KH. Use of artificial intelligence in triaging of chest radiographs to reduce radiologists' workload. Eur Radiol 2024; 34:1094-1103. [PMID: 37615766 DOI: 10.1007/s00330-023-10124-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/15/2022] [Revised: 06/22/2023] [Accepted: 07/03/2023] [Indexed: 08/25/2023]
Abstract
OBJECTIVES To evaluate whether deep learning-based detection algorithms (DLD)-based triaging can reduce outpatient chest radiograph interpretation workload while maintaining noninferior sensitivity. METHODS This retrospective study included patients who underwent initial chest radiography at the outpatient clinic between June 1 and June 30, 2017. Readers interpreted radiographs with/without a commercially available DLD that detects nine radiologic findings (atelectasis, calcification, cardiomegaly, consolidation, fibrosis, nodules, pneumothorax, pleural effusion, and pneumoperitoneum). The reading order was determined in a randomized, crossover manner. The radiographs were classified into negative and positive examinations. In a 50% worklist reduction scenario, radiographs were sorted in descending order of probability scores: the lower half was regarded as negative exams, while the remaining were read with DLD by radiologists. The primary analysis evaluated noninferiority in sensitivity between radiologists reading all radiographs and simulating a 50% worklist reduction, with the inferiority margin of 5%. The specificities were compared using McNemar's test. RESULTS The study included 1964 patients (median age [interquartile range], 55 years [40-67 years]). The sensitivity was 82.6% (195 of 236; 95% CI: 77.5%, 87.3%) when readers interpreted all chest radiographs without DLD and 83.5% (197 of 236; 95% CI: 78.8%, 88.1%) in the 50% worklist reduction scenario. The difference in sensitivity was 0.8% (95% CI: - 3.8%, 5.5%), establishing noninferiority of 50% worklist reduction (p = 0.01). The specificity increased from 86.7% (1498 of 1728) to 90.4% (1562 of 1728) (p < 0.001) with DLD-based triage. CONCLUSION Deep learning-based triaging may substantially reduce workload without lowering sensitivity while improving specificity. CLINICAL RELEVANCE STATEMENT Substantial workload reduction without lowering sensitivity was feasible using deep learning-based triaging of outpatient chest radiograph; however, the legal responsibility for incorrect diagnoses based on AI-standalone interpretation remains an issue that should be defined before clinical implementation. KEY POINTS • A 50% workload reduction simulation using deep learning-based detection algorithm maintained noninferior sensitivity while improving specificity. • The CT recommendation rate significantly decreased in the disease-negative patients, whereas it slightly increased in the disease-positive group without statistical significance. • In the exploratory analysis, the noninferiority of sensitivity was maintained until 70% of the workload was reduced; the difference in sensitivity was 0%.
Collapse
Affiliation(s)
- Sung Hyun Yoon
- Department of Radiology, Seoul National University Bundang Hospital, Seongnam, Korea
| | - Sunyoung Park
- Department of Radiology, Seoul National University Bundang Hospital, Seongnam, Korea
| | - Sowon Jang
- Department of Radiology, Seoul National University Bundang Hospital, Seongnam, Korea
| | - Junghoon Kim
- Department of Radiology, Seoul National University Bundang Hospital, Seongnam, Korea
| | - Kyung Won Lee
- Department of Radiology, Seoul National University Bundang Hospital, Seongnam, Korea
| | - Woojoo Lee
- Department of Public Health Science, Graduate School of Public Health, Seoul National University, Seoul, Korea
| | - Seungjae Lee
- Department of Applied Bioengineering, Graduate School of Convergence Science and Technology, Seoul National University, Seoul, Korea
| | - Gabin Yun
- Department of Radiology, University of Pittsburgh Medical Center, Pittsburgh, PA, USA
| | - Kyung Hee Lee
- Department of Radiology, Seoul National University Bundang Hospital, Seongnam, Korea.
- Department of Radiology, Seoul National University College of Medicine, Seoul National University Bundang Hospital, Seongnam, Korea.
| |
Collapse
|
4
|
Hamanaka R, Oda M. Can Artificial Intelligence Replace Humans for Detecting Lung Tumors on Radiographs? An Examination of Resected Malignant Lung Tumors. J Pers Med 2024; 14:164. [PMID: 38392597 PMCID: PMC10890665 DOI: 10.3390/jpm14020164] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/29/2023] [Revised: 01/18/2024] [Accepted: 01/29/2024] [Indexed: 02/24/2024] Open
Abstract
OBJECTIVE Although lung cancer screening trials have showed the efficacy of computed tomography to decrease mortality compared with chest radiography, the two are widely taken as different kinds of clinical practices. Artificial intelligence can improve outcomes by detecting lung tumors in chest radiographs. Currently, artificial intelligence is used as an aid for physicians to interpret radiograms, but with the future evolution of artificial intelligence, it may become a modality that replaces physicians. Therefore, in this study, we investigated the current situation of lung cancer diagnosis by artificial intelligence. METHODS In total, we recruited 174 consecutive patients with malignant pulmonary tumors who underwent surgery after chest radiography that was checked by artificial intelligence before surgery. Artificial intelligence diagnoses were performed using the medical image analysis software EIRL X-ray Lung Nodule version 1.12, (LPIXEL Inc., Tokyo, Japan). RESULTS The artificial intelligence determined pulmonary tumors in 90 cases (51.7% for all patients and 57.7% excluding 18 patients with adenocarcinoma in situ). There was no significant difference in the detection rate by the artificial intelligence among histological types. All eighteen cases of adenocarcinoma in situ were not detected by either the artificial intelligence or the physicians. In a univariate analysis, the artificial intelligence could detect cases with larger histopathological tumor size (p < 0.0001), larger histopathological invasion size (p < 0.0001), and higher maximum standardized uptake values of positron emission tomography-computed tomography (p < 0.0001). In a multivariate analysis, detection by AI was significantly higher in cases with a large histopathological invasive size (p = 0.006). In 156 cases excluding adenocarcinoma in situ, we examined the rate of artificial intelligence detection based on the tumor site. Tumors in the lower lung field area were less frequently detected (p = 0.019) and tumors in the middle lung field area were more frequently detected (p = 0.014) compared with tumors in the upper lung field area. CONCLUSIONS Our study showed that using artificial intelligence, the diagnosis of tumor-associated findings and the diagnosis of areas that overlap with anatomical structures is not satisfactory. While the current standing of artificial intelligence diagnostics is to assist physicians in making diagnoses, there is the possibility that artificial intelligence can substitute for humans in the future. However, artificial intelligence should be used in the future as an enhancement, to aid physicians in the role of a radiologist in the workflow.
Collapse
Affiliation(s)
- Rurika Hamanaka
- Department of Thoracic Surgery, Shin-Yurigaoka General Hospital, 255 Furusawa Asao-ku, Kawasaki 215-0026, Japan
| | - Makoto Oda
- Department of Thoracic Surgery, Shin-Yurigaoka General Hospital, 255 Furusawa Asao-ku, Kawasaki 215-0026, Japan
| |
Collapse
|
5
|
Kirshenboim Z, Gilat EK, Carl L, Bekker E, Tau N, Klug M, Konen E, Marom EM. Retrospectively assessing evaluation and management of artificial-intelligence detected nodules on uninterpreted chest radiographs in the era of radiologists shortage. Eur J Radiol 2024; 170:111241. [PMID: 38042019 DOI: 10.1016/j.ejrad.2023.111241] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/27/2023] [Revised: 11/17/2023] [Accepted: 11/26/2023] [Indexed: 12/04/2023]
Abstract
PURPOSE High volumes of chest radiographs (CXR) remain uninterpreted due to severe shortage of radiologists. These CXRs may be informally reported by non-radiologist physicians, or not reviewed at all. Artificial intelligence (AI) software can aid lung nodule detection. Our aim was to assess evaluation and management by non-radiologists of uninterpreted CXRs with AI detected nodules, compared to retrospective radiology reports. MATERIALS AND METHODS AI detected nodules on uninterpreted CXRs of adults, performed 30/6/2022-31/1/2023, were evaluated. Excluded were patients with known active malignancy and duplicate CXRs of the same patient. The electronic medical records (EMR) were reviewed, and the clinicians' notes on the CXR and AI detected nodule were documented. Dedicated thoracic radiologists retrospectively interpreted all CXRs, and similarly to the clinicians, they had access to the AI findings, prior imaging and EMR. The radiologists' interpretation served as the ground truth, and determined if the AI-detected nodule was a true lung nodule and if further workup was required. RESULTS A total of 683 patients met the inclusion criteria. The clinicians commented on 386 (56.5%) CXRs, identified true nodules on 113 CXRs (16.5%), incorrectly mentioned 31 (4.5%) false nodules as real nodules, and did not mention the AI detected nodule on 242 (35%) CXRs, of which 68 (10%) patients were retrospectively referred for further workup by the radiologist. For 297 patients (43.5%) there were no comments regarding the CXR in the EMR. Of these, 77 nodules (11.3%) were retrospectively referred for further workup by the radiologist. CONCLUSION AI software for lung nodule detection may be insufficient without a formal radiology report, and may lead to over diagnosis or misdiagnosis of nodules.
Collapse
Affiliation(s)
- Zehavit Kirshenboim
- Division of Diagnostic Radiology, Sheba Medical Center, Ramat Gan, Israel; Faculty of Medicine, Tel Aviv University, Israel.
| | - Efrat Keren Gilat
- Division of Diagnostic Radiology, Sheba Medical Center, Ramat Gan, Israel; Faculty of Medicine, Tel Aviv University, Israel.
| | - Lawrence Carl
- Division of Diagnostic Radiology, Sheba Medical Center, Ramat Gan, Israel; Faculty of Medicine, Tel Aviv University, Israel.
| | - Elena Bekker
- Division of Diagnostic Radiology, Sheba Medical Center, Ramat Gan, Israel; Faculty of Medicine, Tel Aviv University, Israel.
| | - Noam Tau
- Division of Diagnostic Radiology, Sheba Medical Center, Ramat Gan, Israel; Faculty of Medicine, Tel Aviv University, Israel.
| | - Maximiliano Klug
- Division of Diagnostic Radiology, Sheba Medical Center, Ramat Gan, Israel; Faculty of Medicine, Tel Aviv University, Israel.
| | - Eli Konen
- Division of Diagnostic Radiology, Sheba Medical Center, Ramat Gan, Israel; Faculty of Medicine, Tel Aviv University, Israel.
| | - Edith Michelle Marom
- Division of Diagnostic Radiology, Sheba Medical Center, Ramat Gan, Israel; Faculty of Medicine, Tel Aviv University, Israel.
| |
Collapse
|
6
|
Yoo H, Yoo RE, Choi SH, Hwang I, Lee JY, Seo JY, Koh SY, Choi KS, Kang KM, Yun TJ. Deep learning-based reconstruction for acceleration of lumbar spine MRI: a prospective comparison with standard MRI. Eur Radiol 2023; 33:8656-8668. [PMID: 37498386 DOI: 10.1007/s00330-023-09918-0] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/31/2023] [Revised: 05/28/2023] [Accepted: 05/31/2023] [Indexed: 07/28/2023]
Abstract
OBJECTIVE To compare the image quality and diagnostic performance between standard turbo spin-echo MRI and accelerated MRI with deep learning (DL)-based image reconstruction for degenerative lumbar spine diseases. MATERIALS AND METHODS Fifty patients who underwent both the standard and accelerated lumbar MRIs at a 1.5-T scanner for degenerative lumbar spine diseases were prospectively enrolled. DL reconstruction algorithm generated coarse (DL_coarse) and fine (DL_fine) images from the accelerated protocol. Image quality was quantitatively assessed in terms of signal-to-noise ratio (SNR) and contrast-to-noise ratio (CNR) and qualitatively assessed using five-point visual scoring systems. The sensitivity and specificity of four radiologists for the diagnosis of degenerative diseases in both protocols were compared. RESULTS The accelerated protocol reduced the average MRI acquisition time by 32.3% as compared to the standard protocol. As compared with standard images, DL_coarse and DL_fine showed significantly higher SNRs on T1-weighted images (T1WI; both p < .001) and T2-weighted images (T2WI; p = .002 and p < 0.001), higher CNRs on T1WI (both p < 0.001), and similar CNRs on T2WI (p = .49 and p = .27). The average radiologist assessment of overall image quality for DL_coarse and DL_fine was higher on sagittal T1WI (p = .04 and p < .001) and axial T2WI (p = .006 and p = .01) and similar on sagittal T2WI (p = .90 and p = .91). Both DL_coarse and DL_fine had better image quality of cauda equina and paraspinal muscles on axial T2WI (both p = .04 for cauda equina; p = .008 and p = .002 for paraspinal muscles). Differences in sensitivity and specificity for the detection of central canal stenosis and neural foraminal stenosis between standard and DL-reconstructed images were all statistically nonsignificant (p ≥ 0.05). CONCLUSION DL-based protocol reduced MRI acquisition time without degrading image quality and diagnostic performance of readers for degenerative lumbar spine diseases. CLINICAL RELEVANCE STATEMENT The deep learning (DL)-based reconstruction algorithm may be used to further accelerate spine MRI imaging to reduce patient discomfort and increase the cost efficiency of spine MRI imaging. KEY POINTS • By using deep learning (DL)-based reconstruction algorithm in combination with the accelerated MRI protocol, the average acquisition time was reduced by 32.3% as compared with the standard protocol. • DL-reconstructed images had similar or better quantitative/qualitative overall image quality and similar or better image quality for the delineation of most individual anatomical structures. • The average radiologist's sensitivity and specificity for the detection of major degenerative lumbar spine diseases, including central canal stenosis, neural foraminal stenosis, and disc herniation, on standard and DL-reconstructed images, were similar.
Collapse
Affiliation(s)
- Hyunsuk Yoo
- Department of Radiology, Seoul National University Hospital, Seoul National University College of Medicine, 101, Daehangno, Jongno-gu, Seoul, 03080, Republic of Korea
| | - Roh-Eul Yoo
- Department of Radiology, Seoul National University Hospital, Seoul National University College of Medicine, 101, Daehangno, Jongno-gu, Seoul, 03080, Republic of Korea.
| | - Seung Hong Choi
- Department of Radiology, Seoul National University Hospital, Seoul National University College of Medicine, 101, Daehangno, Jongno-gu, Seoul, 03080, Republic of Korea
- Center for Nanoparticle Research, Institute for Basic Science (IBS), Seoul, Republic of Korea
- School of Chemical and Biological Engineering, Seoul National University, Seoul, Republic of Korea
| | - Inpyeong Hwang
- Department of Radiology, Seoul National University Hospital, Seoul National University College of Medicine, 101, Daehangno, Jongno-gu, Seoul, 03080, Republic of Korea
| | - Ji Ye Lee
- Department of Radiology, Seoul National University Hospital, Seoul National University College of Medicine, 101, Daehangno, Jongno-gu, Seoul, 03080, Republic of Korea
| | - June Young Seo
- Department of Radiology, Seoul National University Hospital, Seoul National University College of Medicine, 101, Daehangno, Jongno-gu, Seoul, 03080, Republic of Korea
| | - Seok Young Koh
- Department of Radiology, Seoul National University Hospital, Seoul National University College of Medicine, 101, Daehangno, Jongno-gu, Seoul, 03080, Republic of Korea
| | - Kyu Sung Choi
- Department of Radiology, Seoul National University Hospital, Seoul National University College of Medicine, 101, Daehangno, Jongno-gu, Seoul, 03080, Republic of Korea
| | - Koung Mi Kang
- Department of Radiology, Seoul National University Hospital, Seoul National University College of Medicine, 101, Daehangno, Jongno-gu, Seoul, 03080, Republic of Korea
| | - Tae Jin Yun
- Department of Radiology, Seoul National University Hospital, Seoul National University College of Medicine, 101, Daehangno, Jongno-gu, Seoul, 03080, Republic of Korea
| |
Collapse
|
7
|
Hwang SH, Shin HJ, Kim EK, Lee EH, Lee M. Clinical outcomes and actual consequence of lung nodules incidentally detected on chest radiographs by artificial intelligence. Sci Rep 2023; 13:19732. [PMID: 37957283 PMCID: PMC10643548 DOI: 10.1038/s41598-023-47194-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/30/2023] [Accepted: 11/10/2023] [Indexed: 11/15/2023] Open
Abstract
This study evaluated how often clinically significant lung nodules were detected unexpectedly on chest radiographs (CXR) by artificial intelligence (AI)-based detection software, and whether co-existing findings can aid in differential diagnosis of lung nodules. Patients (> 18 years old) with AI-detected lung nodules at their first visit from March 2021 to February 2022, except for those in the pulmonology or thoracic surgery departments, were retrospectively included. Three radiologists categorized nodules into malignancy, active inflammation, post-inflammatory sequelae, or "other" groups. Characteristics of the nodule and abnormality scores of co-existing lung lesions were compared. Approximately 1% of patients (152/14,563) had unexpected lung nodules. Among 73 patients with follow-up exams, 69.9% had true positive nodules. Increased abnormality scores for nodules were significantly associated with malignancy (odds ratio [OR] 1.076, P = 0.001). Increased abnormality scores for consolidation (OR 1.033, P = 0.040) and pleural effusion (OR 1.025, P = 0.041) were significantly correlated with active inflammation-type nodules. Abnormality scores for fibrosis (OR 1.036, P = 0.013) and nodules (OR 0.940, P = 0.001) were significantly associated with post-inflammatory sequelae categorization. AI-based lesion-detection software of CXRs in daily practice can help identify clinically significant incidental lung nodules, and referring accompanying lung lesions may help classify the nodule.
Collapse
Affiliation(s)
- Shin Hye Hwang
- Department of Radiology, Research Institute of Radiological Science and Center for Clinical Imaging Data Science, Yongin Severance Hospital, Yonsei University College of Medicine, 363, Dongbaekjukjeon-daero, Giheung-gu, Yongin-si, Gyeonggi-do, 16995, Republic of Korea
| | - Hyun Joo Shin
- Department of Radiology, Research Institute of Radiological Science and Center for Clinical Imaging Data Science, Yongin Severance Hospital, Yonsei University College of Medicine, 363, Dongbaekjukjeon-daero, Giheung-gu, Yongin-si, Gyeonggi-do, 16995, Republic of Korea
- Center for Digital Health, Yongin Severance Hospital, Yonsei University College of Medicine, Yongin-si, Gyeonggi‑do, Republic of Korea
| | - Eun-Kyung Kim
- Department of Radiology, Research Institute of Radiological Science and Center for Clinical Imaging Data Science, Yongin Severance Hospital, Yonsei University College of Medicine, 363, Dongbaekjukjeon-daero, Giheung-gu, Yongin-si, Gyeonggi-do, 16995, Republic of Korea
| | - Eun Hye Lee
- Center for Digital Health, Yongin Severance Hospital, Yonsei University College of Medicine, Yongin-si, Gyeonggi‑do, Republic of Korea
- Division of Pulmonology, Allergy and Critical Care Medicine, Department of Internal Medicine, Yongin Severance Hospital, Yonsei University College of Medicine, Yongin-si, Gyeonggi-do, Republic of Korea
| | - Minwook Lee
- Department of Radiology, Research Institute of Radiological Science and Center for Clinical Imaging Data Science, Yongin Severance Hospital, Yonsei University College of Medicine, 363, Dongbaekjukjeon-daero, Giheung-gu, Yongin-si, Gyeonggi-do, 16995, Republic of Korea.
| |
Collapse
|
8
|
Bernstein MH, Atalay MK, Dibble EH, Maxwell AWP, Karam AR, Agarwal S, Ward RC, Healey TT, Baird GL. Can incorrect artificial intelligence (AI) results impact radiologists, and if so, what can we do about it? A multi-reader pilot study of lung cancer detection with chest radiography. Eur Radiol 2023; 33:8263-8269. [PMID: 37266657 PMCID: PMC10235827 DOI: 10.1007/s00330-023-09747-1] [Citation(s) in RCA: 14] [Impact Index Per Article: 14.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/09/2022] [Revised: 04/16/2023] [Accepted: 04/18/2023] [Indexed: 06/03/2023]
Abstract
OBJECTIVE To examine whether incorrect AI results impact radiologist performance, and if so, whether human factors can be optimized to reduce error. METHODS Multi-reader design, 6 radiologists interpreted 90 identical chest radiographs (follow-up CT needed: yes/no) on four occasions (09/20-01/22). No AI result was provided for session 1. Sham AI results were provided for sessions 2-4, and AI for 12 cases were manipulated to be incorrect (8 false positives (FP), 4 false negatives (FN)) (0.87 ROC-AUC). In the Delete AI (No Box) condition, radiologists were told AI results would not be saved for the evaluation. In Keep AI (No Box) and Keep AI (Box), radiologists were told results would be saved. In Keep AI (Box), the ostensible AI program visually outlined the region of suspicion. AI results were constant between conditions. RESULTS Relative to the No AI condition (FN = 2.7%, FP = 51.4%), FN and FPs were higher in the Keep AI (No Box) (FN = 33.0%, FP = 86.0%), Delete AI (No Box) (FN = 26.7%, FP = 80.5%), and Keep AI (Box) (FN = to 20.7%, FP = 80.5%) conditions (all ps < 0.05). FNs were higher in the Keep AI (No Box) condition (33.0%) than in the Keep AI (Box) condition (20.7%) (p = 0.04). FPs were higher in the Keep AI (No Box) (86.0%) condition than in the Delete AI (No Box) condition (80.5%) (p = 0.03). CONCLUSION Incorrect AI causes radiologists to make incorrect follow-up decisions when they were correct without AI. This effect is mitigated when radiologists believe AI will be deleted from the patient's file or a box is provided around the region of interest. CLINICAL RELEVANCE STATEMENT When AI is wrong, radiologists make more errors than they would have without AI. Based on human factors psychology, our manuscript provides evidence for two AI implementation strategies that reduce the deleterious effects of incorrect AI. KEY POINTS • When AI provided incorrect results, false negative and false positive rates among the radiologists increased. • False positives decreased when AI results were deleted, versus kept, in the patient's record. • False negatives and false positives decreased when AI visually outlined the region of suspicion.
Collapse
Affiliation(s)
- Michael H Bernstein
- Department of Diagnostic Imaging, Warren Alpert Medical School of Brown University, Providence, RI, USA.
- Rhode Island Hospital, Providence, RI, USA.
- Brown Radiology Human Factors Laboratory, Providence, RI, USA.
| | - Michael K Atalay
- Department of Diagnostic Imaging, Warren Alpert Medical School of Brown University, Providence, RI, USA
- Brown Radiology Human Factors Laboratory, Providence, RI, USA
| | - Elizabeth H Dibble
- Department of Diagnostic Imaging, Warren Alpert Medical School of Brown University, Providence, RI, USA
| | - Aaron W P Maxwell
- Department of Diagnostic Imaging, Warren Alpert Medical School of Brown University, Providence, RI, USA
- Brown Radiology Human Factors Laboratory, Providence, RI, USA
| | - Adib R Karam
- Department of Diagnostic Imaging, Warren Alpert Medical School of Brown University, Providence, RI, USA
| | - Saurabh Agarwal
- Department of Diagnostic Imaging, Warren Alpert Medical School of Brown University, Providence, RI, USA
| | - Robert C Ward
- Department of Diagnostic Imaging, Warren Alpert Medical School of Brown University, Providence, RI, USA
| | - Terrance T Healey
- Department of Diagnostic Imaging, Warren Alpert Medical School of Brown University, Providence, RI, USA
| | - Grayson L Baird
- Department of Diagnostic Imaging, Warren Alpert Medical School of Brown University, Providence, RI, USA
- Rhode Island Hospital, Providence, RI, USA
- Brown Radiology Human Factors Laboratory, Providence, RI, USA
| |
Collapse
|
9
|
You S, Park JH, Park B, Shin HB, Ha T, Yun JS, Park KJ, Jung Y, Kim YN, Kim M, Sun JS. The diagnostic performance and clinical value of deep learning-based nodule detection system concerning influence of location of pulmonary nodule. Insights Imaging 2023; 14:149. [PMID: 37726452 PMCID: PMC10509107 DOI: 10.1186/s13244-023-01497-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/16/2022] [Accepted: 08/08/2023] [Indexed: 09/21/2023] Open
Abstract
BACKGROUND The deep learning-based nodule detection (DLD) system improves nodule detection performance of observers on chest radiographs (CXRs). However, its performance in different pulmonary nodule (PN) locations remains unknown. METHODS We divided the CXR intrathoracic region into non-danger zone (NDZ) and danger zone (DZ). The DZ included the lung apices, paramediastinal areas, and retrodiaphragmatic areas, where nodules could be missed. We used a dataset of 300 CXRs (100 normal and 200 abnormal images with 216 PNs [107 NDZ and 109 DZ nodules]). Eight observers (two thoracic radiologists [TRs], two non-thoracic radiologists [NTRs], and four radiology residents [RRs]) interpreted each radiograph with and without the DLD system. The metric of lesion localization fraction (LLF; the number of correctly localized lesions divided by the total number of true lesions) was used to evaluate the diagnostic performance according to the nodule location. RESULTS The DLD system demonstrated a lower LLF for the detection of DZ nodules (64.2) than that of NDZ nodules (83.2, p = 0.008). For DZ nodule detection, the LLF of the DLD system (64.2) was lower than that of TRs (81.7, p < 0.001), which was comparable to that of NTRs (56.4, p = 0.531) and RRs (56.7, p = 0.459). Nonetheless, the LLF of RRs significantly improved from 56.7 to 65.6 using the DLD system (p = 0.021) for DZ nodule detection. CONCLUSION The performance of the DLD system was lower in the detection of DZ nodules compared to that of NDZ nodules. Nonetheless, RR performance in detecting DZ nodules improved upon using the DLD system. CRITICAL RELEVANCE STATEMENT Despite the deep learning-based nodule detection system's limitations in detecting danger zone nodules, it proves beneficial for less-experienced observers by providing valuable assistance in identifying these nodules, thereby advancing nodule detection in clinical practice. KEY POINTS • The deep learning-based nodule detection (DLD) system can improve the diagnostic performance of observers in nodule detection. • The DLD system shows poor diagnostic performance in detecting danger zone nodules. • For less-experienced observers, the DLD system is helpful in detecting danger zone nodules.
Collapse
Affiliation(s)
- Seulgi You
- Department of Radiology, Ajou University School of Medicine, Suwon, Republic of Korea
| | - Ji Hyun Park
- Office of Biostatistics, Ajou Research Institute for Innovative Medicine, Ajou University Medical Center, Suwon, Republic of Korea
| | - Bumhee Park
- Office of Biostatistics, Ajou Research Institute for Innovative Medicine, Ajou University Medical Center, Suwon, Republic of Korea
- Departments of Biomedical Informatics, Ajou Research Institute for Innovative Medicine, Ajou University School of Medicine, Suwon, Republic of Korea
| | - Han-Bit Shin
- Office of Biostatistics, Ajou Research Institute for Innovative Medicine, Ajou University Medical Center, Suwon, Republic of Korea
| | - Taeyang Ha
- Department of Radiology, Ajou University School of Medicine, Suwon, Republic of Korea
| | - Jae Sung Yun
- Department of Radiology, Ajou University School of Medicine, Suwon, Republic of Korea
| | - Kyoung Joo Park
- Department of Radiology, Ajou University School of Medicine, Suwon, Republic of Korea
| | - Yongjun Jung
- Department of Radiology, Ajou University School of Medicine, Suwon, Republic of Korea
| | - You Na Kim
- Department of Radiology, Ajou University School of Medicine, Suwon, Republic of Korea
| | - Minji Kim
- Department of Radiology, Ajou University School of Medicine, Suwon, Republic of Korea
| | - Joo Sung Sun
- Department of Radiology, Ajou University School of Medicine, Suwon, Republic of Korea.
| |
Collapse
|
10
|
Ueno M, Yoshida K, Takamatsu A, Kobayashi T, Aoki T, Gabata T. Deep learning-based automatic detection for pulmonary nodules on chest radiographs: The relationship with background lung condition, nodule characteristics, and location. Eur J Radiol 2023; 166:111002. [PMID: 37499478 DOI: 10.1016/j.ejrad.2023.111002] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/03/2023] [Revised: 07/11/2023] [Accepted: 07/20/2023] [Indexed: 07/29/2023]
Abstract
PURPOSE Computer-aided diagnosis (CAD), which assists in the interpretation of chest radiographs, is becoming common. However, few studies have evaluated the benefits and pitfalls of CAD in the real world. This study aimed to evaluate the independent performance of commercially available deep learning-based automatic detection (DLAD) software, EIRL Chest X-ray Lung Nodule, in a cohort that included patients with background pulmonary abnormalities often encountered in clinical situations. METHODS Patients with clinically suspected lung cancer for whom chest radiography was performed within a month before or after CT scan between June 2020 and May 2022 in our institution were enrolled. The reference standard was created using a bounding box annotated by two radiologists with reference to the CT. The visibility score, characteristics, location of the pulmonary nodules, presence of overlapping structures or pulmonary disease, and background lung score were manually determined. RESULTS We included 388 patients. The DLAD software detected 222 of the 322 nodules visible on manual evaluation, with a sensitivity of 0.689 and a false-positive rate of 0.168. The detectability of the DLAD software was significantly lower for small and subsolid and nodules with overlapping structures. The visibility score and sensitivity of detection by the DLAD software were positively correlated. The relationship between the background lung score and detection by the DLAD software was unclear. CONCLUSION The standalone performance of DLAD in detecting pulmonary nodules exhibited a sensitivity of 0.689 and a false-positive rate of 0.168. Understanding the characteristics of DLAD is crucial when interpreting chest radiographs with the assistance of the DLAD.
Collapse
Affiliation(s)
- Midori Ueno
- Department of Radiology, Kanazawa University Graduate School of Medical Science, 1-13 Takaramachi, Kanazawa City, Ishikawa Prefecture 920-8641, Japan; Department of Radiology, University of Occupational and Environmental Health School of Medicine, 1-1 Iseigaoka, Kitakyushu City, Fukuoka Prefecture 807-8555, Japan.
| | - Kotaro Yoshida
- Department of Radiology, Kanazawa University Graduate School of Medical Science, 1-13 Takaramachi, Kanazawa City, Ishikawa Prefecture 920-8641, Japan.
| | - Atsushi Takamatsu
- Department of Radiology, Kanazawa University Graduate School of Medical Science, 1-13 Takaramachi, Kanazawa City, Ishikawa Prefecture 920-8641, Japan.
| | - Takeshi Kobayashi
- Department of Diagnostic and Interventional Radiology, Ishikawa Prefectural Central Hospital, 1-2, Kuratsuki-Higashi, Kanazawa City, Ishikawa Prefecture 920-8530, Japan.
| | - Takatoshi Aoki
- Department of Radiology, University of Occupational and Environmental Health School of Medicine, 1-1 Iseigaoka, Kitakyushu City, Fukuoka Prefecture 807-8555, Japan.
| | - Toshifumi Gabata
- Department of Radiology, Kanazawa University Graduate School of Medical Science, 1-13 Takaramachi, Kanazawa City, Ishikawa Prefecture 920-8641, Japan.
| |
Collapse
|
11
|
Liu Y, Feng Z, Fan Z, Zhang Y, Li C, Liu X, Duan H, Cui X, Zhang L, Sheng C, Yang L, Gao Y, Wang X, Zhang Q, Lyu Z, Song F, Huang Y, Song F. Associations of chest X-ray trajectories, smoking, and the risk of lung cancer in two population-based cohort studies. Front Oncol 2023; 13:1203320. [PMID: 37534249 PMCID: PMC10392917 DOI: 10.3389/fonc.2023.1203320] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/10/2023] [Accepted: 06/30/2023] [Indexed: 08/04/2023] Open
Abstract
Objectives Despite the increasing use of computed tomography (CT), chest X-ray (CXR) remains the first-line investigation for suspected lung cancer (LC) in primary care. However, the associations of CXR trajectories, smoking and LC risk remain unknown. Methods A total of 52,486 participants from the PLCO and 22,194 participants from the NLST were included. The associations of CXR trajectories with LC risk were evaluated with multivariable COX regression models and pooled with meta-analyses. Further analyses were conducted to explore the stratified associations by smoking status and the factors associated with progression and regression in CXR. Results Compared to stable negative CXR (CXRSN), HRs (95%CIs) of LC incidence were 2.88(1.50-5.52), 3.86(2.03-7.35), and 1.08(0.80-1.46) for gain of positive CXR (CXRGP), stable positive CXR (CXRSP), and loss of positive CXR (CXRLP), while the risk of LC mortality were 1.58(1.33-1.87), 2.56(1.53-4.29), and 1.05(0.89-1.25). Similar trends were observed across different smoking status. However, LC risk with CXRGP overweighed that with CXRSP among ever smokers [2.95(2.25-3.88) vs. 2.59(1.33-5.02)] and current smokers [2.33(1.70-3.18) vs. 2.26(1.06-4.83)]. Moreover, compared to CXRSN among never smokers, even no progression in CXR, the HRs(95%CIs) of LC incidence were 7.39(5.60-9.75) and 31.45(23.58-41.95) for ever and current smokers, while risks of LC mortality were 6.30(5.07-7.81) and 27.17(21.65-34.11). If participants gained positive CXR, LC incidence risk significantly climbed to 22.04(15.37-31.60) and 71.97(48.82-106.09) for ever and current smokers, while LC mortality risk climbed to 11.90(8.58-16.50) and 38.92(27.04-56.02). CXRLP was associated with decreased LC risk. However, even smokers lost their positive CXR, and the increased risks of LC incidence and mortality did not decrease to non-significant level. Additionally, smoking was significantly associated with increased risk of CXRGP but not CXRLP. Conclusion LC risk differed across CXR trajectories and would be modified by smoking status. Comprehensive intervention incorporating CXR trajectories and smoking status should be recommended to reduce LC risk.
Collapse
Affiliation(s)
- Ya Liu
- Department of Epidemiology and Biostatistics, National Clinical Research Center for Cancer, Key Laboratory of Molecular Cancer Epidemiology of Tianjin, Key Laboratory of Cancer Prevention and Therapy of Tianjin, Tianjin’s Clinical Research Center for Cancer, Tianjin Medical University Cancer Institute and Hospital, Tianjin Medical University, Tianjin, China
| | - Zhuowei Feng
- Department of Epidemiology and Biostatistics, National Clinical Research Center for Cancer, Key Laboratory of Molecular Cancer Epidemiology of Tianjin, Key Laboratory of Cancer Prevention and Therapy of Tianjin, Tianjin’s Clinical Research Center for Cancer, Tianjin Medical University Cancer Institute and Hospital, Tianjin Medical University, Tianjin, China
| | - Zeyu Fan
- Department of Epidemiology and Biostatistics, National Clinical Research Center for Cancer, Key Laboratory of Molecular Cancer Epidemiology of Tianjin, Key Laboratory of Cancer Prevention and Therapy of Tianjin, Tianjin’s Clinical Research Center for Cancer, Tianjin Medical University Cancer Institute and Hospital, Tianjin Medical University, Tianjin, China
| | - Yu Zhang
- Department of Epidemiology and Biostatistics, National Clinical Research Center for Cancer, Key Laboratory of Molecular Cancer Epidemiology of Tianjin, Key Laboratory of Cancer Prevention and Therapy of Tianjin, Tianjin’s Clinical Research Center for Cancer, Tianjin Medical University Cancer Institute and Hospital, Tianjin Medical University, Tianjin, China
| | - Chenyang Li
- Department of Epidemiology and Biostatistics, National Clinical Research Center for Cancer, Key Laboratory of Molecular Cancer Epidemiology of Tianjin, Key Laboratory of Cancer Prevention and Therapy of Tianjin, Tianjin’s Clinical Research Center for Cancer, Tianjin Medical University Cancer Institute and Hospital, Tianjin Medical University, Tianjin, China
| | - Xiaomin Liu
- Department of Epidemiology and Biostatistics, National Clinical Research Center for Cancer, Key Laboratory of Molecular Cancer Epidemiology of Tianjin, Key Laboratory of Cancer Prevention and Therapy of Tianjin, Tianjin’s Clinical Research Center for Cancer, Tianjin Medical University Cancer Institute and Hospital, Tianjin Medical University, Tianjin, China
| | - Hongyuan Duan
- Department of Epidemiology and Biostatistics, National Clinical Research Center for Cancer, Key Laboratory of Molecular Cancer Epidemiology of Tianjin, Key Laboratory of Cancer Prevention and Therapy of Tianjin, Tianjin’s Clinical Research Center for Cancer, Tianjin Medical University Cancer Institute and Hospital, Tianjin Medical University, Tianjin, China
| | - Xiaonan Cui
- Department of Radiology, National Clinical Research Centre for Cancer, Key Laboratory of Cancer Prevention and Therapy of Tianjin, Tianjin Medical University Cancer Institute and Hospital, Tianjin, China
| | - Liwen Zhang
- Hebei Key Laboratory of Environment and Human Health, Department of Epidemiology and Statistics, School of Public Health, Hebei Medical University, Shijiazhuang, China
| | - Chao Sheng
- Department of Epidemiology and Biostatistics, National Clinical Research Center for Cancer, Key Laboratory of Molecular Cancer Epidemiology of Tianjin, Key Laboratory of Cancer Prevention and Therapy of Tianjin, Tianjin’s Clinical Research Center for Cancer, Tianjin Medical University Cancer Institute and Hospital, Tianjin Medical University, Tianjin, China
| | - Lei Yang
- Key Laboratory of Carcinogenesis and Translational Research (Ministry of Education), Beijing Office for Cancer Prevention and Control, Peking University Cancer Hospital and Institute, Beijing, China
| | - Ying Gao
- Health Management Center, Tianjin Medical University General Hospital, Tianjin, China
| | - Xing Wang
- Health Management Center, Tianjin Medical University General Hospital, Tianjin, China
| | - Qing Zhang
- Health Management Center, Tianjin Medical University General Hospital, Tianjin, China
| | - Zhangyan Lyu
- Department of Epidemiology and Biostatistics, National Clinical Research Center for Cancer, Key Laboratory of Molecular Cancer Epidemiology of Tianjin, Key Laboratory of Cancer Prevention and Therapy of Tianjin, Tianjin’s Clinical Research Center for Cancer, Tianjin Medical University Cancer Institute and Hospital, Tianjin Medical University, Tianjin, China
| | - Fangfang Song
- Department of Epidemiology and Biostatistics, National Clinical Research Center for Cancer, Key Laboratory of Molecular Cancer Epidemiology of Tianjin, Key Laboratory of Cancer Prevention and Therapy of Tianjin, Tianjin’s Clinical Research Center for Cancer, Tianjin Medical University Cancer Institute and Hospital, Tianjin Medical University, Tianjin, China
| | - Yubei Huang
- Department of Epidemiology and Biostatistics, National Clinical Research Center for Cancer, Key Laboratory of Molecular Cancer Epidemiology of Tianjin, Key Laboratory of Cancer Prevention and Therapy of Tianjin, Tianjin’s Clinical Research Center for Cancer, Tianjin Medical University Cancer Institute and Hospital, Tianjin Medical University, Tianjin, China
| | - Fengju Song
- Department of Epidemiology and Biostatistics, National Clinical Research Center for Cancer, Key Laboratory of Molecular Cancer Epidemiology of Tianjin, Key Laboratory of Cancer Prevention and Therapy of Tianjin, Tianjin’s Clinical Research Center for Cancer, Tianjin Medical University Cancer Institute and Hospital, Tianjin Medical University, Tianjin, China
| |
Collapse
|
12
|
Lim CY, Cha YK, Chung MJ, Park S, Park S, Woo JH, Kim JH. Estimating the Volume of Nodules and Masses on Serial Chest Radiography Using a Deep-Learning-Based Automatic Detection Algorithm: A Preliminary Study. Diagnostics (Basel) 2023; 13:2060. [PMID: 37370955 DOI: 10.3390/diagnostics13122060] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/12/2023] [Revised: 06/09/2023] [Accepted: 06/10/2023] [Indexed: 06/29/2023] Open
Abstract
BACKGROUND The purpose of this study was to assess the volume of the pulmonary nodules and masses on serial chest X-rays (CXRs) from deep-learning-based automatic detection algorithm (DLAD)-based parameters. METHODS In a retrospective single-institutional study, 72 patients, who obtained serial CXRs (n = 147) for pulmonary nodules or masses with corresponding chest CT images as the reference standards, were included. A pre-trained DLAD based on a convolutional neural network was developed to detect and localize nodules using 13,710 radiographs and to calculate a localization map and the derived parameters (e.g., the area and mean probability value of pulmonary nodules) for each CXR, including serial follow-ups. For validation, reference 3D CT volumes were measured semi-automatically. Volume prediction models for pulmonary nodules were established through univariable or multivariable, and linear or non-linear regression analyses with the parameters. A polynomial regression analysis was performed as a method of a non-linear regression model. RESULTS Of the 147 CXRs and 208 nodules of 72 patients, the mean volume of nodules or masses was measured as 9.37 ± 11.69 cm3 (mean ± standard deviation). The area and CT volume demonstrated a linear correlation of moderate strength (i.e., R = 0.58, RMSE: 9449.9 mm3 m3 in a linear regression analysis). The area and mean probability values exhibited a strong linear correlation (R = 0.73). The volume prediction performance based on a multivariable regression model was best with a mean probability and unit-adjusted area (i.e. , RMSE 7975.6 mm3, the smallest among the other variable parameters). CONCLUSIONS The prediction model with the area and the mean probability based on the DLAD showed a rather accurate quantitative estimation of pulmonary nodule or mass volume and the change in serial CXRs.
Collapse
Affiliation(s)
- Chae Young Lim
- Department of Radiology, Samsung Medical Center, Sungkyunkwan University School of Medicine, Seoul 06351, Republic of Korea
| | - Yoon Ki Cha
- Department of Radiology, Samsung Medical Center, Sungkyunkwan University School of Medicine, Seoul 06351, Republic of Korea
| | - Myung Jin Chung
- Department of Radiology, Samsung Medical Center, Sungkyunkwan University School of Medicine, Seoul 06351, Republic of Korea
| | - Subin Park
- Department of Health Sciences and Technology, SAIHST, Sungkyunkwan University, Suwon 16419, Republic of Korea
| | - Soyoung Park
- Department of Health Sciences and Technology, SAIHST, Sungkyunkwan University, Suwon 16419, Republic of Korea
| | - Jung Han Woo
- Department of Radiology, Samsung Medical Center, Sungkyunkwan University School of Medicine, Seoul 06351, Republic of Korea
| | - Jong Hee Kim
- Department of Radiology, Samsung Medical Center, Sungkyunkwan University School of Medicine, Seoul 06351, Republic of Korea
| |
Collapse
|
13
|
Nuutinen M, Leskelä RL. Systematic review of the performance evaluation of clinicians with or without the aid of machine learning clinical decision support system. HEALTH AND TECHNOLOGY 2023; 13:1-14. [PMID: 37363342 PMCID: PMC10262137 DOI: 10.1007/s12553-023-00763-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/13/2023] [Accepted: 06/01/2023] [Indexed: 06/28/2023]
Abstract
Background For the adoption of machine learning clinical decision support systems (ML-CDSS) it is critical to understand the performance aid of the ML-CDSS. However, it is not trivial, how the performance aid should be evaluated. To design reliable performance evaluation study, both the knowledge from the practical framework of experimental study design and the understanding of domain specific design factors are required. Objective The aim of this review study was to form a practical framework and identify key design factors for experimental design in evaluating the performance of clinicians with or without the aid of ML-CDSS. Methods The study was based on published ML-CDSS performance evaluation studies. We systematically searched articles published between January 2016 and December 2022. From the articles we collected a set of design factors. Only the articles comparing the performance of clinicians with or without the aid of ML-CDSS using experimental study methods were considered. Results The identified key design factors for the practical framework of ML-CDSS experimental study design were performance measures, user interface, ground truth data and the selection of samples and participants. In addition, we identified the importance of randomization, crossover design and training and practice rounds. Previous studies had shortcomings in the rationale and documentation of choices regarding the number of participants and the duration of the experiment. Conclusion The design factors of ML-CDSS experimental study are interdependent and all factors must be considered in individual choices. Supplementary Information The online version contains supplementary material available at 10.1007/s12553-023-00763-1.
Collapse
Affiliation(s)
- Mikko Nuutinen
- Nordic Healthcare Group, Helsinki, Finland
- Haartman Institute, University of Helsinki, Helsinki, Finland
| | | |
Collapse
|
14
|
Sfayyih AH, Sulaiman N, Sabry AH. A review on lung disease recognition by acoustic signal analysis with deep learning networks. JOURNAL OF BIG DATA 2023; 10:101. [PMID: 37333945 PMCID: PMC10259357 DOI: 10.1186/s40537-023-00762-z] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 11/08/2022] [Accepted: 05/08/2023] [Indexed: 06/20/2023]
Abstract
Recently, assistive explanations for difficulties in the health check area have been made viable thanks in considerable portion to technologies like deep learning and machine learning. Using auditory analysis and medical imaging, they also increase the predictive accuracy for prompt and early disease detection. Medical professionals are thankful for such technological support since it helps them manage further patients because of the shortage of skilled human resources. In addition to serious illnesses like lung cancer and respiratory diseases, the plurality of breathing difficulties is gradually rising and endangering society. Because early prediction and immediate treatment are crucial for respiratory disorders, chest X-rays and respiratory sound audio are proving to be quite helpful together. Compared to related review studies on lung disease classification/detection using deep learning algorithms, only two review studies based on signal analysis for lung disease diagnosis have been conducted in 2011 and 2018. This work provides a review of lung disease recognition with acoustic signal analysis with deep learning networks. We anticipate that physicians and researchers working with sound-signal-based machine learning will find this material beneficial.
Collapse
Affiliation(s)
- Alyaa Hamel Sfayyih
- Department of Electrical and Electronic Engineering, Faculty of Engineering, Universiti Putra Malaysia, 43400 Serdang, Malaysia
| | - Nasri Sulaiman
- Department of Electrical and Electronic Engineering, Faculty of Engineering, Universiti Putra Malaysia, 43400 Serdang, Malaysia
| | - Ahmad H. Sabry
- Department of Computer Engineering, Al-Nahrain University, Al Jadriyah Bridge, 64074 Baghdad, Iraq
| |
Collapse
|
15
|
Sfayyih AH, Sabry AH, Jameel SM, Sulaiman N, Raafat SM, Humaidi AJ, Kubaiaisi YMA. Acoustic-Based Deep Learning Architectures for Lung Disease Diagnosis: A Comprehensive Overview. Diagnostics (Basel) 2023; 13:diagnostics13101748. [PMID: 37238233 DOI: 10.3390/diagnostics13101748] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/28/2023] [Revised: 05/04/2023] [Accepted: 05/11/2023] [Indexed: 05/28/2023] Open
Abstract
Lung auscultation has long been used as a valuable medical tool to assess respiratory health and has gotten a lot of attention in recent years, notably following the coronavirus epidemic. Lung auscultation is used to assess a patient's respiratory role. Modern technological progress has guided the growth of computer-based respiratory speech investigation, a valuable tool for detecting lung abnormalities and diseases. Several recent studies have reviewed this important area, but none are specific to lung sound-based analysis with deep-learning architectures from one side and the provided information was not sufficient for a good understanding of these techniques. This paper gives a complete review of prior deep-learning-based architecture lung sound analysis. Deep-learning-based respiratory sound analysis articles are found in different databases including the Plos, ACM Digital Libraries, Elsevier, PubMed, MDPI, Springer, and IEEE. More than 160 publications were extracted and submitted for assessment. This paper discusses different trends in pathology/lung sound, the common features for classifying lung sounds, several considered datasets, classification methods, signal processing techniques, and some statistical information based on previous study findings. Finally, the assessment concludes with a discussion of potential future improvements and recommendations.
Collapse
Affiliation(s)
- Alyaa Hamel Sfayyih
- Department of Electrical and Electronic Engineering, Faculty of Engineering, University Putra Malaysia, Serdang 43400, Malaysia
| | - Ahmad H Sabry
- Department of Computer Engineering, Al-Nahrain University Al Jadriyah Bridge, Baghdad 64074, Iraq
| | | | - Nasri Sulaiman
- Department of Electrical and Electronic Engineering, Faculty of Engineering, University Putra Malaysia, Serdang 43400, Malaysia
| | - Safanah Mudheher Raafat
- Department of Control and Systems Engineering, University of Technology, Baghdad 10011, Iraq
| | - Amjad J Humaidi
- Department of Control and Systems Engineering, University of Technology, Baghdad 10011, Iraq
| | - Yasir Mahmood Al Kubaiaisi
- Department of Sustainability Management, Dubai Academic Health Corporation, Dubai 4545, United Arab Emirates
| |
Collapse
|
16
|
Dikici E, Nguyen XV, Takacs N, Prevedello LM. Prediction of model generalizability for unseen data: Methodology and case study in brain metastases detection in T1-Weighted contrast-enhanced 3D MRI. Comput Biol Med 2023; 159:106901. [PMID: 37068317 DOI: 10.1016/j.compbiomed.2023.106901] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/22/2022] [Revised: 03/08/2023] [Accepted: 04/09/2023] [Indexed: 04/19/2023]
Abstract
BACKGROUND AND PURPOSE A medical AI system's generalizability describes the continuity of its performance acquired from varying geographic, historical, and methodologic settings. Previous literature on this topic has mostly focused on "how" to achieve high generalizability (e.g., via larger datasets, transfer learning, data augmentation, model regularization schemes), with limited success. Instead, we aim to understand "when" the generalizability is achieved: Our study presents a medical AI system that could estimate its generalizability status for unseen data on-the-fly. MATERIALS AND METHODS We introduce a latent space mapping (LSM) approach utilizing Fréchet distance loss to force the underlying training data distribution into a multivariate normal distribution. During the deployment, a given test data's LSM distribution is processed to detect its deviation from the forced distribution; hence, the AI system could predict its generalizability status for any previously unseen data set. If low model generalizability is detected, then the user is informed by a warning message integrated into a sample deployment workflow. While the approach is applicable for most classification deep neural networks (DNNs), we demonstrate its application to a brain metastases (BM) detector for T1-weighted contrast-enhanced (T1c) 3D MRI. The BM detection model was trained using 175 T1c studies acquired internally (from the authors' institution) and tested using (1) 42 internally acquired exams and (2) 72 externally acquired exams from the publicly distributed Brain Mets dataset provided by the Stanford University School of Medicine. Generalizability scores, false positive (FP) rates, and sensitivities of the BM detector were computed for the test datasets. RESULTS AND CONCLUSION The model predicted its generalizability to be low for 31% of the testing data (i.e., two of the internally and 33 of the externally acquired exams), where it produced (1) ∼13.5 false positives (FPs) at 76.1% BM detection sensitivity for the low and (2) ∼10.5 FPs at 89.2% BM detection sensitivity for the high generalizability groups respectively. These results suggest that the proposed formulation enables a model to predict its generalizability for unseen data.
Collapse
Affiliation(s)
- Engin Dikici
- The Ohio State University, College of Medicine, Department of Radiology, Columbus, OH, 43210, USA.
| | - Xuan V Nguyen
- The Ohio State University, College of Medicine, Department of Radiology, Columbus, OH, 43210, USA
| | - Noah Takacs
- The Ohio State University, College of Medicine, Department of Radiology, Columbus, OH, 43210, USA
| | - Luciano M Prevedello
- The Ohio State University, College of Medicine, Department of Radiology, Columbus, OH, 43210, USA
| |
Collapse
|
17
|
Plesner LL, Müller FC, Nybing JD, Laustrup LC, Rasmussen F, Nielsen OW, Boesen M, Andersen MB. Autonomous Chest Radiograph Reporting Using AI: Estimation of Clinical Impact. Radiology 2023; 307:e222268. [PMID: 36880947 DOI: 10.1148/radiol.222268] [Citation(s) in RCA: 14] [Impact Index Per Article: 14.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 03/08/2023]
Abstract
Background Automated interpretation of normal chest radiographs could alleviate the workload of radiologists. However, the performance of such an artificial intelligence (AI) tool compared with clinical radiology reports has not been established. Purpose To perform an external evaluation of a commercially available AI tool for (a) the number of chest radiographs autonomously reported, (b) the sensitivity for AI detection of abnormal chest radiographs, and (c) the performance of AI compared with that of the clinical radiology reports. Materials and Methods In this retrospective study, consecutive posteroanterior chest radiographs from adult patients in four hospitals in the capital region of Denmark were obtained in January 2020, including images from emergency department patients, in-hospital patients, and outpatients. Three thoracic radiologists labeled chest radiographs in a reference standard based on chest radiograph findings into the following categories: critical, other remarkable, unremarkable, or normal (no abnormalities). AI classified chest radiographs as high confidence normal (normal) or not high confidence normal (abnormal). Results A total of 1529 patients were included for analysis (median age, 69 years [IQR, 55-69 years]; 776 women), with 1100 (72%) classified by the reference standard as having abnormal radiographs, 617 (40%) as having critical abnormal radiographs, and 429 (28%) as having normal radiographs. For comparison, clinical radiology reports were classified based on the text and insufficient reports excluded (n = 22). The sensitivity of AI was 99.1% (95% CI: 98.3, 99.6; 1090 of 1100 patients) for abnormal radiographs and 99.8% (95% CI: 99.1, 99.9; 616 of 617 patients) for critical radiographs. Corresponding sensitivities for radiologist reports were 72.3% (95% CI: 69.5, 74.9; 779 of 1078 patients) and 93.5% (95% CI: 91.2, 95.3; 558 of 597 patients), respectively. Specificity of AI, and hence the potential autonomous reporting rate, was 28.0% of all normal posteroanterior chest radiographs (95% CI: 23.8, 32.5; 120 of 429 patients), or 7.8% (120 of 1529 patients) of all posteroanterior chest radiographs. Conclusion Of all normal posteroanterior chest radiographs, 28% were autonomously reported by AI with a sensitivity for any abnormalities higher than 99%. This corresponded to 7.8% of the entire posteroanterior chest radiograph production. © RSNA, 2023 Supplemental material is available for this article. See also the editorial by Park in this issue.
Collapse
Affiliation(s)
- Louis L Plesner
- From the Department of Radiology, Herlev and Gentofte Hospital, Borgmester Ib Juuls vej 1, 2730 Herlev, Copenhagen, Denmark (L.L.P., F.C.M., L.C.L., M.B.A.); Faculty of Health Sciences, University of Copenhagen, Copenhagen, Denmark (L.L.P., O.W.N., M.B., M.B.A.); Radiological Artificial Intelligence Testcenter, RAIT.dk, Capital region of Denmark (L.L.P., F.C.M., J.D.N., M.B., M.B.A.); Department of Radiology, Bispebjerg and Frederiksberg Hospital, Copenhagen, Denmark (J.D.N., M.B.); Department of Radiology, Aarhus University Hospital, Aarhus, Denmark (F.R.); and Department of Cardiology, Bispebjerg and Frederiksberg Hospital, Copenhagen, Denmark (O.W.N.)
| | - Felix C Müller
- From the Department of Radiology, Herlev and Gentofte Hospital, Borgmester Ib Juuls vej 1, 2730 Herlev, Copenhagen, Denmark (L.L.P., F.C.M., L.C.L., M.B.A.); Faculty of Health Sciences, University of Copenhagen, Copenhagen, Denmark (L.L.P., O.W.N., M.B., M.B.A.); Radiological Artificial Intelligence Testcenter, RAIT.dk, Capital region of Denmark (L.L.P., F.C.M., J.D.N., M.B., M.B.A.); Department of Radiology, Bispebjerg and Frederiksberg Hospital, Copenhagen, Denmark (J.D.N., M.B.); Department of Radiology, Aarhus University Hospital, Aarhus, Denmark (F.R.); and Department of Cardiology, Bispebjerg and Frederiksberg Hospital, Copenhagen, Denmark (O.W.N.)
| | - Janus D Nybing
- From the Department of Radiology, Herlev and Gentofte Hospital, Borgmester Ib Juuls vej 1, 2730 Herlev, Copenhagen, Denmark (L.L.P., F.C.M., L.C.L., M.B.A.); Faculty of Health Sciences, University of Copenhagen, Copenhagen, Denmark (L.L.P., O.W.N., M.B., M.B.A.); Radiological Artificial Intelligence Testcenter, RAIT.dk, Capital region of Denmark (L.L.P., F.C.M., J.D.N., M.B., M.B.A.); Department of Radiology, Bispebjerg and Frederiksberg Hospital, Copenhagen, Denmark (J.D.N., M.B.); Department of Radiology, Aarhus University Hospital, Aarhus, Denmark (F.R.); and Department of Cardiology, Bispebjerg and Frederiksberg Hospital, Copenhagen, Denmark (O.W.N.)
| | - Lene C Laustrup
- From the Department of Radiology, Herlev and Gentofte Hospital, Borgmester Ib Juuls vej 1, 2730 Herlev, Copenhagen, Denmark (L.L.P., F.C.M., L.C.L., M.B.A.); Faculty of Health Sciences, University of Copenhagen, Copenhagen, Denmark (L.L.P., O.W.N., M.B., M.B.A.); Radiological Artificial Intelligence Testcenter, RAIT.dk, Capital region of Denmark (L.L.P., F.C.M., J.D.N., M.B., M.B.A.); Department of Radiology, Bispebjerg and Frederiksberg Hospital, Copenhagen, Denmark (J.D.N., M.B.); Department of Radiology, Aarhus University Hospital, Aarhus, Denmark (F.R.); and Department of Cardiology, Bispebjerg and Frederiksberg Hospital, Copenhagen, Denmark (O.W.N.)
| | - Finn Rasmussen
- From the Department of Radiology, Herlev and Gentofte Hospital, Borgmester Ib Juuls vej 1, 2730 Herlev, Copenhagen, Denmark (L.L.P., F.C.M., L.C.L., M.B.A.); Faculty of Health Sciences, University of Copenhagen, Copenhagen, Denmark (L.L.P., O.W.N., M.B., M.B.A.); Radiological Artificial Intelligence Testcenter, RAIT.dk, Capital region of Denmark (L.L.P., F.C.M., J.D.N., M.B., M.B.A.); Department of Radiology, Bispebjerg and Frederiksberg Hospital, Copenhagen, Denmark (J.D.N., M.B.); Department of Radiology, Aarhus University Hospital, Aarhus, Denmark (F.R.); and Department of Cardiology, Bispebjerg and Frederiksberg Hospital, Copenhagen, Denmark (O.W.N.)
| | - Olav W Nielsen
- From the Department of Radiology, Herlev and Gentofte Hospital, Borgmester Ib Juuls vej 1, 2730 Herlev, Copenhagen, Denmark (L.L.P., F.C.M., L.C.L., M.B.A.); Faculty of Health Sciences, University of Copenhagen, Copenhagen, Denmark (L.L.P., O.W.N., M.B., M.B.A.); Radiological Artificial Intelligence Testcenter, RAIT.dk, Capital region of Denmark (L.L.P., F.C.M., J.D.N., M.B., M.B.A.); Department of Radiology, Bispebjerg and Frederiksberg Hospital, Copenhagen, Denmark (J.D.N., M.B.); Department of Radiology, Aarhus University Hospital, Aarhus, Denmark (F.R.); and Department of Cardiology, Bispebjerg and Frederiksberg Hospital, Copenhagen, Denmark (O.W.N.)
| | - Mikael Boesen
- From the Department of Radiology, Herlev and Gentofte Hospital, Borgmester Ib Juuls vej 1, 2730 Herlev, Copenhagen, Denmark (L.L.P., F.C.M., L.C.L., M.B.A.); Faculty of Health Sciences, University of Copenhagen, Copenhagen, Denmark (L.L.P., O.W.N., M.B., M.B.A.); Radiological Artificial Intelligence Testcenter, RAIT.dk, Capital region of Denmark (L.L.P., F.C.M., J.D.N., M.B., M.B.A.); Department of Radiology, Bispebjerg and Frederiksberg Hospital, Copenhagen, Denmark (J.D.N., M.B.); Department of Radiology, Aarhus University Hospital, Aarhus, Denmark (F.R.); and Department of Cardiology, Bispebjerg and Frederiksberg Hospital, Copenhagen, Denmark (O.W.N.)
| | - Michael B Andersen
- From the Department of Radiology, Herlev and Gentofte Hospital, Borgmester Ib Juuls vej 1, 2730 Herlev, Copenhagen, Denmark (L.L.P., F.C.M., L.C.L., M.B.A.); Faculty of Health Sciences, University of Copenhagen, Copenhagen, Denmark (L.L.P., O.W.N., M.B., M.B.A.); Radiological Artificial Intelligence Testcenter, RAIT.dk, Capital region of Denmark (L.L.P., F.C.M., J.D.N., M.B., M.B.A.); Department of Radiology, Bispebjerg and Frederiksberg Hospital, Copenhagen, Denmark (J.D.N., M.B.); Department of Radiology, Aarhus University Hospital, Aarhus, Denmark (F.R.); and Department of Cardiology, Bispebjerg and Frederiksberg Hospital, Copenhagen, Denmark (O.W.N.)
| |
Collapse
|
18
|
Choi D, Kim T, Jang J, Sunwoo L, Lee KJ. Intracranial steno-occlusive lesion detection on time-of-flight MR angiography using multi-task learning. Comput Med Imaging Graph 2023; 107:102220. [PMID: 37023509 DOI: 10.1016/j.compmedimag.2023.102220] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/18/2022] [Revised: 03/04/2023] [Accepted: 03/19/2023] [Indexed: 03/28/2023]
Abstract
Steno-occlusive lesions in intracranial arteries refer to segments of narrowed or occluded blood vessels that increase the risk of ischemic strokes. Steno-occlusive lesion detection is crucial in clinical settings; however, automatic detection methods have hardly been studied. Therefore, we propose a novel automatic method to detect steno-occlusive lesions in sequential transverse slices on time-of-flight magnetic resonance angiography. Our method simultaneously detects lesions while segmenting blood vessels based on end-to-end multi-task learning, reflecting that the lesions are closely related to the connectivity of blood vessels. We design classification and localization modules that can be attached to arbitrary segmentation network. As blood vessels are segmented, both modules simultaneously predict the presence and location of lesions for each transverse slice. By combining outputs from the two modules, we devise a simple operation that boosts the performance of lesion localization. Experimental results show that lesion prediction and localization performance is improved by incorporating blood vessel extraction. Our ablation study demonstrates that the proposed operation enhances lesion localization accuracy. We also verify the effectiveness of multi-task learning by comparing our approach with those that individually detect lesions with extracted blood vessels.
Collapse
|
19
|
Ahmad HK, Milne MR, Buchlak QD, Ektas N, Sanderson G, Chamtie H, Karunasena S, Chiang J, Holt X, Tang CHM, Seah JCY, Bottrell G, Esmaili N, Brotchie P, Jones C. Machine Learning Augmented Interpretation of Chest X-rays: A Systematic Review. Diagnostics (Basel) 2023; 13:diagnostics13040743. [PMID: 36832231 PMCID: PMC9955112 DOI: 10.3390/diagnostics13040743] [Citation(s) in RCA: 4] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/19/2023] [Revised: 02/13/2023] [Accepted: 02/14/2023] [Indexed: 02/18/2023] Open
Abstract
Limitations of the chest X-ray (CXR) have resulted in attempts to create machine learning systems to assist clinicians and improve interpretation accuracy. An understanding of the capabilities and limitations of modern machine learning systems is necessary for clinicians as these tools begin to permeate practice. This systematic review aimed to provide an overview of machine learning applications designed to facilitate CXR interpretation. A systematic search strategy was executed to identify research into machine learning algorithms capable of detecting >2 radiographic findings on CXRs published between January 2020 and September 2022. Model details and study characteristics, including risk of bias and quality, were summarized. Initially, 2248 articles were retrieved, with 46 included in the final review. Published models demonstrated strong standalone performance and were typically as accurate, or more accurate, than radiologists or non-radiologist clinicians. Multiple studies demonstrated an improvement in the clinical finding classification performance of clinicians when models acted as a diagnostic assistance device. Device performance was compared with that of clinicians in 30% of studies, while effects on clinical perception and diagnosis were evaluated in 19%. Only one study was prospectively run. On average, 128,662 images were used to train and validate models. Most classified less than eight clinical findings, while the three most comprehensive models classified 54, 72, and 124 findings. This review suggests that machine learning devices designed to facilitate CXR interpretation perform strongly, improve the detection performance of clinicians, and improve the efficiency of radiology workflow. Several limitations were identified, and clinician involvement and expertise will be key to driving the safe implementation of quality CXR machine learning systems.
Collapse
Affiliation(s)
- Hassan K. Ahmad
- Annalise.ai, Sydney, NSW 2000, Australia
- Department of Emergency Medicine, Royal North Shore Hospital, Sydney, NSW 2065, Australia
- Correspondence:
| | | | - Quinlan D. Buchlak
- Annalise.ai, Sydney, NSW 2000, Australia
- School of Medicine, University of Notre Dame Australia, Sydney, NSW 2007, Australia
- Department of Neurosurgery, Monash Health, Melbourne, VIC 3168, Australia
| | | | | | | | | | - Jason Chiang
- Annalise.ai, Sydney, NSW 2000, Australia
- Department of General Practice, University of Melbourne, Melbourne, VIC 3010, Australia
- Westmead Applied Research Centre, University of Sydney, Sydney, NSW 2006, Australia
| | | | | | - Jarrel C. Y. Seah
- Annalise.ai, Sydney, NSW 2000, Australia
- Department of Radiology, Alfred Health, Melbourne, VIC 3004, Australia
| | | | - Nazanin Esmaili
- School of Medicine, University of Notre Dame Australia, Sydney, NSW 2007, Australia
- Faculty of Engineering and Information Technology, University of Technology Sydney, Sydney, NSW 2007, Australia
| | - Peter Brotchie
- Annalise.ai, Sydney, NSW 2000, Australia
- Department of Radiology, St Vincent’s Health Australia, Melbourne, VIC 3065, Australia
| | - Catherine Jones
- Annalise.ai, Sydney, NSW 2000, Australia
- I-MED Radiology Network, Brisbane, QLD 4006, Australia
- School of Public and Preventive Health, Monash University, Clayton, VIC 3800, Australia
- Department of Clinical Imaging Science, University of Sydney, Sydney, NSW 2006, Australia
| |
Collapse
|
20
|
Nuutinen M, Hiltunen AM, Korhonen S, Haavisto I, Poikonen-Saksela P, Mattson J, Manikis G, Kondylakis H, Simos P, Mazzocco K, Pat-Horenczyk R, Sousa B, Cardoso F, Manica I, Kudel I, Leskelä RL. Aid of a machine learning algorithm can improve clinician predictions of patient quality of life during breast cancer treatments. HEALTH AND TECHNOLOGY 2023. [DOI: 10.1007/s12553-023-00733-7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/12/2023]
|
21
|
Nam JG, Hwang EJ, Kim J, Park N, Lee EH, Kim HJ, Nam M, Lee JH, Park CM, Goo JM. AI Improves Nodule Detection on Chest Radiographs in a Health Screening Population: A Randomized Controlled Trial. Radiology 2023; 307:e221894. [PMID: 36749213 DOI: 10.1148/radiol.221894] [Citation(s) in RCA: 24] [Impact Index Per Article: 24.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/08/2023]
Abstract
Background The impact of artificial intelligence (AI)-based computer-aided detection (CAD) software has not been prospectively explored in real-world populations. Purpose To investigate whether commercial AI-based CAD software could improve the detection rate of actionable lung nodules on chest radiographs in participants undergoing health checkups. Materials and Methods In this single-center, pragmatic, open-label randomized controlled trial, participants who underwent chest radiography between July 2020 and December 2021 in a health screening center were enrolled and randomized into intervention (AI group) and control (non-AI group) arms. One of three designated radiologists with 13-36 years of experience interpreted each radiograph, referring to the AI-based CAD results for the AI group. The primary outcome was the detection rate, that is, the number of true-positive radiographs divided by the total number of radiographs, of actionable lung nodules confirmed on CT scans obtained within 3 months. Actionable nodules were defined as solid nodules larger than 8 mm or subsolid nodules with a solid portion larger than 6 mm (Lung Imaging Reporting and Data System, or Lung-RADS, category 4). Secondary outcomes included the positive-report rate, sensitivity, false-referral rate, and malignant lung nodule detection rate. Clinical outcomes were compared between the two groups using univariable logistic regression analyses. Results A total of 10 476 participants (median age, 59 years [IQR, 50-66 years]; 5121 men) were randomized to an AI group (n = 5238) or non-AI group (n = 5238). The trial met the predefined primary outcome, demonstrating an improved detection rate of actionable nodules in the AI group compared with the non-AI group (0.59% [31 of 5238 participants] vs 0.25% [13 of 5238 participants], respectively; odds ratio, 2.4; 95% CI: 1.3, 4.7; P = .008). The detection rate for malignant lung nodules was higher in the AI group compared with the non-AI group (0.15% [eight of 5238 participants] vs 0.0% [0 of 5238 participants], respectively; P = .008). The AI and non-AI groups showed similar false-referral rates (45.9% [56 of 122 participants] vs 56.0% [56 of 100 participants], respectively; P = .14) and positive-report rates (2.3% [122 of 5238 participants] vs 1.9% [100 of 5238 participants]; P = .14). Conclusion In health checkup participants, artificial intelligence-based software improved the detection of actionable lung nodules on chest radiographs. © RSNA, 2023 Supplemental material is available for this article. See also the editorial by Auffermann in this isssue.
Collapse
Affiliation(s)
- Ju Gang Nam
- From the Department of Radiology (J.G.N., E.J.H., J.H.L., C.M.P., J.M.G.), Artificial Intelligence Collaborative Network (J.G.N.), Medical Research Collaborating Center (J.K., N.P.), and Center for Health Promotion and Optimal Aging (E.H.L., M.N.), Seoul National University Hospital and College of Medicine, 101 Daehak-ro, Jongno-gu, Seoul 03080, Republic of Korea; Department of Radiology, Ewha Womans University Seoul Hospital, Seoul, Republic of Korea (H.J.K.); Institute of Medical and Biological Engineering, Seoul National University Medical Research Center, Seoul, Republic of Korea (C.M.P.); and Cancer Research Institute, Seoul National University, Seoul, Republic of Korea (J.M.G.)
| | - Eui Jin Hwang
- From the Department of Radiology (J.G.N., E.J.H., J.H.L., C.M.P., J.M.G.), Artificial Intelligence Collaborative Network (J.G.N.), Medical Research Collaborating Center (J.K., N.P.), and Center for Health Promotion and Optimal Aging (E.H.L., M.N.), Seoul National University Hospital and College of Medicine, 101 Daehak-ro, Jongno-gu, Seoul 03080, Republic of Korea; Department of Radiology, Ewha Womans University Seoul Hospital, Seoul, Republic of Korea (H.J.K.); Institute of Medical and Biological Engineering, Seoul National University Medical Research Center, Seoul, Republic of Korea (C.M.P.); and Cancer Research Institute, Seoul National University, Seoul, Republic of Korea (J.M.G.)
| | - Jayoun Kim
- From the Department of Radiology (J.G.N., E.J.H., J.H.L., C.M.P., J.M.G.), Artificial Intelligence Collaborative Network (J.G.N.), Medical Research Collaborating Center (J.K., N.P.), and Center for Health Promotion and Optimal Aging (E.H.L., M.N.), Seoul National University Hospital and College of Medicine, 101 Daehak-ro, Jongno-gu, Seoul 03080, Republic of Korea; Department of Radiology, Ewha Womans University Seoul Hospital, Seoul, Republic of Korea (H.J.K.); Institute of Medical and Biological Engineering, Seoul National University Medical Research Center, Seoul, Republic of Korea (C.M.P.); and Cancer Research Institute, Seoul National University, Seoul, Republic of Korea (J.M.G.)
| | - Nanhee Park
- From the Department of Radiology (J.G.N., E.J.H., J.H.L., C.M.P., J.M.G.), Artificial Intelligence Collaborative Network (J.G.N.), Medical Research Collaborating Center (J.K., N.P.), and Center for Health Promotion and Optimal Aging (E.H.L., M.N.), Seoul National University Hospital and College of Medicine, 101 Daehak-ro, Jongno-gu, Seoul 03080, Republic of Korea; Department of Radiology, Ewha Womans University Seoul Hospital, Seoul, Republic of Korea (H.J.K.); Institute of Medical and Biological Engineering, Seoul National University Medical Research Center, Seoul, Republic of Korea (C.M.P.); and Cancer Research Institute, Seoul National University, Seoul, Republic of Korea (J.M.G.)
| | - Eun Hee Lee
- From the Department of Radiology (J.G.N., E.J.H., J.H.L., C.M.P., J.M.G.), Artificial Intelligence Collaborative Network (J.G.N.), Medical Research Collaborating Center (J.K., N.P.), and Center for Health Promotion and Optimal Aging (E.H.L., M.N.), Seoul National University Hospital and College of Medicine, 101 Daehak-ro, Jongno-gu, Seoul 03080, Republic of Korea; Department of Radiology, Ewha Womans University Seoul Hospital, Seoul, Republic of Korea (H.J.K.); Institute of Medical and Biological Engineering, Seoul National University Medical Research Center, Seoul, Republic of Korea (C.M.P.); and Cancer Research Institute, Seoul National University, Seoul, Republic of Korea (J.M.G.)
| | - Hyun Jin Kim
- From the Department of Radiology (J.G.N., E.J.H., J.H.L., C.M.P., J.M.G.), Artificial Intelligence Collaborative Network (J.G.N.), Medical Research Collaborating Center (J.K., N.P.), and Center for Health Promotion and Optimal Aging (E.H.L., M.N.), Seoul National University Hospital and College of Medicine, 101 Daehak-ro, Jongno-gu, Seoul 03080, Republic of Korea; Department of Radiology, Ewha Womans University Seoul Hospital, Seoul, Republic of Korea (H.J.K.); Institute of Medical and Biological Engineering, Seoul National University Medical Research Center, Seoul, Republic of Korea (C.M.P.); and Cancer Research Institute, Seoul National University, Seoul, Republic of Korea (J.M.G.)
| | - Miyeon Nam
- From the Department of Radiology (J.G.N., E.J.H., J.H.L., C.M.P., J.M.G.), Artificial Intelligence Collaborative Network (J.G.N.), Medical Research Collaborating Center (J.K., N.P.), and Center for Health Promotion and Optimal Aging (E.H.L., M.N.), Seoul National University Hospital and College of Medicine, 101 Daehak-ro, Jongno-gu, Seoul 03080, Republic of Korea; Department of Radiology, Ewha Womans University Seoul Hospital, Seoul, Republic of Korea (H.J.K.); Institute of Medical and Biological Engineering, Seoul National University Medical Research Center, Seoul, Republic of Korea (C.M.P.); and Cancer Research Institute, Seoul National University, Seoul, Republic of Korea (J.M.G.)
| | - Jong Hyuk Lee
- From the Department of Radiology (J.G.N., E.J.H., J.H.L., C.M.P., J.M.G.), Artificial Intelligence Collaborative Network (J.G.N.), Medical Research Collaborating Center (J.K., N.P.), and Center for Health Promotion and Optimal Aging (E.H.L., M.N.), Seoul National University Hospital and College of Medicine, 101 Daehak-ro, Jongno-gu, Seoul 03080, Republic of Korea; Department of Radiology, Ewha Womans University Seoul Hospital, Seoul, Republic of Korea (H.J.K.); Institute of Medical and Biological Engineering, Seoul National University Medical Research Center, Seoul, Republic of Korea (C.M.P.); and Cancer Research Institute, Seoul National University, Seoul, Republic of Korea (J.M.G.)
| | - Chang Min Park
- From the Department of Radiology (J.G.N., E.J.H., J.H.L., C.M.P., J.M.G.), Artificial Intelligence Collaborative Network (J.G.N.), Medical Research Collaborating Center (J.K., N.P.), and Center for Health Promotion and Optimal Aging (E.H.L., M.N.), Seoul National University Hospital and College of Medicine, 101 Daehak-ro, Jongno-gu, Seoul 03080, Republic of Korea; Department of Radiology, Ewha Womans University Seoul Hospital, Seoul, Republic of Korea (H.J.K.); Institute of Medical and Biological Engineering, Seoul National University Medical Research Center, Seoul, Republic of Korea (C.M.P.); and Cancer Research Institute, Seoul National University, Seoul, Republic of Korea (J.M.G.)
| | - Jin Mo Goo
- From the Department of Radiology (J.G.N., E.J.H., J.H.L., C.M.P., J.M.G.), Artificial Intelligence Collaborative Network (J.G.N.), Medical Research Collaborating Center (J.K., N.P.), and Center for Health Promotion and Optimal Aging (E.H.L., M.N.), Seoul National University Hospital and College of Medicine, 101 Daehak-ro, Jongno-gu, Seoul 03080, Republic of Korea; Department of Radiology, Ewha Womans University Seoul Hospital, Seoul, Republic of Korea (H.J.K.); Institute of Medical and Biological Engineering, Seoul National University Medical Research Center, Seoul, Republic of Korea (C.M.P.); and Cancer Research Institute, Seoul National University, Seoul, Republic of Korea (J.M.G.)
| |
Collapse
|
22
|
Zhang Y, Liu M, Zhang L, Wang L, Zhao K, Hu S, Chen X, Xie X. Comparison of Chest Radiograph Captions Based on Natural Language Processing vs Completed by Radiologists. JAMA Netw Open 2023; 6:e2255113. [PMID: 36753278 PMCID: PMC9909497 DOI: 10.1001/jamanetworkopen.2022.55113] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 02/09/2023] Open
Abstract
IMPORTANCE Artificial intelligence (AI) can interpret abnormal signs in chest radiography (CXR) and generate captions, but a prospective study is needed to examine its practical value. OBJECTIVE To prospectively compare natural language processing (NLP)-generated CXR captions and the diagnostic findings of radiologists. DESIGN, SETTING, AND PARTICIPANTS A multicenter diagnostic study was conducted. The training data set included CXR images and reports retrospectively collected from February 1, 2014, to February 28, 2018. The retrospective test data set included consecutive images and reports from April 1 to July 31, 2019. The prospective test data set included consecutive images and reports from May 1 to September 30, 2021. EXPOSURES A bidirectional encoder representation from a transformers model was used to extract language entities and relationships from unstructured CXR reports to establish 23 labels of abnormal signs to train convolutional neural networks. The participants in the prospective test group were randomly assigned to 1 of 3 different caption generation models: a normal template, NLP-generated captions, and rule-based captions based on convolutional neural networks. For each case, a resident drafted the report based on the randomly assigned captions and an experienced radiologist finalized the report blinded to the original captions. A total of 21 residents and 19 radiologists were involved. MAIN OUTCOMES AND MEASURES Time to write reports based on different caption generation models. RESULTS The training data set consisted of 74 082 cases (39 254 [53.0%] women; mean [SD] age, 50.0 [17.1] years). In the retrospective (n = 8126; 4345 [53.5%] women; mean [SD] age, 47.9 [15.9] years) and prospective (n = 5091; 2416 [47.5%] women; mean [SD] age, 45.1 [15.6] years) test data sets, the mean (SD) area under the curve of abnormal signs was 0.87 (0.11) in the retrospective data set and 0.84 (0.09) in the prospective data set. The residents' mean (SD) reporting time using the NLP-generated model was 283 (37) seconds-significantly shorter than the normal template (347 [58] seconds; P < .001) and the rule-based model (296 [46] seconds; P < .001). The NLP-generated captions showed the highest similarity to the final reports with a mean (SD) bilingual evaluation understudy score of 0.69 (0.24)-significantly higher than the normal template (0.37 [0.09]; P < .001) and the rule-based model (0.57 [0.19]; P < .001). CONCLUSIONS AND RELEVANCE In this diagnostic study of NLP-generated CXR captions, prior information provided by NLP was associated with greater efficiency in the reporting process, while maintaining good consistency with the findings of radiologists.
Collapse
Affiliation(s)
- Yaping Zhang
- Radiology Department, Shanghai General Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China
| | | | - Lu Zhang
- Radiology Department, Shanghai General Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China
| | - Lingyun Wang
- Radiology Department, Shanghai General Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China
| | - Keke Zhao
- Radiology Department, Shanghai General Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China
| | - Shundong Hu
- Radiology Department, Shanghai Sixth People Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China
| | - Xu Chen
- Winning Health Technology Ltd, Shanghai, China
| | - Xueqian Xie
- Radiology Department, Shanghai General Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China
| |
Collapse
|
23
|
de Margerie-Mellon C, Chassagnon G. Artificial intelligence: A critical review of applications for lung nodule and lung cancer. Diagn Interv Imaging 2023; 104:11-17. [PMID: 36513593 DOI: 10.1016/j.diii.2022.11.007] [Citation(s) in RCA: 12] [Impact Index Per Article: 12.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/21/2022] [Accepted: 11/22/2022] [Indexed: 12/14/2022]
Abstract
Artificial intelligence (AI) is a broad concept that usually refers to computer programs that can learn from data and perform certain specific tasks. In the recent years, the growth of deep learning, a successful technique for computer vision tasks that does not require explicit programming, coupled with the availability of large imaging databases fostered the development of multiple applications in the medical imaging field, especially for lung nodules and lung cancer, mostly through convolutional neural networks (CNN). Some of the first applications of AI is this field were dedicated to automated detection of lung nodules on X-ray and computed tomography (CT) examinations, with performances now reaching or exceeding those of radiologists. For lung nodule segmentation, CNN-based algorithms applied to CT images show excellent spatial overlap index with manual segmentation, even for irregular and ground glass nodules. A third application of AI is the classification of lung nodules between malignant and benign, which could limit the number of follow-up CT examinations for less suspicious lesions. Several algorithms have demonstrated excellent capabilities for the prediction of the malignancy risk when a nodule is discovered. These different applications of AI for lung nodules are particularly appealing in the context of lung cancer screening. In the field of lung cancer, AI tools applied to lung imaging have been investigated for distinct aims. First, they could play a role for the non-invasive characterization of tumors, especially for histological subtype and somatic mutation predictions, with a potential therapeutic impact. Additionally, they could help predict the patient prognosis, in combination to clinical data. Despite these encouraging perspectives, clinical implementation of AI tools is only beginning because of the lack of generalizability of published studies, of an inner obscure working and because of limited data about the impact of such tools on the radiologists' decision and on the patient outcome. Radiologists must be active participants in the process of evaluating AI tools, as such tools could support their daily work and offer them more time for high added value tasks.
Collapse
Affiliation(s)
- Constance de Margerie-Mellon
- Université Paris Cité, Laboratory of Imaging Biomarkers, Center for Research on Inflammation, UMR 1149, INSERM, 75018 Paris, France; Department of Radiology, Hôpital Saint-Louis APHP, 75010 Paris, France
| | - Guillaume Chassagnon
- Université Paris Cité, Faculté de Médecine, 75006 Paris, France; Department of Radiology, Hôpital Cochin APHP, 75014 Paris, France
| |
Collapse
|
24
|
Okolo GI, Katsigiannis S, Ramzan N. IEViT: An enhanced vision transformer architecture for chest X-ray image classification. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2022; 226:107141. [PMID: 36162246 DOI: 10.1016/j.cmpb.2022.107141] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 06/06/2022] [Revised: 08/02/2022] [Accepted: 09/14/2022] [Indexed: 06/16/2023]
Abstract
BACKGROUND AND OBJECTIVE Chest X-ray imaging is a relatively cheap and accessible diagnostic tool that can assist in the diagnosis of various conditions, including pneumonia, tuberculosis, COVID-19, and others. However, the requirement for expert radiologists to view and interpret chest X-ray images can be a bottleneck, especially in remote and deprived areas. Recent advances in machine learning have made possible the automated diagnosis of chest X-ray scans. In this work, we examine the use of a novel Transformer-based deep learning model for the task of chest X-ray image classification. METHODS We first examine the performance of the Vision Transformer (ViT) state-of-the-art image classification machine learning model for the task of chest X-ray image classification, and then propose and evaluate the Input Enhanced Vision Transformer (IEViT), a novel enhanced Vision Transformer model that can achieve improved performance on chest X-ray images associated with various pathologies. RESULTS Experiments on four chest X-ray image data sets containing various pathologies (tuberculosis, pneumonia, COVID-19) demonstrated that the proposed IEViT model outperformed ViT for all the data sets and variants examined, achieving an F1-score between 96.39% and 100%, and an improvement over ViT of up to +5.82% in terms of F1-score across the four examined data sets. IEViT's maximum sensitivity (recall) ranged between 93.50% and 100% across the four data sets, with an improvement over ViT of up to +3%, whereas IEViT's maximum precision ranged between 97.96% and 100% across the four data sets, with an improvement over ViT of up to +6.41%. CONCLUSIONS Results showed that the proposed IEViT model outperformed all ViT's variants for all the examined chest X-ray image data sets, demonstrating its superiority and generalisation ability. Given the relatively low cost and the widespread accessibility of chest X-ray imaging, the use of the proposed IEViT model can potentially offer a powerful, but relatively cheap and accessible method for assisting diagnosis using chest X-ray images.
Collapse
Affiliation(s)
| | | | - Naeem Ramzan
- University of the West of Scotland, High St., Paisley, PA1 2BE, UK
| |
Collapse
|
25
|
Yoo H, Kim EY, Kim H, Choi YR, Kim MY, Hwang SH, Kim YJ, Cho YJ, Jin KN. Artificial Intelligence-Based Identification of Normal Chest Radiographs: A Simulation Study in a Multicenter Health Screening Cohort. Korean J Radiol 2022; 23:1009-1018. [PMID: 36175002 PMCID: PMC9523233 DOI: 10.3348/kjr.2022.0189] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/20/2022] [Revised: 08/11/2022] [Accepted: 08/12/2022] [Indexed: 01/17/2023] Open
Abstract
Objective This study aimed to investigate the feasibility of using artificial intelligence (AI) to identify normal chest radiography (CXR) from the worklist of radiologists in a health-screening environment. Materials and Methods This retrospective simulation study was conducted using the CXRs of 5887 adults (mean age ± standard deviation, 55.4 ± 11.8 years; male, 4329) from three health screening centers in South Korea using a commercial AI (Lunit INSIGHT CXR3, version 3.5.8.8). Three board-certified thoracic radiologists reviewed CXR images for referable thoracic abnormalities and grouped the images into those with visible referable abnormalities (identified as abnormal by at least one reader) and those with clearly visible referable abnormalities (identified as abnormal by at least two readers). With AI-based simulated exclusion of normal CXR images, the percentages of normal images sorted and abnormal images erroneously removed were analyzed. Additionally, in a random subsample of 480 patients, the ability to identify visible referable abnormalities was compared among AI-unassisted reading (i.e., all images read by human readers without AI), AI-assisted reading (i.e., all images read by human readers with AI assistance as concurrent readers), and reading with AI triage (i.e., human reading of only those rendered abnormal by AI). Results Of 5887 CXR images, 405 (6.9%) and 227 (3.9%) contained visible and clearly visible abnormalities, respectively. With AI-based triage, 42.9% (2354/5482) of normal CXR images were removed at the cost of erroneous removal of 3.5% (14/405) and 1.8% (4/227) of CXR images with visible and clearly visible abnormalities, respectively. In the diagnostic performance study, AI triage removed 41.6% (188/452) of normal images from the worklist without missing visible abnormalities and increased the specificity for some readers without decreasing sensitivity. Conclusion This study suggests the feasibility of sorting and removing normal CXRs using AI with a tailored cut-off to increase efficiency and reduce the workload of radiologists.
Collapse
Affiliation(s)
- Hyunsuk Yoo
- Lunit Inc, Seoul, Korea.,Department of Radiology, Seoul National University College of Medicine, Seoul National University Hospital, Seoul, Korea
| | - Eun Young Kim
- Department of Radiology, Gil Medical Center, Gachon University College of Medicine, Incheon, Korea
| | - Hyungjin Kim
- Department of Radiology, Seoul National University College of Medicine, Seoul National University Hospital, Seoul, Korea
| | - Ye Ra Choi
- Department of Radiology, Seoul National University-Seoul Metropolitan Government Boramae Medical Center, Seoul, Korea
| | - Moon Young Kim
- Department of Radiology, Seoul National University-Seoul Metropolitan Government Boramae Medical Center, Seoul, Korea
| | - Sung Ho Hwang
- Department of Radiology, Korea University Anam Hospital, Seoul, Korea
| | - Young Joong Kim
- Department of Radiology, Konyang University Hospital, Konyang University College of Medicine, Daejeon, Korea
| | - Young Jun Cho
- Department of Radiology, Konyang University Hospital, Konyang University College of Medicine, Daejeon, Korea
| | - Kwang Nam Jin
- Department of Radiology, Seoul National University-Seoul Metropolitan Government Boramae Medical Center, Seoul, Korea.
| |
Collapse
|
26
|
Hwang EJ, Park J, Hong W, Lee HJ, Choi H, Kim H, Nam JG, Goo JM, Yoon SH, Lee CH, Park CM. Artificial intelligence system for identification of false-negative interpretations in chest radiographs. Eur Radiol 2022; 32:4468-4478. [PMID: 35195744 DOI: 10.1007/s00330-022-08593-x] [Citation(s) in RCA: 6] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/24/2021] [Revised: 01/04/2022] [Accepted: 01/25/2022] [Indexed: 12/11/2022]
Abstract
OBJECTIVES To investigate the efficacy of an artificial intelligence (AI) system for the identification of false negatives in chest radiographs that were interpreted as normal by radiologists. METHODS We consecutively collected chest radiographs that were read as normal during 1 month (March 2020) in a single institution. A commercialized AI system was retrospectively applied to these radiographs. Radiographs with abnormal AI results were then re-interpreted by the radiologist who initially read the radiograph ("AI as the advisor" scenario). The reference standards for the true presence of relevant abnormalities in radiographs were defined by majority voting of three thoracic radiologists. The efficacy of the AI system was evaluated by detection yield (proportion of true-positive identification among the entire examination) and false-referral rate (FRR, proportion of false-positive identification among all examinations). Decision curve analyses were performed to evaluate the net benefits of applying the AI system. RESULTS A total of 4208 radiographs from 3778 patients (M:F = 1542:2236; median age, 56 years) were included. The AI system identified initially overlooked relevant abnormalities with a detection yield and an FRR of 2.4% and 14.0%, respectively. In the "AI as the advisor" scenario, radiologists detected initially overlooked relevant abnormalities with a detection yield and FRR of 1.2% and 0.97%, respectively. In a decision curve analysis, AI as an advisor scenario exhibited a positive net benefit when the cost-to-benefit ratio was below 1:0.8. CONCLUSION An AI system could identify relevant abnormalities overlooked by radiologists and could enable radiologists to correct their false-negative interpretations by providing feedback to radiologists. KEY POINTS • In consecutive chest radiographs with normal interpretations, an artificial intelligence system could identify relevant abnormalities that were initially overlooked by radiologists. • The artificial intelligence system could enable radiologists to correct their initial false-negative interpretations by providing feedback to radiologists when overlooked abnormalities were present.
Collapse
Affiliation(s)
- Eui Jin Hwang
- Department of Radiology, Seoul National University Hospital, 101 Daehak-ro, Jongno-gu, Seoul, 03080, Korea.,Department of Radiology, Seoul National University College of Medicine, 103 Daehak-ro, Jongno-gu, Seoul, 03080, Korea
| | - Jongsoo Park
- Department of Radiology, Seoul National University Hospital, 101 Daehak-ro, Jongno-gu, Seoul, 03080, Korea
| | - Wonju Hong
- Department of Radiology, Seoul National University Hospital, 101 Daehak-ro, Jongno-gu, Seoul, 03080, Korea
| | - Hyun-Ju Lee
- Department of Radiology, Seoul National University Hospital, 101 Daehak-ro, Jongno-gu, Seoul, 03080, Korea.,Department of Radiology, Seoul National University College of Medicine, 103 Daehak-ro, Jongno-gu, Seoul, 03080, Korea
| | - Hyewon Choi
- Department of Radiology, Seoul National University Hospital, 101 Daehak-ro, Jongno-gu, Seoul, 03080, Korea.,Department of Radiology, Chung-Ang University Hospital, 102 Heukseok-ro, Dongjak-gu, Seoul, 06973, Korea
| | - Hyungjin Kim
- Department of Radiology, Seoul National University Hospital, 101 Daehak-ro, Jongno-gu, Seoul, 03080, Korea.,Department of Radiology, Seoul National University College of Medicine, 103 Daehak-ro, Jongno-gu, Seoul, 03080, Korea
| | - Ju Gang Nam
- Department of Radiology, Seoul National University Hospital, 101 Daehak-ro, Jongno-gu, Seoul, 03080, Korea.,Department of Radiology, Seoul National University College of Medicine, 103 Daehak-ro, Jongno-gu, Seoul, 03080, Korea
| | - Jin Mo Goo
- Department of Radiology, Seoul National University Hospital, 101 Daehak-ro, Jongno-gu, Seoul, 03080, Korea.,Department of Radiology, Seoul National University College of Medicine, 103 Daehak-ro, Jongno-gu, Seoul, 03080, Korea
| | - Soon Ho Yoon
- Department of Radiology, Seoul National University Hospital, 101 Daehak-ro, Jongno-gu, Seoul, 03080, Korea.,Department of Radiology, Seoul National University College of Medicine, 103 Daehak-ro, Jongno-gu, Seoul, 03080, Korea
| | - Chang Hyun Lee
- Department of Radiology, Seoul National University Hospital, 101 Daehak-ro, Jongno-gu, Seoul, 03080, Korea.,Department of Radiology, Seoul National University College of Medicine, 103 Daehak-ro, Jongno-gu, Seoul, 03080, Korea
| | - Chang Min Park
- Department of Radiology, Seoul National University Hospital, 101 Daehak-ro, Jongno-gu, Seoul, 03080, Korea. .,Department of Radiology, Seoul National University College of Medicine, 103 Daehak-ro, Jongno-gu, Seoul, 03080, Korea. .,Institute of Radiation Medicine, Seoul National University College of Medicine, 101 Daehak-ro, Jongno-gu, Seoul, 03080, Korea.
| |
Collapse
|
27
|
Jones CM, Danaher L, Milne MR, Tang C, Seah J, Oakden-Rayner L, Johnson A, Buchlak QD, Esmaili N. Assessment of the effect of a comprehensive chest radiograph deep learning model on radiologist reports and patient outcomes: a real-world observational study. BMJ Open 2021; 11:e052902. [PMID: 34930738 PMCID: PMC8689166 DOI: 10.1136/bmjopen-2021-052902] [Citation(s) in RCA: 14] [Impact Index Per Article: 4.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 12/31/2022] Open
Abstract
OBJECTIVES Artificial intelligence (AI) algorithms have been developed to detect imaging features on chest X-ray (CXR) with a comprehensive AI model capable of detecting 124 CXR findings being recently developed. The aim of this study was to evaluate the real-world usefulness of the model as a diagnostic assistance device for radiologists. DESIGN This prospective real-world multicentre study involved a group of radiologists using the model in their daily reporting workflow to report consecutive CXRs and recording their feedback on level of agreement with the model findings and whether this significantly affected their reporting. SETTING The study took place at radiology clinics and hospitals within a large radiology network in Australia between November and December 2020. PARTICIPANTS Eleven consultant diagnostic radiologists of varying levels of experience participated in this study. PRIMARY AND SECONDARY OUTCOME MEASURES Proportion of CXR cases where use of the AI model led to significant material changes to the radiologist report, to patient management, or to imaging recommendations. Additionally, level of agreement between radiologists and the model findings, and radiologist attitudes towards the model were assessed. RESULTS Of 2972 cases reviewed with the model, 92 cases (3.1%) had significant report changes, 43 cases (1.4%) had changed patient management and 29 cases (1.0%) had further imaging recommendations. In terms of agreement with the model, 2569 cases showed complete agreement (86.5%). 390 (13%) cases had one or more findings rejected by the radiologist. There were 16 findings across 13 cases (0.5%) deemed to be missed by the model. Nine out of 10 radiologists felt their accuracy was improved with the model and were more positive towards AI poststudy. CONCLUSIONS Use of an AI model in a real-world reporting environment significantly improved radiologist reporting and showed good agreement with radiologists, highlighting the potential for AI diagnostic support to improve clinical practice.
Collapse
Affiliation(s)
- Catherine M Jones
- Annalise-AI, Sydney, New South Wales, Australia
- I-Med Radiology Network, Sydney, New South Wales, Australia
| | - Luke Danaher
- I-Med Radiology Network, Sydney, New South Wales, Australia
| | - Michael R Milne
- Annalise-AI, Sydney, New South Wales, Australia
- I-Med Radiology Network, Sydney, New South Wales, Australia
| | - Cyril Tang
- Annalise-AI, Sydney, New South Wales, Australia
| | - Jarrel Seah
- Annalise-AI, Sydney, New South Wales, Australia
- Department of Radiology, Alfred Health, Melbourne, Victoria, Australia
| | - Luke Oakden-Rayner
- Australian Institute for Machine Learning, The University of Adelaide, Adelaide, South Australia, Australia
| | | | - Quinlan D Buchlak
- Annalise-AI, Sydney, New South Wales, Australia
- School of Medicine, The University of Notre Dame Australia School of Medicine Sydney Campus, Darlinghurst, New South Wales, Australia
| | - Nazanin Esmaili
- School of Medicine, The University of Notre Dame Australia School of Medicine Sydney Campus, Darlinghurst, New South Wales, Australia
- Faculty of Engineering and IT, University of Technology Sydney, Sydney, New South Wales, Australia
| |
Collapse
|
28
|
Eche T, Schwartz LH, Mokrane FZ, Dercle L. Toward Generalizability in the Deployment of Artificial Intelligence in Radiology: Role of Computation Stress Testing to Overcome Underspecification. Radiol Artif Intell 2021; 3:e210097. [PMID: 34870222 DOI: 10.1148/ryai.2021210097] [Citation(s) in RCA: 48] [Impact Index Per Article: 16.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/09/2021] [Revised: 09/20/2021] [Accepted: 10/12/2021] [Indexed: 12/20/2022]
Abstract
The clinical deployment of artificial intelligence (AI) applications in medical imaging is perhaps the greatest challenge facing radiology in the next decade. One of the main obstacles to the incorporation of automated AI-based decision-making tools in medicine is the failure of models to generalize when deployed across institutions with heterogeneous populations and imaging protocols. The most well-understood pitfall in developing these AI models is overfitting, which has, in part, been overcome by optimizing training protocols. However, overfitting is not the only obstacle to the success and generalizability of AI. Underspecification is also a serious impediment that requires conceptual understanding and correction. It is well known that a single AI pipeline, with prescribed training and testing sets, can produce several models with various levels of generalizability. Underspecification defines the inability of the pipeline to identify whether these models have embedded the structure of the underlying system by using a test set independent of, but distributed identically, to the training set. An underspecified pipeline is unable to assess the degree to which the models will be generalizable. Stress testing is a known tool in AI that can limit underspecification and, importantly, assure broad generalizability of AI models. However, the application of stress tests is new in radiologic applications. This report describes the concept of underspecification from a radiologist perspective, discusses stress testing as a specific strategy to overcome underspecification, and explains how stress tests could be designed in radiology-by modifying medical images or stratifying testing datasets. In the upcoming years, stress tests should become in radiology the standard that crash tests have become in the automotive industry. Keywords: Computer Applications-General, Informatics, Computer-aided Diagnosis © RSNA, 2021.
Collapse
Affiliation(s)
- Thomas Eche
- Department of Radiology, Toulouse Rangueil Hospital, Toulouse, France (T.E., F.Z.M.); and Department of Radiology, NewYork-Presbyterian Hospital, Columbia University Irving Medical Center, 622 West 168th St, New York, NY 10032 (T.E., L.H.S., L.D.)
| | - Lawrence H Schwartz
- Department of Radiology, Toulouse Rangueil Hospital, Toulouse, France (T.E., F.Z.M.); and Department of Radiology, NewYork-Presbyterian Hospital, Columbia University Irving Medical Center, 622 West 168th St, New York, NY 10032 (T.E., L.H.S., L.D.)
| | - Fatima-Zohra Mokrane
- Department of Radiology, Toulouse Rangueil Hospital, Toulouse, France (T.E., F.Z.M.); and Department of Radiology, NewYork-Presbyterian Hospital, Columbia University Irving Medical Center, 622 West 168th St, New York, NY 10032 (T.E., L.H.S., L.D.)
| | - Laurent Dercle
- Department of Radiology, Toulouse Rangueil Hospital, Toulouse, France (T.E., F.Z.M.); and Department of Radiology, NewYork-Presbyterian Hospital, Columbia University Irving Medical Center, 622 West 168th St, New York, NY 10032 (T.E., L.H.S., L.D.)
| |
Collapse
|
29
|
Homayounieh F, Digumarthy S, Ebrahimian S, Rueckel J, Hoppe BF, Sabel BO, Conjeti S, Ridder K, Sistermanns M, Wang L, Preuhs A, Ghesu F, Mansoor A, Moghbel M, Botwin A, Singh R, Cartmell S, Patti J, Huemmer C, Fieselmann A, Joerger C, Mirshahzadeh N, Muse V, Kalra M. An Artificial Intelligence-Based Chest X-ray Model on Human Nodule Detection Accuracy From a Multicenter Study. JAMA Netw Open 2021; 4:e2141096. [PMID: 34964851 PMCID: PMC8717119 DOI: 10.1001/jamanetworkopen.2021.41096] [Citation(s) in RCA: 37] [Impact Index Per Article: 12.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 12/20/2022] Open
Abstract
IMPORTANCE Most early lung cancers present as pulmonary nodules on imaging, but these can be easily missed on chest radiographs. OBJECTIVE To assess if a novel artificial intelligence (AI) algorithm can help detect pulmonary nodules on radiographs at different levels of detection difficulty. DESIGN, SETTING, AND PARTICIPANTS This diagnostic study included 100 posteroanterior chest radiograph images taken between 2000 and 2010 of adult patients from an ambulatory health care center in Germany and a lung image database in the US. Included images were selected to represent nodules with different levels of detection difficulties (from easy to difficult), and comprised both normal and nonnormal control. EXPOSURES All images were processed with a novel AI algorithm, the AI Rad Companion Chest X-ray. Two thoracic radiologists established the ground truth and 9 test radiologists from Germany and the US independently reviewed all images in 2 sessions (unaided and AI-aided mode) with at least a 1-month washout period. MAIN OUTCOMES AND MEASURES Each test radiologist recorded the presence of 5 findings (pulmonary nodules, atelectasis, consolidation, pneumothorax, and pleural effusion) and their level of confidence for detecting the individual finding on a scale of 1 to 10 (1 representing lowest confidence; 10, highest confidence). The analyzed metrics for nodules included sensitivity, specificity, accuracy, and receiver operating characteristics curve area under the curve (AUC). RESULTS Images from 100 patients were included, with a mean (SD) age of 55 (20) years and including 64 men and 36 women. Mean detection accuracy across the 9 radiologists improved by 6.4% (95% CI, 2.3% to 10.6%) with AI-aided interpretation compared with unaided interpretation. Partial AUCs within the effective interval range of 0 to 0.2 false positive rate improved by 5.6% (95% CI, -1.4% to 12.0%) with AI-aided interpretation. Junior radiologists saw greater improvement in sensitivity for nodule detection with AI-aided interpretation as compared with their senior counterparts (12%; 95% CI, 4% to 19% vs 9%; 95% CI, 1% to 17%) while senior radiologists experienced similar improvement in specificity (4%; 95% CI, -2% to 9%) as compared with junior radiologists (4%; 95% CI, -3% to 5%). CONCLUSIONS AND RELEVANCE In this diagnostic study, an AI algorithm was associated with improved detection of pulmonary nodules on chest radiographs compared with unaided interpretation for different levels of detection difficulty and for readers with different experience.
Collapse
Affiliation(s)
- Fatemeh Homayounieh
- Department of Radiology, Massachusetts General Hospital and Harvard Medical School, Boston, Massachusetts
| | - Subba Digumarthy
- Department of Radiology, Massachusetts General Hospital and Harvard Medical School, Boston, Massachusetts
| | - Shadi Ebrahimian
- Department of Radiology, Massachusetts General Hospital and Harvard Medical School, Boston, Massachusetts
| | - Johannes Rueckel
- Department of Radiology, University Hospital, Ludwig Maximilian University of Munich, Munich, Germany
| | - Boj Friedrich Hoppe
- Department of Radiology, University Hospital, Ludwig Maximilian University of Munich, Munich, Germany
| | - Bastian Oliver Sabel
- Department of Radiology, University Hospital, Ludwig Maximilian University of Munich, Munich, Germany
| | | | - Karsten Ridder
- Medizinisches Versorgungszentrum Professor Uhlenbrock & Partner
| | | | | | | | | | | | - Mateen Moghbel
- Department of Radiology, Massachusetts General Hospital and Harvard Medical School, Boston, Massachusetts
| | - Ariel Botwin
- Department of Radiology, Massachusetts General Hospital and Harvard Medical School, Boston, Massachusetts
| | - Ramandeep Singh
- Department of Radiology, Massachusetts General Hospital and Harvard Medical School, Boston, Massachusetts
| | - Samuel Cartmell
- Department of Radiology, Massachusetts General Hospital and Harvard Medical School, Boston, Massachusetts
| | - John Patti
- Department of Radiology, Massachusetts General Hospital and Harvard Medical School, Boston, Massachusetts
| | | | | | | | | | - Victorine Muse
- Department of Radiology, Massachusetts General Hospital and Harvard Medical School, Boston, Massachusetts
| | - Mannudeep Kalra
- Department of Radiology, Massachusetts General Hospital and Harvard Medical School, Boston, Massachusetts
| |
Collapse
|
30
|
A Highly Reliable Convolutional Neural Network Based Soft Tissue Sarcoma Metastasis Detection from Chest X-ray Images: A Retrospective Cohort Study. Cancers (Basel) 2021; 13:cancers13194961. [PMID: 34638445 PMCID: PMC8508001 DOI: 10.3390/cancers13194961] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/30/2021] [Revised: 09/29/2021] [Accepted: 09/30/2021] [Indexed: 12/15/2022] Open
Abstract
Simple Summary Soft tissue sarcomas are relatively rare malignant diseases. Part of the diagnosis and follow-up includes medical imaging of the thorax for detection of lung metastases. A Python script was created and trained using a set of lung X-rays and concordant CT scans from a high-volume German-speaking sarcoma center. It is capable of detecting malignant metastasis in the lung with a precision of 71.2%, specificity of 90.5%, sensitivity of 94% and accuracy of 91.2%. Furthermore, the program was able to detect even small nodules with a size <1 cm in conventional X-rays of the thorax. This algorithm was implemented into our daily clinical practice alongside with the radiologists’ findings. With this tool we aim to improve the quality of our service and reduce the expenditure of time. Abstract Introduction: soft tissue sarcomas are a subset of malignant tumors that are relatively rare and make up 1% of all malignant tumors in adulthood. Due to the rarity of these tumors, there are significant differences in quality in the diagnosis and treatment of these tumors. One paramount aspect is the diagnosis of hematogenous metastases in the lungs. Guidelines recommend routine lung imaging by means of X-rays. With the ever advancing AI-based diagnostic support, there has so far been no implementation for sarcomas. The aim of the study was to utilize AI to obtain analyzes regarding metastasis on lung X-rays in the most possible sensitive and specific manner in sarcoma patients. Methods: a Python script was created and trained using a set of lung X-rays with sarcoma metastases from a high-volume German-speaking sarcoma center. 26 patients with lung metastasis were included. For all patients chest X-ray with corresponding lung CT scans, and histological biopsies were available. The number of trainable images were expanded to 600. In order to evaluate the biological sensitivity and specificity, the script was tested on lung X-rays with a lung CT as control. Results: in this study we present a new type of convolutional neural network-based system with a precision of 71.2%, specificity of 90.5%, sensitivity of 94%, recall of 94% and accuracy of 91.2%. A good detection of even small findings was determined. Discussion: the created script establishes the option to check lung X-rays for metastases at a safe level, especially given this rare tumor entity.
Collapse
|
31
|
Jones CM, Buchlak QD, Oakden‐Rayner L, Milne M, Seah J, Esmaili N, Hachey B. Chest radiographs and machine learning - Past, present and future. J Med Imaging Radiat Oncol 2021; 65:538-544. [PMID: 34169648 PMCID: PMC8453538 DOI: 10.1111/1754-9485.13274] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/25/2021] [Accepted: 06/08/2021] [Indexed: 01/15/2023]
Abstract
Despite its simple acquisition technique, the chest X-ray remains the most common first-line imaging tool for chest assessment globally. Recent evidence for image analysis using modern machine learning points to possible improvements in both the efficiency and the accuracy of chest X-ray interpretation. While promising, these machine learning algorithms have not provided comprehensive assessment of findings in an image and do not account for clinical history or other relevant clinical information. However, the rapid evolution in technology and evidence base for its use suggests that the next generation of comprehensive, well-tested machine learning algorithms will be a revolution akin to early advances in X-ray technology. Current use cases, strengths, limitations and applications of chest X-ray machine learning systems are discussed.
Collapse
Affiliation(s)
- Catherine M Jones
- I‐MED Radiology NetworkBrisbaneQueenslandAustralia
- Annalise.aiSydneyNew South WalesAustralia
| | - Quinlan D Buchlak
- Annalise.aiSydneyNew South WalesAustralia
- School of MedicineThe University of Notre Dame AustraliaSydneyNew South WalesAustralia
- Harrison.aiSydneyNew South WalesAustralia
| | - Luke Oakden‐Rayner
- Australian Institute for Machine LearningThe University of AdelaideAdelaideSouth AustraliaAustralia
| | - Michael Milne
- I‐MED Radiology NetworkBrisbaneQueenslandAustralia
- Annalise.aiSydneyNew South WalesAustralia
| | - Jarrel Seah
- Annalise.aiSydneyNew South WalesAustralia
- Harrison.aiSydneyNew South WalesAustralia
- Department of RadiologyAlfred HealthMelbourneVictoriaAustralia
| | - Nazanin Esmaili
- School of MedicineThe University of Notre Dame AustraliaSydneyNew South WalesAustralia
- Faculty of Engineering and Information TechnologyUniversity of Technology SydneySydneyNew South WalesAustralia
| | - Ben Hachey
- Annalise.aiSydneyNew South WalesAustralia
- Harrison.aiSydneyNew South WalesAustralia
| |
Collapse
|
32
|
Seah JCY, Tang CHM, Buchlak QD, Holt XG, Wardman JB, Aimoldin A, Esmaili N, Ahmad H, Pham H, Lambert JF, Hachey B, Hogg SJF, Johnston BP, Bennett C, Oakden-Rayner L, Brotchie P, Jones CM. Effect of a comprehensive deep-learning model on the accuracy of chest x-ray interpretation by radiologists: a retrospective, multireader multicase study. Lancet Digit Health 2021; 3:e496-e506. [PMID: 34219054 DOI: 10.1016/s2589-7500(21)00106-0] [Citation(s) in RCA: 85] [Impact Index Per Article: 28.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/03/2020] [Revised: 05/02/2021] [Accepted: 05/12/2021] [Indexed: 02/01/2023]
Abstract
BACKGROUND Chest x-rays are widely used in clinical practice; however, interpretation can be hindered by human error and a lack of experienced thoracic radiologists. Deep learning has the potential to improve the accuracy of chest x-ray interpretation. We therefore aimed to assess the accuracy of radiologists with and without the assistance of a deep-learning model. METHODS In this retrospective study, a deep-learning model was trained on 821 681 images (284 649 patients) from five data sets from Australia, Europe, and the USA. 2568 enriched chest x-ray cases from adult patients (≥16 years) who had at least one frontal chest x-ray were included in the test dataset; cases were representative of inpatient, outpatient, and emergency settings. 20 radiologists reviewed cases with and without the assistance of the deep-learning model with a 3-month washout period. We assessed the change in accuracy of chest x-ray interpretation across 127 clinical findings when the deep-learning model was used as a decision support by calculating area under the receiver operating characteristic curve (AUC) for each radiologist with and without the deep-learning model. We also compared AUCs for the model alone with those of unassisted radiologists. If the lower bound of the adjusted 95% CI of the difference in AUC between the model and the unassisted radiologists was more than -0·05, the model was considered to be non-inferior for that finding. If the lower bound exceeded 0, the model was considered to be superior. FINDINGS Unassisted radiologists had a macroaveraged AUC of 0·713 (95% CI 0·645-0·785) across the 127 clinical findings, compared with 0·808 (0·763-0·839) when assisted by the model. The deep-learning model statistically significantly improved the classification accuracy of radiologists for 102 (80%) of 127 clinical findings, was statistically non-inferior for 19 (15%) findings, and no findings showed a decrease in accuracy when radiologists used the deep-learning model. Unassisted radiologists had a macroaveraged mean AUC of 0·713 (0·645-0·785) across all findings, compared with 0·957 (0·954-0·959) for the model alone. Model classification alone was significantly more accurate than unassisted radiologists for 117 (94%) of 124 clinical findings predicted by the model and was non-inferior to unassisted radiologists for all other clinical findings. INTERPRETATION This study shows the potential of a comprehensive deep-learning model to improve chest x-ray interpretation across a large breadth of clinical practice. FUNDING Annalise.ai.
Collapse
Affiliation(s)
- Jarrel C Y Seah
- Annalise.ai, Sydney, NSW, Australia; Department of Radiology, Alfred Health, Melbourne, VIC, Australia
| | | | | | | | | | | | - Nazanin Esmaili
- School of Medicine, University of Notre Dame Australia, Sydney, NSW, Australia; Faculty of Engineering and IT, University of Technology Sydney, Sydney, NSW, Australia
| | | | | | | | | | | | | | - Christine Bennett
- School of Medicine, University of Notre Dame Australia, Sydney, NSW, Australia
| | - Luke Oakden-Rayner
- Australian Institute for Machine Learning, The University of Adelaide, Adelaide, SA, Australia
| | - Peter Brotchie
- Annalise.ai, Sydney, NSW, Australia; Department of Radiology, St Vincent's Health Australia, Melbourne, VIC, Australia
| | | |
Collapse
|
33
|
Moses DA. Deep learning applied to automatic disease detection using chest X-rays. J Med Imaging Radiat Oncol 2021; 65:498-517. [PMID: 34231311 DOI: 10.1111/1754-9485.13273] [Citation(s) in RCA: 6] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/29/2021] [Accepted: 06/08/2021] [Indexed: 12/24/2022]
Abstract
Deep learning (DL) has shown rapid advancement and considerable promise when applied to the automatic detection of diseases using CXRs. This is important given the widespread use of CXRs across the world in diagnosing significant pathologies, and the lack of trained radiologists to report them. This review article introduces the basic concepts of DL as applied to CXR image analysis including basic deep neural network (DNN) structure, the use of transfer learning and the application of data augmentation. It then reviews the current literature on how DNN models have been applied to the detection of common CXR abnormalities (e.g. lung nodules, pneumonia, tuberculosis and pneumothorax) over the last few years. This includes DL approaches employed for the classification of multiple different diseases (multi-class classification). Performance of different techniques and models and their comparison with human observers are presented. Some of the challenges facing DNN models, including their future implementation and relationships to radiologists, are also discussed.
Collapse
Affiliation(s)
- Daniel A Moses
- Graduate School of Biomedical Engineering, Faculty of Engineering, University of New South Wales, Sydney, New South Wales, Australia.,Department of Medical Imaging, Prince of Wales Hospital, Sydney, New South Wales, Australia
| |
Collapse
|
34
|
Yoo H, Lee SH, Arru CD, Doda Khera R, Singh R, Siebert S, Kim D, Lee Y, Park JH, Eom HJ, Digumarthy SR, Kalra MK. AI-based improvement in lung cancer detection on chest radiographs: results of a multi-reader study in NLST dataset. Eur Radiol 2021; 31:9664-9674. [PMID: 34089072 DOI: 10.1007/s00330-021-08074-7] [Citation(s) in RCA: 11] [Impact Index Per Article: 3.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/02/2021] [Revised: 03/20/2021] [Accepted: 05/17/2021] [Indexed: 01/17/2023]
Abstract
OBJECTIVE Assess if deep learning-based artificial intelligence (AI) algorithm improves reader performance for lung cancer detection on chest X-rays (CXRs). METHODS This reader study included 173 images from cancer-positive patients (n = 98) and 346 images from cancer-negative patients (n = 196) selected from National Lung Screening Trial (NLST). Eight readers, including three radiology residents, and five board-certified radiologists, participated in the observer performance test. AI algorithm provided image-level probability of pulmonary nodule or mass on CXRs and a heatmap of detected lesions. Reader performance was compared with AUC, sensitivity, specificity, false-positives per image (FPPI), and rates of chest CT recommendations. RESULTS With AI, the average sensitivity of readers for the detection of visible lung cancer increased for residents, but was similar for radiologists compared to that without AI (0.61 [95% CI, 0.55-0.67] vs. 0.72 [95% CI, 0.66-0.77], p = 0.016 for residents, and 0.76 [95% CI, 0.72-0.81] vs. 0.76 [95% CI, 0.72-0.81, p = 1.00 for radiologists), while false-positive findings per image (FPPI) was similar for residents, but decreased for radiologists (0.15 [95% CI, 0.11-0.18] vs. 0.12 [95% CI, 0.09-0.16], p = 0.13 for residents, and 0.24 [95% CI, 0.20-0.29] vs. 0.17 [95% CI, 0.13-0.20], p < 0.001 for radiologists). With AI, the average rate of chest CT recommendation in patients positive for visible cancer increased for residents, but was similar for radiologists (54.7% [95% CI, 48.2-61.2%] vs. 70.2% [95% CI, 64.2-76.2%], p < 0.001 for residents and 72.5% [95% CI, 68.0-77.1%] vs. 73.9% [95% CI, 69.4-78.3%], p = 0.68 for radiologists), while that in cancer-negative patients was similar for residents, but decreased for radiologists (11.2% [95% CI, 9.6-13.1%] vs. 9.8% [95% CI, 8.0-11.6%], p = 0.32 for residents and 16.4% [95% CI, 14.7-18.2%] vs. 11.7% [95% CI, 10.2-13.3%], p < 0.001 for radiologists). CONCLUSIONS AI algorithm can enhance the performance of readers for the detection of lung cancers on chest radiographs when used as second reader. KEY POINTS • Reader study in the NLST dataset shows that AI algorithm had sensitivity benefit for residents and specificity benefit for radiologists for the detection of visible lung cancer. • With AI, radiology residents were able to recommend more chest CT examinations (54.7% vs 70.2%, p < 0.001) for patients with visible lung cancer. • With AI, radiologists recommended significantly less proportion of unnecessary chest CT examinations (16.4% vs. 11.7%, p < 0.001) in cancer-negative patients.
Collapse
Affiliation(s)
| | | | - Chiara Daniela Arru
- Division of Thoracic Imaging, Department of Radiology, Massachusetts General Hospital, 75 Blossom Court, Boston, MA, 02114, USA.,Harvard Medical School, Boston, MA, USA
| | - Ruhani Doda Khera
- Division of Thoracic Imaging, Department of Radiology, Massachusetts General Hospital, 75 Blossom Court, Boston, MA, 02114, USA.,Harvard Medical School, Boston, MA, USA
| | - Ramandeep Singh
- Division of Thoracic Imaging, Department of Radiology, Massachusetts General Hospital, 75 Blossom Court, Boston, MA, 02114, USA.,Harvard Medical School, Boston, MA, USA
| | - Sean Siebert
- Division of Thoracic Imaging, Department of Radiology, Massachusetts General Hospital, 75 Blossom Court, Boston, MA, 02114, USA.,Harvard Medical School, Boston, MA, USA
| | - Dohoon Kim
- Department of Radiology, Seoul National University College of Medicine, Seoul, Korea
| | - Yuna Lee
- Department of Radiology, Seoul National University College of Medicine, Seoul, Korea
| | - Ju Hyun Park
- Suwon Total Healthcare Center, Kangbuk Samsung Hospital, Sungkyunkwan University School of Medicine, Youngin-si, Gyeongi-do, 16954, Korea
| | - Hye Joung Eom
- Cheju Halla General Hospital, 65 Doryeong-ro, Yeon-dong, Jeju-si, Jeju-do, Korea
| | - Subba R Digumarthy
- Division of Thoracic Imaging, Department of Radiology, Massachusetts General Hospital, 75 Blossom Court, Boston, MA, 02114, USA.,Harvard Medical School, Boston, MA, USA
| | - Mannudeep K Kalra
- Division of Thoracic Imaging, Department of Radiology, Massachusetts General Hospital, 75 Blossom Court, Boston, MA, 02114, USA. .,Harvard Medical School, Boston, MA, USA.
| |
Collapse
|
35
|
Song H, Yoon SH, Kim J, Kim J, Lee KW, Lee W, Lee S, Kim K, Lee CT, Chung JH, Lee KH. Application of N Descriptors Proposed by the International Association for the Study of Lung Cancer in Clinical Staging. Radiology 2021; 300:450-457. [PMID: 34060941 DOI: 10.1148/radiol.2021204461] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/11/2022]
Abstract
Background Patients with N1 or N2 non-small cell lung cancer exhibit prognostic heterogeneity. To refine the current N staging system, new N stages were proposed by the International Association for the Study of Lung Cancer. However, those proposed new N stages have not been validated. Purpose To evaluate the prognostic performance of the proposed N descriptors for clinical staging. Materials and Methods Participants with non-small cell lung cancer without distant metastasis from January 2010 to December 2014 were retrospectively included. Each patient's clinical N (cN) stage was assigned to one of seven categories (cN0, cN1a, cN1b, cN2a1, cN2a2, cN2b, cN3). The 5-year overall survival rates were estimated with the Kaplan-Meier method. The adjusted hazard ratios (HRs) and their 95% CIs were estimated by using a multivariable Cox proportional hazard model. Ad hoc analyses according to lymph node (LN) size were performed. Results A total of 1271 patients (median age, 66 years; interquartile range, 59-73 years; 812 men) were included. The 5-year overall survival rates were 77.3%, 53.7%, 36.0%, 29.2%, 34.4%, 18.0%, and 12.4% for stages cN0, cN1a, cN1b, cN2a1, cN2a2, cN2b, and cN3, respectively. Patients with cN2b disease had a worse prognosis than patients with cN2a disease (HR, 1.53; 95% CI: 1.06, 2.22; P = .02). There was no prognostic difference between cN1b and cN1a (HR, 1.13; 95% CI: 0.61, 2.09; P = .71); however, there was a difference between cN1 subgroups when stratified by LN size (≥2 cm; HR, 2.26; 95% CI: 1.16, 4.44; P = .02). Within cN2a disease, there were no differences between cN2a1 and cN2a2 (HR, 0.98; 95% CI: 0.61, 1.56; P = .93) or between subgroups according to LN size (HR, 0.74; 95% CI: 0.40, 1.37; P = .34). Conclusion A survival difference was observed between single- and multistation involvement among cN2 disease. The number of involved lymph node stations in patients with cN1 disease and the presence of skip metastasis in patients with cN2 disease were not associated with survival differences. © RSNA, 2021 Online supplemental material is available for this article.
Collapse
Affiliation(s)
- Hwayoung Song
- From the Department of Radiology (H.S., S.H.Y., Junghoon Kim, Jihang Kim, K.W.L., K.H.L.), Department of Thoracic and Cardiovascular Surgery (K.K.), Division of Pulmonology and Critical Care Medicine, Department of Internal Medicine (C.T.L.), and Department of Pathology and Translational Medicine (J.H.C.), Seoul National University Bundang Hospital, Seoul National University College of Medicine, 82 Gumi-ro 173 Beon-gil, Bundang-gu, Seongnam 13620, Korea; and Departments of Public Health Science, Graduate School of Public Health (W.L.) and Applied Bioengineering, Graduate School of Convergence Science and Technology (S.L.), Seoul National University, Seoul, Korea
| | - Sung Hyun Yoon
- From the Department of Radiology (H.S., S.H.Y., Junghoon Kim, Jihang Kim, K.W.L., K.H.L.), Department of Thoracic and Cardiovascular Surgery (K.K.), Division of Pulmonology and Critical Care Medicine, Department of Internal Medicine (C.T.L.), and Department of Pathology and Translational Medicine (J.H.C.), Seoul National University Bundang Hospital, Seoul National University College of Medicine, 82 Gumi-ro 173 Beon-gil, Bundang-gu, Seongnam 13620, Korea; and Departments of Public Health Science, Graduate School of Public Health (W.L.) and Applied Bioengineering, Graduate School of Convergence Science and Technology (S.L.), Seoul National University, Seoul, Korea
| | - Junghoon Kim
- From the Department of Radiology (H.S., S.H.Y., Junghoon Kim, Jihang Kim, K.W.L., K.H.L.), Department of Thoracic and Cardiovascular Surgery (K.K.), Division of Pulmonology and Critical Care Medicine, Department of Internal Medicine (C.T.L.), and Department of Pathology and Translational Medicine (J.H.C.), Seoul National University Bundang Hospital, Seoul National University College of Medicine, 82 Gumi-ro 173 Beon-gil, Bundang-gu, Seongnam 13620, Korea; and Departments of Public Health Science, Graduate School of Public Health (W.L.) and Applied Bioengineering, Graduate School of Convergence Science and Technology (S.L.), Seoul National University, Seoul, Korea
| | - Jihang Kim
- From the Department of Radiology (H.S., S.H.Y., Junghoon Kim, Jihang Kim, K.W.L., K.H.L.), Department of Thoracic and Cardiovascular Surgery (K.K.), Division of Pulmonology and Critical Care Medicine, Department of Internal Medicine (C.T.L.), and Department of Pathology and Translational Medicine (J.H.C.), Seoul National University Bundang Hospital, Seoul National University College of Medicine, 82 Gumi-ro 173 Beon-gil, Bundang-gu, Seongnam 13620, Korea; and Departments of Public Health Science, Graduate School of Public Health (W.L.) and Applied Bioengineering, Graduate School of Convergence Science and Technology (S.L.), Seoul National University, Seoul, Korea
| | - Kyoung Won Lee
- From the Department of Radiology (H.S., S.H.Y., Junghoon Kim, Jihang Kim, K.W.L., K.H.L.), Department of Thoracic and Cardiovascular Surgery (K.K.), Division of Pulmonology and Critical Care Medicine, Department of Internal Medicine (C.T.L.), and Department of Pathology and Translational Medicine (J.H.C.), Seoul National University Bundang Hospital, Seoul National University College of Medicine, 82 Gumi-ro 173 Beon-gil, Bundang-gu, Seongnam 13620, Korea; and Departments of Public Health Science, Graduate School of Public Health (W.L.) and Applied Bioengineering, Graduate School of Convergence Science and Technology (S.L.), Seoul National University, Seoul, Korea
| | - Woojoo Lee
- From the Department of Radiology (H.S., S.H.Y., Junghoon Kim, Jihang Kim, K.W.L., K.H.L.), Department of Thoracic and Cardiovascular Surgery (K.K.), Division of Pulmonology and Critical Care Medicine, Department of Internal Medicine (C.T.L.), and Department of Pathology and Translational Medicine (J.H.C.), Seoul National University Bundang Hospital, Seoul National University College of Medicine, 82 Gumi-ro 173 Beon-gil, Bundang-gu, Seongnam 13620, Korea; and Departments of Public Health Science, Graduate School of Public Health (W.L.) and Applied Bioengineering, Graduate School of Convergence Science and Technology (S.L.), Seoul National University, Seoul, Korea
| | - Seungjae Lee
- From the Department of Radiology (H.S., S.H.Y., Junghoon Kim, Jihang Kim, K.W.L., K.H.L.), Department of Thoracic and Cardiovascular Surgery (K.K.), Division of Pulmonology and Critical Care Medicine, Department of Internal Medicine (C.T.L.), and Department of Pathology and Translational Medicine (J.H.C.), Seoul National University Bundang Hospital, Seoul National University College of Medicine, 82 Gumi-ro 173 Beon-gil, Bundang-gu, Seongnam 13620, Korea; and Departments of Public Health Science, Graduate School of Public Health (W.L.) and Applied Bioengineering, Graduate School of Convergence Science and Technology (S.L.), Seoul National University, Seoul, Korea
| | - Kwhanmien Kim
- From the Department of Radiology (H.S., S.H.Y., Junghoon Kim, Jihang Kim, K.W.L., K.H.L.), Department of Thoracic and Cardiovascular Surgery (K.K.), Division of Pulmonology and Critical Care Medicine, Department of Internal Medicine (C.T.L.), and Department of Pathology and Translational Medicine (J.H.C.), Seoul National University Bundang Hospital, Seoul National University College of Medicine, 82 Gumi-ro 173 Beon-gil, Bundang-gu, Seongnam 13620, Korea; and Departments of Public Health Science, Graduate School of Public Health (W.L.) and Applied Bioengineering, Graduate School of Convergence Science and Technology (S.L.), Seoul National University, Seoul, Korea
| | - Choon-Taek Lee
- From the Department of Radiology (H.S., S.H.Y., Junghoon Kim, Jihang Kim, K.W.L., K.H.L.), Department of Thoracic and Cardiovascular Surgery (K.K.), Division of Pulmonology and Critical Care Medicine, Department of Internal Medicine (C.T.L.), and Department of Pathology and Translational Medicine (J.H.C.), Seoul National University Bundang Hospital, Seoul National University College of Medicine, 82 Gumi-ro 173 Beon-gil, Bundang-gu, Seongnam 13620, Korea; and Departments of Public Health Science, Graduate School of Public Health (W.L.) and Applied Bioengineering, Graduate School of Convergence Science and Technology (S.L.), Seoul National University, Seoul, Korea
| | - Jin-Haeng Chung
- From the Department of Radiology (H.S., S.H.Y., Junghoon Kim, Jihang Kim, K.W.L., K.H.L.), Department of Thoracic and Cardiovascular Surgery (K.K.), Division of Pulmonology and Critical Care Medicine, Department of Internal Medicine (C.T.L.), and Department of Pathology and Translational Medicine (J.H.C.), Seoul National University Bundang Hospital, Seoul National University College of Medicine, 82 Gumi-ro 173 Beon-gil, Bundang-gu, Seongnam 13620, Korea; and Departments of Public Health Science, Graduate School of Public Health (W.L.) and Applied Bioengineering, Graduate School of Convergence Science and Technology (S.L.), Seoul National University, Seoul, Korea
| | - Kyung Hee Lee
- From the Department of Radiology (H.S., S.H.Y., Junghoon Kim, Jihang Kim, K.W.L., K.H.L.), Department of Thoracic and Cardiovascular Surgery (K.K.), Division of Pulmonology and Critical Care Medicine, Department of Internal Medicine (C.T.L.), and Department of Pathology and Translational Medicine (J.H.C.), Seoul National University Bundang Hospital, Seoul National University College of Medicine, 82 Gumi-ro 173 Beon-gil, Bundang-gu, Seongnam 13620, Korea; and Departments of Public Health Science, Graduate School of Public Health (W.L.) and Applied Bioengineering, Graduate School of Convergence Science and Technology (S.L.), Seoul National University, Seoul, Korea
| |
Collapse
|