1
|
Ikenoue T, Kataoka Y, Matsuoka Y, Matsumoto J, Kumasawa J, Tochitatni K, Funakoshi H, Hosoda T, Kugimiya A, Shirano M, Hamabe F, Iwata S, Fukuma S. Accuracy of deep learning-based computed tomography diagnostic system for COVID-19: A consecutive sampling external validation cohort study. PLoS One 2021; 16:e0258760. [PMID: 34735458 PMCID: PMC8568139 DOI: 10.1371/journal.pone.0258760] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/04/2021] [Accepted: 10/06/2021] [Indexed: 11/18/2022] Open
Abstract
Ali-M3, an artificial intelligence program, analyzes chest computed tomography (CT) and detects the likelihood of coronavirus disease (COVID-19) based on scores ranging from 0 to 1. However, Ali-M3 has not been externally validated. Our aim was to evaluate the accuracy of Ali-M3 for detecting COVID-19 and discuss its clinical value. We evaluated the external validity of Ali-M3 using sequential Japanese sampling data. In this retrospective cohort study, COVID-19 infection probabilities for 617 symptomatic patients were determined using Ali-M3. In 11 Japanese tertiary care facilities, these patients underwent reverse transcription-polymerase chain reaction (RT-PCR) testing. They also underwent chest CT to confirm a diagnosis of COVID-19. Of the 617 patients, 289 (46.8%) were RT-PCR-positive. The area under the curve (AUC) of Ali-M3 for predicting a COVID-19 diagnosis was 0.797 (95% confidence interval: 0.762‒0.833) and the goodness-of-fit was P = 0.156. With a cut-off probability of a diagnosis of COVID-19 by Ali-M3 set at 0.5, the sensitivity and specificity were 80.6% and 68.3%, respectively. A cut-off of 0.2 yielded a sensitivity and specificity of 89.2% and 43.2%, respectively. Among the 223 patients who required oxygen, the AUC was 0.825. Sensitivity at a cut-off of 0.5% and 0.2% was 88.7% and 97.9%, respectively. Although the sensitivity was lower when the days from symptom onset were fewer, the sensitivity increased for both cut-off values after 5 days. We evaluated Ali-M3 using external validation with symptomatic patient data from Japanese tertiary care facilities. As Ali-M3 showed sufficient sensitivity performance, despite a lower specificity performance, Ali-M3 could be useful in excluding a diagnosis of COVID-19.
Collapse
Affiliation(s)
- Tatsuyoshi Ikenoue
- Human Health Sciences, Kyoto University Graduate School of Medicine, Kyoto, Japan
| | - Yuki Kataoka
- Hospital Care Research Unit, Hyogo Prefectural Amagasaki General Medical Center, Amagasaki, Hyogo, Japan
- Department of Respiratory Medicine, Hyogo Prefectural Amagasaki General Medical Center, Amagasaki, Hyogo, Japan
| | - Yoshinori Matsuoka
- Department of Emergency Medicine, Kobe City Medical Center General Hospital, Kobe City, Hyogo, Japan
| | - Junichi Matsumoto
- Department of Emergency and Critical Care Medicine, St. Marianna University School of Medicine, Kawasaki, Kanagawa, Japan
| | - Junji Kumasawa
- Department of Critical Care Medicine, Sakai City Medical Center, Sakai, Osaka, Japan
| | - Kentaro Tochitatni
- Department of Infectious Diseases, Kyoto City Hospital, Kyoto-city, Kyoto, Japan
| | - Hiraku Funakoshi
- Department of Emergency and Critical Care Medicine, Tokyobay Urayasu Ichikawa Medical Center, Urayasu, Chiba, Japan
| | - Tomohiro Hosoda
- Department of Infectious Disease, Kawasaki Municipal Kawasaki Hospital, Kawasaki-ku, Kawasaki Kanagawa, Japan
| | - Aiko Kugimiya
- Department of Emergency and Critical Care Medicine, Yamanashi Prefectural Central Hospital, Kofu, Yamanashi, Japan
| | - Michinori Shirano
- Department of Infectious Diseases, Osaka City General Hospital, Osaka, Japan
| | - Fumiko Hamabe
- Department of Radiology, National Defense Medical College Hospital, Tokorozawa, Saitama, Japan
| | - Sachiyo Iwata
- Division of Cardiovascular Medicine, Hyogo Prefectural Kakogawa Medical Center, Kakogawa, Japan
| | - Shingo Fukuma
- Human Health Sciences, Kyoto University Graduate School of Medicine, Kyoto, Japan
| | | |
Collapse
|