1
|
Cahan N, Klang E, Marom EM, Soffer S, Barash Y, Burshtein E, Konen E, Greenspan H. Multimodal fusion models for pulmonary embolism mortality prediction. Sci Rep 2023; 13:7544. [PMID: 37160926 PMCID: PMC10170065 DOI: 10.1038/s41598-023-34303-8] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/22/2022] [Accepted: 04/27/2023] [Indexed: 05/11/2023] Open
Abstract
Pulmonary embolism (PE) is a common, life threatening cardiovascular emergency. Risk stratification is one of the core principles of acute PE management and determines the choice of diagnostic and therapeutic strategies. In routine clinical practice, clinicians rely on the patient's electronic health record (EHR) to provide a context for their medical imaging interpretation. Most deep learning models for radiology applications only consider pixel-value information without the clinical context. Only a few integrate both clinical and imaging data. In this work, we develop and compare multimodal fusion models that can utilize multimodal data by combining both volumetric pixel data and clinical patient data for automatic risk stratification of PE. Our best performing model is an intermediate fusion model that incorporates both bilinear attention and TabNet, and can be trained in an end-to-end manner. The results show that multimodality boosts performance by up to 14% with an area under the curve (AUC) of 0.96 for assessing PE severity, with a sensitivity of 90% and specificity of 94%, thus pointing to the value of using multimodal data to automatically assess PE severity.
Collapse
Affiliation(s)
- Noa Cahan
- Department of Biomedical Engineering, Tel-Aviv University, Tel Aviv, Israel.
| | - Eyal Klang
- Department of Diagnostic Imaging, Sheba Medical Center, Ramat Gan, Israel affiliated with the Tel Aviv University, Tel Aviv, Israel
| | - Edith M Marom
- Department of Diagnostic Imaging, Sheba Medical Center, Ramat Gan, Israel affiliated with the Tel Aviv University, Tel Aviv, Israel
| | - Shelly Soffer
- Department of Diagnostic Imaging, Sheba Medical Center, Ramat Gan, Israel affiliated with the Tel Aviv University, Tel Aviv, Israel
| | - Yiftach Barash
- Department of Diagnostic Imaging, Sheba Medical Center, Ramat Gan, Israel affiliated with the Tel Aviv University, Tel Aviv, Israel
| | - Evyatar Burshtein
- Department of Biomedical Engineering, Tel-Aviv University, Tel Aviv, Israel
| | - Eli Konen
- Department of Diagnostic Imaging, Sheba Medical Center, Ramat Gan, Israel affiliated with the Tel Aviv University, Tel Aviv, Israel
| | - Hayit Greenspan
- Department of Biomedical Engineering, Tel-Aviv University, Tel Aviv, Israel.
- Biomedical Engineering and Imaging Institute, Radiology Dept., Icahn School of Medicine at Mount Sinai, New York, United States.
| |
Collapse
|