1
|
Blankemeier L, Cohen JP, Kumar A, Van Veen D, Gardezi SJS, Paschali M, Chen Z, Delbrouck JB, Reis E, Truyts C, Bluethgen C, Jensen MEK, Ostmeier S, Varma M, Valanarasu JMJ, Fang Z, Huo Z, Nabulsi Z, Ardila D, Weng WH, Amaro E, Ahuja N, Fries J, Shah NH, Johnston A, Boutin RD, Wentland A, Langlotz CP, Hom J, Gatidis S, Chaudhari AS. Merlin: A Vision Language Foundation Model for 3D Computed Tomography. RESEARCH SQUARE 2024:rs.3.rs-4546309. [PMID: 38978576 PMCID: PMC11230513 DOI: 10.21203/rs.3.rs-4546309/v1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 07/10/2024]
Abstract
Over 85 million computed tomography (CT) scans are performed annually in the US, of which approximately one quarter focus on the abdomen. Given the current shortage of both general and specialized radiologists, there is a large impetus to use artificial intelligence to alleviate the burden of interpreting these complex imaging studies while simultaneously using the images to extract novel physiological insights. Prior state-of-the-art approaches for automated medical image interpretation leverage vision language models (VLMs) that utilize both the image and the corresponding textual radiology reports. However, current medical VLMs are generally limited to 2D images and short reports. To overcome these shortcomings for abdominal CT interpretation, we introduce Merlin - a 3D VLM that leverages both structured electronic health records (EHR) and unstructured radiology reports for pretraining without requiring additional manual annotations. We train Merlin using a high-quality clinical dataset of paired CT scans (6+ million images from 15,331 CTs), EHR diagnosis codes (1.8+ million codes), and radiology reports (6+ million tokens) for training. We comprehensively evaluate Merlin on 6 task types and 752 individual tasks. The non-adapted (off-the-shelf) tasks include zero-shot findings classification (31 findings), phenotype classification (692 phenotypes), and zero-shot cross-modal retrieval (image to findings and image to impressions), while model adapted tasks include 5-year chronic disease prediction (6 diseases), radiology report generation, and 3D semantic segmentation (20 organs). We perform internal validation on a test set of 5,137 CTs, and external validation on 7,000 clinical CTs and on two public CT datasets (VerSe, TotalSegmentator). Beyond these clinically-relevant evaluations, we assess the efficacy of various network architectures and training strategies to depict that Merlin has favorable performance to existing task-specific baselines. We derive data scaling laws to empirically assess training data needs for requisite downstream task performance. Furthermore, unlike conventional VLMs that require hundreds of GPUs for training, we perform all training on a single GPU. This computationally efficient design can help democratize foundation model training, especially for health systems with compute constraints. We plan to release our trained models, code, and dataset, pending manual removal of all protected health information.
Collapse
Affiliation(s)
- Louis Blankemeier
- Department of Electrical Engineering, Stanford University
- Stanford Center for Artificial Intelligence in Medicine and Imaging, Stanford University
- Department of Radiology, Stanford University
| | - Joseph Paul Cohen
- Stanford Center for Artificial Intelligence in Medicine and Imaging, Stanford University
| | - Ashwin Kumar
- Stanford Center for Artificial Intelligence in Medicine and Imaging, Stanford University
- Department of Radiology, Stanford University
| | - Dave Van Veen
- Department of Electrical Engineering, Stanford University
- Stanford Center for Artificial Intelligence in Medicine and Imaging, Stanford University
- Department of Radiology, Stanford University
| | | | - Magdalini Paschali
- Stanford Center for Artificial Intelligence in Medicine and Imaging, Stanford University
- Department of Radiology, Stanford University
| | - Zhihong Chen
- Stanford Center for Artificial Intelligence in Medicine and Imaging, Stanford University
- Department of Radiology, Stanford University
| | - Jean-Benoit Delbrouck
- Stanford Center for Artificial Intelligence in Medicine and Imaging, Stanford University
- Department of Radiology, Stanford University
| | - Eduardo Reis
- Stanford Center for Artificial Intelligence in Medicine and Imaging, Stanford University
- Department of Radiology, Stanford University
| | - Cesar Truyts
- Department of Radiology, Hospital Israelita Albert Einstein
| | - Christian Bluethgen
- Stanford Center for Artificial Intelligence in Medicine and Imaging, Stanford University
- Department of Radiology, University Hospital Zurich
| | - Malte Engmann Kjeldskov Jensen
- Stanford Center for Artificial Intelligence in Medicine and Imaging, Stanford University
- Department of Radiology, Stanford University
| | - Sophie Ostmeier
- Stanford Center for Artificial Intelligence in Medicine and Imaging, Stanford University
- Department of Radiology, Stanford University
| | - Maya Varma
- Stanford Center for Artificial Intelligence in Medicine and Imaging, Stanford University
- Department of Radiology, Stanford University
- Department of Computer Science, Stanford University
| | - Jeya Maria Jose Valanarasu
- Stanford Center for Artificial Intelligence in Medicine and Imaging, Stanford University
- Department of Radiology, Stanford University
- Department of Computer Science, Stanford University
| | | | - Zepeng Huo
- Department of Biomedical Data Science, Stanford University
| | | | | | | | - Edson Amaro
- Department of Radiology, Hospital Israelita Albert Einstein
| | | | - Jason Fries
- Department of Computer Science, Stanford University
- Department of Biomedical Data Science, Stanford University
| | - Nigam H. Shah
- Department of Radiology, Stanford University
- Department of Biomedical Data Science, Stanford University
| | | | | | | | - Curtis P. Langlotz
- Stanford Center for Artificial Intelligence in Medicine and Imaging, Stanford University
- Department of Radiology, Stanford University
| | - Jason Hom
- Department of Medicine, Stanford University
| | | | - Akshay S. Chaudhari
- Stanford Center for Artificial Intelligence in Medicine and Imaging, Stanford University
- Department of Radiology, Stanford University
- Department of Biomedical Data Science, Stanford University
| |
Collapse
|
2
|
Zambrano Chaves JM, Lenchik L, Gallegos IO, Blankemeier L, Rubin DL, Willis MH, Chaudhari AS, Boutin RD. Abdominal CT metrics in 17,646 patients reveal associations between myopenia, myosteatosis, and medical phenotypes: a phenome-wide association study. EBioMedicine 2024; 103:105116. [PMID: 38636199 PMCID: PMC11031722 DOI: 10.1016/j.ebiom.2024.105116] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/21/2023] [Revised: 03/29/2024] [Accepted: 03/30/2024] [Indexed: 04/20/2024] Open
Abstract
BACKGROUND Deep learning facilitates large-scale automated imaging evaluation of body composition. However, associations of body composition biomarkers with medical phenotypes have been underexplored. Phenome-wide association study (PheWAS) techniques search for medical phenotypes associated with biomarkers. A PheWAS integrating large-scale analysis of imaging biomarkers and electronic health record (EHR) data could discover previously unreported associations and validate expected associations. Here we use PheWAS methodology to determine the association of abdominal CT-based skeletal muscle metrics with medical phenotypes in a large North American cohort. METHODS An automated deep learning pipeline was used to measure skeletal muscle index (SMI; biomarker of myopenia) and skeletal muscle density (SMD; biomarker of myosteatosis) from abdominal CT scans of adults between 2012 and 2018. A PheWAS was performed with logistic regression using patient sex and age as covariates to assess for associations between CT-derived muscle metrics and 611 common EHR-derived medical phenotypes. PheWAS P values were considered significant at a Bonferroni corrected threshold (α = 0.05/1222). FINDINGS 17,646 adults (mean age, 56 years ± 19 [SD]; 57.5% women) were included. CT-derived SMI was significantly associated with 268 medical phenotypes; SMD with 340 medical phenotypes. Previously unreported associations with the highest magnitude of significance included higher SMI with decreased cardiac dysrhythmias (OR [95% CI], 0.59 [0.55-0.64]; P < 0.0001), decreased epilepsy (OR, 0.59 [0.50-0.70]; P < 0.0001), and increased elevated prostate-specific antigen (OR, 1.84 [1.47-2.31]; P < 0.0001), and higher SMD with decreased decubitus ulcers (OR, 0.36 [0.31-0.42]; P < 0.0001), sleep disorders (OR, 0.39 [0.32-0.47]; P < 0.0001), and osteomyelitis (OR, 0.43 [0.36-0.52]; P < 0.0001). INTERPRETATION PheWAS methodology reveals previously unreported associations between CT-derived biomarkers of myopenia and myosteatosis and EHR medical phenotypes. The high-throughput PheWAS technique applied on a population scale can generate research hypotheses related to myopenia and myosteatosis and can be adapted to research possible associations of other imaging biomarkers with hundreds of EHR medical phenotypes. FUNDING National Institutes of Health, Stanford AIMI-HAI pilot grant, Stanford Precision Health and Integrated Diagnostics, Stanford Cardiovascular Institute, Stanford Center for Digital Health, and Stanford Knight-Hennessy Scholars.
Collapse
Affiliation(s)
- Juan M Zambrano Chaves
- Department of Biomedical Data Science, Stanford University School of Medicine, Stanford, CA, USA; Department of Radiology, Stanford University, Stanford, CA, USA
| | - Leon Lenchik
- Department of Diagnostic Radiology, Wake Forest University School of Medicine, Winston-Salem, North Carolina, USA
| | - Isabel O Gallegos
- Department of Computer Science, (IOG), Stanford University, Stanford, CA, USA
| | - Louis Blankemeier
- Department of Electrical Engineering, Stanford University, Stanford, CA, USA
| | - Daniel L Rubin
- Department of Biomedical Data Science, Stanford University School of Medicine, Stanford, CA, USA; Department of Radiology, Stanford University, Stanford, CA, USA
| | - Marc H Willis
- Department of Radiology, Stanford University, Stanford, CA, USA
| | - Akshay S Chaudhari
- Department of Biomedical Data Science, Stanford University School of Medicine, Stanford, CA, USA; Department of Radiology, Stanford University, Stanford, CA, USA
| | - Robert D Boutin
- Department of Radiology, Stanford University, Stanford, CA, USA.
| |
Collapse
|
3
|
Milosevic M, Jin Q, Singh A, Amal S. Applications of AI in multi-modal imaging for cardiovascular disease. FRONTIERS IN RADIOLOGY 2024; 3:1294068. [PMID: 38283302 PMCID: PMC10811170 DOI: 10.3389/fradi.2023.1294068] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 09/14/2023] [Accepted: 12/22/2023] [Indexed: 01/30/2024]
Abstract
Data for healthcare is diverse and includes many different modalities. Traditional approaches to Artificial Intelligence for cardiovascular disease were typically limited to single modalities. With the proliferation of diverse datasets and new methods in AI, we are now able to integrate different modalities, such as magnetic resonance scans, computerized tomography scans, echocardiography, x-rays, and electronic health records. In this paper, we review research from the last 5 years in applications of AI to multi-modal imaging. There have been many promising results in registration, segmentation, and fusion of different magnetic resonance imaging modalities with each other and computer tomography scans, but there are still many challenges that need to be addressed. Only a few papers have addressed modalities such as x-ray, echocardiography, or non-imaging modalities. As for prediction or classification tasks, there have only been a couple of papers that use multiple modalities in the cardiovascular domain. Furthermore, no models have been implemented or tested in real world cardiovascular clinical settings.
Collapse
Affiliation(s)
- Marko Milosevic
- Roux Institute, Northeastern University, Portland, ME, United States
| | - Qingchu Jin
- Roux Institute, Northeastern University, Portland, ME, United States
| | - Akarsh Singh
- College of Engineering, Northeastern University, Boston, MA, United States
| | - Saeed Amal
- Roux Institute, Northeastern University, Portland, ME, United States
| |
Collapse
|