1
|
Kim H, Kim P, Joo I, Kim JH, Park CM, Yoon SH. ChatGPT Vision for Radiological Interpretation: An Investigation Using Medical School Radiology Examinations. Korean J Radiol 2024; 25:403-406. [PMID: 38528699 PMCID: PMC10973733 DOI: 10.3348/kjr.2024.0017] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/09/2024] [Revised: 01/11/2024] [Accepted: 01/14/2024] [Indexed: 03/27/2024] Open
Affiliation(s)
- Hyungjin Kim
- Department of Radiology, Seoul National University Hospital, Seoul National University College of Medicine, Seoul, Republic of Korea
| | - Paul Kim
- Graduate School of Education, Stanford University, Stanford, CA, USA
| | - Ijin Joo
- Department of Radiology, Seoul National University Hospital, Seoul National University College of Medicine, Seoul, Republic of Korea
| | - Jung Hoon Kim
- Department of Radiology, Seoul National University Hospital, Seoul National University College of Medicine, Seoul, Republic of Korea
| | - Chang Min Park
- Department of Radiology, Seoul National University Hospital, Seoul National University College of Medicine, Seoul, Republic of Korea
| | - Soon Ho Yoon
- Department of Radiology, Seoul National University Hospital, Seoul National University College of Medicine, Seoul, Republic of Korea.
| |
Collapse
|
2
|
He F, Fei R, Gao M, Su L, Zhang X, Xu D. Parameter-Efficient Fine-Tuning Enhances Adaptation of Single Cell Large Language Model for Cell Type Identification. bioRxiv 2024:2024.01.27.577455. [PMID: 38352605 PMCID: PMC10862733 DOI: 10.1101/2024.01.27.577455] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Indexed: 02/21/2024]
Abstract
Single-cell sequencing transformed biology and medicine, providing an unprecedented high-resolution view at the cellular level. However, the vast variability inherent in single-cell sequencing data impedes its utility for in-depth downstream analysis. Inspired by the foundation models in natural language processing, recent advancements have led to the development of single-cell Large Language Models (scLLMs). These models are designed to discern universal patterns across diverse single-cell datasets, thereby enhancing the signal-to-noise ratio. Despite their potential, multiple studies indicate existing scLLMs do not perform well in zero-short settings, highlighting a pressing need for more effective adaptation techniques. This research proposes several adaptation techniques for scLLMs by preserving the original model parameters while selectively updating newly introduced tensors. This approach aims to overcome the limitations associated with traditional fine-tuning practices, such as catastrophic forgetting and computational inefficiencies. We introduce two Parameter-Efficient Fine-Tuning (PEFT) strategies specifically tailored to refine scLLMs for cell type identification. Our investigations utilizing scGPT demonstrate that PEFT can enhance performance, with the added benefit of up to a 90% reduction in parameter training compared to conventional fine-tuning methodologies. This work paves the way for a new direction in leveraging single-cell models with greater efficiency and efficacy in single-cell biology.
Collapse
Affiliation(s)
- Fei He
- Department of Electrical Engineering and Computer Science, Bond Life Sciences Center, University of Missouri, Columbia, MO, 65211, USA
| | - Ruixin Fei
- School of Information Science and Technology, Northeast Normal University, Changchun Jilin 130017, China
| | - Mingyue Gao
- School of Information Science and Technology, Northeast Normal University, Changchun Jilin 130017, China
| | - Li Su
- Department of Electrical Engineering and Computer Science, Bond Life Sciences Center, University of Missouri, Columbia, MO, 65211, USA
| | - Xinyu Zhang
- School of Information Science and Technology, Northeast Normal University, Changchun Jilin 130017, China
| | - Dong Xu
- Department of Electrical Engineering and Computer Science, Bond Life Sciences Center, University of Missouri, Columbia, MO, 65211, USA
| |
Collapse
|
3
|
Cui W, Akrami H, Zhao G, Joshi AA, Leahy RM. Meta Transfer of Self-Supervised Knowledge: Foundation Model in Action for Post-Traumatic Epilepsy Prediction. ArXiv 2023:arXiv:2312.14204v1. [PMID: 38196751 PMCID: PMC10775348] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Grants] [Download PDF] [Subscribe] [Scholar Register] [Indexed: 01/11/2024]
Abstract
Despite the impressive advancements achieved using deep-learning for functional brain activity analysis, the heterogeneity of functional patterns and scarcity of imaging data still pose challenges in tasks such as prediction of future onset of Post-Traumatic Epilepsy (PTE) from data acquired shortly after traumatic brain injury (TBI). Foundation models pre-trained on separate large-scale datasets can improve the performance from scarce and heterogeneous datasets. For functional Magnetic Resonance Imaging (fMRI), while data may be abundantly available from healthy controls, clinical data is often scarce, limiting the ability of foundation models to identify clinically-relevant features. We overcome this limitation by introducing a novel training strategy for our foundation model by integrating meta-learning with self-supervised learning to improve the generalization from normal to clinical features. In this way we enable generalization to other downstream clinical tasks, in our case prediction of PTE. To achieve this, we perform self-supervised training on the control dataset to focus on inherent features that are not limited to a particular supervised task while applying meta-learning, which strongly improves the model's generalizability using bi-level optimization. Through experiments on neurological disorder classification tasks, we demonstrate that the proposed strategy significantly improves task performance on small-scale clinical datasets. To explore the generalizability of the foundation model in downstream applications, we then apply the model to an unseen TBI dataset for prediction of PTE using zero-shot learning. Results further demonstrated the enhanced generalizability of our foundation model.
Collapse
Affiliation(s)
- Wenhui Cui
- Ming Hsieh Department of Electrical and Computer Engineering, University of Southern California, Los Angeles 90089, United States
| | - Haleh Akrami
- Ming Hsieh Department of Electrical and Computer Engineering, University of Southern California, Los Angeles 90089, United States
| | - Ganning Zhao
- Ming Hsieh Department of Electrical and Computer Engineering, University of Southern California, Los Angeles 90089, United States
| | - Anand A. Joshi
- Ming Hsieh Department of Electrical and Computer Engineering, University of Southern California, Los Angeles 90089, United States
| | - Richard M. Leahy
- Ming Hsieh Department of Electrical and Computer Engineering, University of Southern California, Los Angeles 90089, United States
| |
Collapse
|
5
|
Wang EY, Fahey PG, Ponder K, Ding Z, Chang A, Muhammad T, Patel S, Ding Z, Tran D, Fu J, Papadopoulos S, Franke K, Ecker AS, Reimer J, Pitkow X, Sinz FH, Tolias AS. Towards a Foundation Model of the Mouse Visual Cortex. bioRxiv 2023:2023.03.21.533548. [PMID: 36993435 PMCID: PMC10055288 DOI: 10.1101/2023.03.21.533548] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Indexed: 05/04/2023]
Abstract
Understanding the brain's perception algorithm is a highly intricate problem, as the inherent complexity of sensory inputs and the brain's nonlinear processing make characterizing sensory representations difficult. Recent studies have shown that functional models-capable of predicting large-scale neuronal activity in response to arbitrary sensory input-can be powerful tools for characterizing neuronal representations by enabling high-throughput in silico experiments. However, accurately modeling responses to dynamic and ecologically relevant inputs like videos remains challenging, particularly when generalizing to new stimulus domains outside the training distribution. Inspired by recent breakthroughs in artificial intelligence, where foundation models-trained on vast quantities of data-have demonstrated remarkable capabilities and generalization, we developed a "foundation model" of the mouse visual cortex: a deep neural network trained on large amounts of neuronal responses to ecological videos from multiple visual cortical areas and mice. The model accurately predicted neuronal responses not only to natural videos but also to various new stimulus domains, such as coherent moving dots and noise patterns, underscoring its generalization abilities. The foundation model could also be adapted to new mice with minimal natural movie training data. We applied the foundation model to the MICrONS dataset: a study of the brain that integrates structure with function at unprecedented scale, containing nanometer-scale morphology, connectivity with >500,000,000 synapses, and function of >70,000 neurons within a ~1mm3 volume spanning multiple areas of the mouse visual cortex. This accurate functional model of the MICrONS data opens the possibility for a systematic characterization of the relationship between circuit structure and function. By precisely capturing the response properties of the visual cortex and generalizing to new stimulus domains and mice, foundation models can pave the way for a deeper understanding of visual computation.
Collapse
Affiliation(s)
- Eric Y Wang
- Center for Neuroscience and Artificial Intelligence, Baylor College of Medicine, Houston, USA
- Department of Neuroscience, Baylor College of Medicine, Houston, USA
| | - Paul G Fahey
- Center for Neuroscience and Artificial Intelligence, Baylor College of Medicine, Houston, USA
- Department of Neuroscience, Baylor College of Medicine, Houston, USA
| | - Kayla Ponder
- Center for Neuroscience and Artificial Intelligence, Baylor College of Medicine, Houston, USA
- Department of Neuroscience, Baylor College of Medicine, Houston, USA
| | - Zhuokun Ding
- Center for Neuroscience and Artificial Intelligence, Baylor College of Medicine, Houston, USA
- Department of Neuroscience, Baylor College of Medicine, Houston, USA
| | - Andersen Chang
- Center for Neuroscience and Artificial Intelligence, Baylor College of Medicine, Houston, USA
- Department of Neuroscience, Baylor College of Medicine, Houston, USA
| | - Taliah Muhammad
- Center for Neuroscience and Artificial Intelligence, Baylor College of Medicine, Houston, USA
- Department of Neuroscience, Baylor College of Medicine, Houston, USA
| | - Saumil Patel
- Center for Neuroscience and Artificial Intelligence, Baylor College of Medicine, Houston, USA
- Department of Neuroscience, Baylor College of Medicine, Houston, USA
| | - Zhiwei Ding
- Center for Neuroscience and Artificial Intelligence, Baylor College of Medicine, Houston, USA
- Department of Neuroscience, Baylor College of Medicine, Houston, USA
| | - Dat Tran
- Center for Neuroscience and Artificial Intelligence, Baylor College of Medicine, Houston, USA
- Department of Neuroscience, Baylor College of Medicine, Houston, USA
| | - Jiakun Fu
- Center for Neuroscience and Artificial Intelligence, Baylor College of Medicine, Houston, USA
- Department of Neuroscience, Baylor College of Medicine, Houston, USA
| | - Stelios Papadopoulos
- Center for Neuroscience and Artificial Intelligence, Baylor College of Medicine, Houston, USA
- Department of Neuroscience, Baylor College of Medicine, Houston, USA
| | - Katrin Franke
- Center for Neuroscience and Artificial Intelligence, Baylor College of Medicine, Houston, USA
- Department of Neuroscience, Baylor College of Medicine, Houston, USA
| | - Alexander S Ecker
- Institute for Computer Science, University Göttingen, Göttingen, Germany
- Max Planck Institute for Dynamics and Self-Organization, Göttingen, Germany
| | - Jacob Reimer
- Center for Neuroscience and Artificial Intelligence, Baylor College of Medicine, Houston, USA
- Department of Neuroscience, Baylor College of Medicine, Houston, USA
| | - Xaq Pitkow
- Center for Neuroscience and Artificial Intelligence, Baylor College of Medicine, Houston, USA
- Department of Neuroscience, Baylor College of Medicine, Houston, USA
- Department of Electrical and Computer Engineering, Rice University, Houston, TX, USA
| | - Fabian H Sinz
- Center for Neuroscience and Artificial Intelligence, Baylor College of Medicine, Houston, USA
- Department of Neuroscience, Baylor College of Medicine, Houston, USA
- Institute for Computer Science, University Göttingen, Göttingen, Germany
- Institute for Bioinformatics and Medical Informatics, University of Tübingen, Germany
| | - Andreas S Tolias
- Center for Neuroscience and Artificial Intelligence, Baylor College of Medicine, Houston, USA
- Department of Neuroscience, Baylor College of Medicine, Houston, USA
- Department of Electrical and Computer Engineering, Rice University, Houston, TX, USA
| |
Collapse
|