1
|
Sohn W, Di X, Liang Z, Zhang Z, Biswal BB. Explorations of using a convolutional neural network to understand brain activations during movie watching. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2024:2024.01.20.576341. [PMID: 38328194 PMCID: PMC10849516 DOI: 10.1101/2024.01.20.576341] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 02/09/2024]
Abstract
Neuroimaging studies increasingly use naturalistic stimuli like video clips to trigger complex brain activations, but the complexity of such stimuli makes it difficult to assign specific functions to the resulting brain activations, particularly for higher-level content like social interactions. To address this challenge, researchers have turned to deep neural networks, e.g., convolutional neural networks (CNNs). CNNs have shown success in image recognition due to their different levels of features enabling high performance. In this study, we used pre-trained VGG-16, a popular CNN model, to analyze video data and extract hierarchical features from low-level shallow layers to high-level deeper layers, linking these activations to different levels of activation of the human brain. We hypothesized that activations in different layers of VGG-16 would be associated with different levels of brain activation and visual processing hierarchy in the brain. We were also curious about which brain regions would be associated with deeper convolutional layers in VGG-16. The study analyzed a functional MRI (fMRI) dataset where participants watched the cartoon movie Partly Cloudy. Frames of the videos were fed into VGG-16, and activation maps from different kernels and layers were extracted. Time series of the average activation patterns for each kernel were created and fed into a voxel-wise model to study brain activations. Results showed that lower convolutional layers (1st convolutional layer) were mostly associated with lower visual regions, but some kernels (6, 19, 24, 42, 55, and 58) surprisingly showed associations with activations in the posterior cingulate cortex, part of the default mode network. Deeper convolutional layers were associated with more anterior and lateral portions of the visual cortex (e.g., the lateral occipital complex) and the supramarginal gyrus. Analyzing activation features associated with different brain regions showed the promise and limitations of using CNNs to link video content to brain functions.
Collapse
Affiliation(s)
- Wonbum Sohn
- Department of Biomedical Engineering, New Jersey Institute of Technology, Newark, NJ, 07029, USA
- Rutgers Biomedical and Health Sciences, Rutgers School of Graduate Studies, Newark, NJ, 07039, USA
| | - Xin Di
- Department of Biomedical Engineering, New Jersey Institute of Technology, Newark, NJ, 07029, USA
| | - Zhen Liang
- School of Biomedical Engineering, Health Science Center, Shenzhen University, Shenzhen, China, 518060
| | - Zhiguo Zhang
- School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen, 518060, China
| | - Bharat B. Biswal
- Department of Biomedical Engineering, New Jersey Institute of Technology, Newark, NJ, 07029, USA
| |
Collapse
|
2
|
Morgenroth E, Vilaclara L, Muszynski M, Gaviria J, Vuilleumier P, Van De Ville D. Probing neurodynamics of experienced emotions-a Hitchhiker's guide to film fMRI. Soc Cogn Affect Neurosci 2023; 18:nsad063. [PMID: 37930850 PMCID: PMC10656947 DOI: 10.1093/scan/nsad063] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/16/2023] [Revised: 08/04/2023] [Accepted: 11/01/2023] [Indexed: 11/08/2023] Open
Abstract
Film functional magnetic resonance imaging (fMRI) has gained tremendous popularity in many areas of neuroscience. However, affective neuroscience remains somewhat behind in embracing this approach, even though films lend themselves to study how brain function gives rise to complex, dynamic and multivariate emotions. Here, we discuss the unique capabilities of film fMRI for emotion research, while providing a general guide of conducting such research. We first give a brief overview of emotion theories as these inform important design choices. Next, we discuss films as experimental paradigms for emotion elicitation and address the process of annotating them. We then situate film fMRI in the context of other fMRI approaches, and present an overview of results from extant studies so far with regard to advantages of film fMRI. We also give an overview of state-of-the-art analysis techniques including methods that probe neurodynamics. Finally, we convey limitations of using film fMRI to study emotion. In sum, this review offers a practitioners' guide to the emerging field of film fMRI and underscores how it can advance affective neuroscience.
Collapse
Affiliation(s)
- Elenor Morgenroth
- Neuro-X Institute, École Polytechnique Fédérale de Lausanne, Geneva 1202, Switzerland
- Department of Radiology and Medical Informatics, University of Geneva, Geneva 1202, Switzerland
- Swiss Center for Affective Sciences, University of Geneva, Geneva 1202, Switzerland
| | - Laura Vilaclara
- Neuro-X Institute, École Polytechnique Fédérale de Lausanne, Geneva 1202, Switzerland
- Department of Radiology and Medical Informatics, University of Geneva, Geneva 1202, Switzerland
| | - Michal Muszynski
- Department of Basic Neurosciences, University of Geneva, Geneva 1202, Switzerland
| | - Julian Gaviria
- Swiss Center for Affective Sciences, University of Geneva, Geneva 1202, Switzerland
- Department of Basic Neurosciences, University of Geneva, Geneva 1202, Switzerland
- Department of Psychiatry, University of Geneva, Geneva 1202, Switzerland
| | - Patrik Vuilleumier
- Swiss Center for Affective Sciences, University of Geneva, Geneva 1202, Switzerland
- Department of Basic Neurosciences, University of Geneva, Geneva 1202, Switzerland
- CIBM Center for Biomedical Imaging, Geneva 1202, Switzerland
| | - Dimitri Van De Ville
- Neuro-X Institute, École Polytechnique Fédérale de Lausanne, Geneva 1202, Switzerland
- Department of Radiology and Medical Informatics, University of Geneva, Geneva 1202, Switzerland
- CIBM Center for Biomedical Imaging, Geneva 1202, Switzerland
| |
Collapse
|
3
|
Betzel RF, Faskowitz J, Sporns O. Living on the edge: network neuroscience beyond nodes. Trends Cogn Sci 2023; 27:1068-1084. [PMID: 37716895 PMCID: PMC10592364 DOI: 10.1016/j.tics.2023.08.009] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/15/2023] [Revised: 07/14/2023] [Accepted: 08/10/2023] [Indexed: 09/18/2023]
Abstract
Network neuroscience has emphasized the connectional properties of neural elements - cells, populations, and regions. This has come at the expense of the anatomical and functional connections that link these elements to one another. A new perspective - namely one that emphasizes 'edges' - may prove fruitful in addressing outstanding questions in network neuroscience. We highlight one recently proposed 'edge-centric' method and review its current applications, merits, and limitations. We also seek to establish conceptual and mathematical links between this method and previously proposed approaches in the network science and neuroimaging literature. We conclude by presenting several avenues for future work to extend and refine existing edge-centric analysis.
Collapse
Affiliation(s)
- Richard F Betzel
- Department of Psychological and Brain Sciences, Indiana University, Bloomington, IN 47405, USA; Cognitive Science Program, Indiana University, Bloomington, IN 47405, USA; Program in Neuroscience, Indiana University, Bloomington, IN 47405, USA; Network Science Institute, Indiana University, Bloomington, IN 47405, USA.
| | - Joshua Faskowitz
- Section on Functional Imaging Methods, National Institute of Mental Health, Bethesda, MD, USA
| | - Olaf Sporns
- Department of Psychological and Brain Sciences, Indiana University, Bloomington, IN 47405, USA; Cognitive Science Program, Indiana University, Bloomington, IN 47405, USA; Program in Neuroscience, Indiana University, Bloomington, IN 47405, USA; Network Science Institute, Indiana University, Bloomington, IN 47405, USA
| |
Collapse
|
4
|
Di X, Zhang Z, Xu T, Biswal BB. Dynamic and stationary brain connectivity during movie watching as revealed by functional MRI. Brain Struct Funct 2022; 227:2299-2312. [PMID: 35767066 PMCID: PMC9420792 DOI: 10.1007/s00429-022-02522-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/23/2021] [Accepted: 06/04/2022] [Indexed: 11/25/2022]
Abstract
Spatially remote brain regions show synchronized activity as typically revealed by correlated functional MRI (fMRI) signals. An emerging line of research has focused on the temporal fluctuations of connectivity; however, its relationships with stationary connectivity have not been clearly illustrated. We examined dynamic and stationary connectivity when the participants watched four different movie clips. We calculated point-by-point multiplication between two regional time series to estimate the time-resolved dynamic connectivity, and estimated the inter-individual consistency of the dynamic connectivity time series. Widespread consistent dynamic connectivity was observed for each movie clip, which also showed differences between the clips. For example, a cartoon movie clip, Wall-E, showed more consistent of dynamic connectivity with the posterior cingulate cortex and supramarginal gyrus, while a court drama clip, A Few Good Men, showed more consistent of dynamic connectivity with the auditory cortex and temporoparietal junction, which might suggest the involvement of specific brain processing for different movie contents. In contrast, the stationary connectivity as measured by the correlations between regional time series was highly similar among the movie clips, and showed fewer statistically significant differences. The patterns of consistent dynamic connectivity could be used to classify different movie clips with higher accuracy than the stationary connectivity and regional activity. These results support the functional significance of dynamic connectivity in reflecting functional brain changes, which could provide more functionally relevant information than stationary connectivity.
Collapse
Affiliation(s)
- Xin Di
- Department of Biomedical Engineering, New Jersey Institute of Technology, University Height, Newark, NJ, 07102, USA.
| | - Zhiguo Zhang
- School of Biomedical Engineering, Health Science Center, Shenzhen University, Shenzhen, People's Republic of China
- Guangdong Provincial Key Laboratory of Biomedical Measurements and Ultrasound Imaging, Shenzhen, People's Republic of China
| | - Ting Xu
- Center for the Developing Brain, Child Mind Institute, New York, NY, 10022, USA
| | - Bharat B Biswal
- Department of Biomedical Engineering, New Jersey Institute of Technology, University Height, Newark, NJ, 07102, USA.
| |
Collapse
|