1
|
Jang G, Kragel PA. Understanding human amygdala function with artificial neural networks. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2024:2024.07.29.605621. [PMID: 39131372 PMCID: PMC11312467 DOI: 10.1101/2024.07.29.605621] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Download PDF] [Subscribe] [Scholar Register] [Indexed: 08/13/2024]
Abstract
The amygdala is a cluster of subcortical nuclei that receives diverse sensory inputs and projects to the cortex, midbrain and other subcortical structures. Numerous accounts of amygdalar contributions to social and emotional behavior have been offered, yet an overarching description of amygdala function remains elusive. Here we adopt a computationally explicit framework that aims to develop a model of amygdala function based on the types of sensory inputs it receives, rather than individual constructs such as threat, arousal, or valence. Characterizing human fMRI signal acquired as participants viewed a full-length film, we developed encoding models that predict both patterns of amygdala activity and self-reported valence evoked by naturalistic images. We use deep image synthesis to generate artificial stimuli that distinctly engage encoding models of amygdala subregions that systematically differ from one another in terms of their low-level visual properties. These findings characterize how the amygdala compresses high-dimensional sensory inputs into low-dimensional representations relevant for behavior.
Collapse
|
2
|
Lahner B, Dwivedi K, Iamshchinina P, Graumann M, Lascelles A, Roig G, Gifford AT, Pan B, Jin S, Ratan Murty NA, Kay K, Oliva A, Cichy R. Modeling short visual events through the BOLD moments video fMRI dataset and metadata. Nat Commun 2024; 15:6241. [PMID: 39048577 PMCID: PMC11269733 DOI: 10.1038/s41467-024-50310-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/14/2023] [Accepted: 07/04/2024] [Indexed: 07/27/2024] Open
Abstract
Studying the neural basis of human dynamic visual perception requires extensive experimental data to evaluate the large swathes of functionally diverse brain neural networks driven by perceiving visual events. Here, we introduce the BOLD Moments Dataset (BMD), a repository of whole-brain fMRI responses to over 1000 short (3 s) naturalistic video clips of visual events across ten human subjects. We use the videos' extensive metadata to show how the brain represents word- and sentence-level descriptions of visual events and identify correlates of video memorability scores extending into the parietal cortex. Furthermore, we reveal a match in hierarchical processing between cortical regions of interest and video-computable deep neural networks, and we showcase that BMD successfully captures temporal dynamics of visual events at second resolution. With its rich metadata, BMD offers new perspectives and accelerates research on the human brain basis of visual event perception.
Collapse
Affiliation(s)
- Benjamin Lahner
- Computer Science and Artificial Intelligence Laboratory, MIT, Cambridge, MA, USA.
| | - Kshitij Dwivedi
- Department of Education and Psychology, Freie Universität Berlin, Berlin, Germany
- Department of Computer Science, Goethe University Frankfurt, Frankfurt am Main, Germany
| | - Polina Iamshchinina
- Department of Education and Psychology, Freie Universität Berlin, Berlin, Germany
| | - Monika Graumann
- Department of Education and Psychology, Freie Universität Berlin, Berlin, Germany
| | - Alex Lascelles
- Computer Science and Artificial Intelligence Laboratory, MIT, Cambridge, MA, USA
| | - Gemma Roig
- Department of Computer Science, Goethe University Frankfurt, Frankfurt am Main, Germany
- The Hessian Center for AI (hessian.AI), Darmstadt, Germany
| | | | - Bowen Pan
- Computer Science and Artificial Intelligence Laboratory, MIT, Cambridge, MA, USA
| | - SouYoung Jin
- Computer Science and Artificial Intelligence Laboratory, MIT, Cambridge, MA, USA
| | - N Apurva Ratan Murty
- Department of Brain and Cognitive Science, MIT, Cambridge, MA, USA
- School of Psychology, Georgia Institute of Technology, Atlanta, GA, USA
| | - Kendrick Kay
- Center for Magnetic Resonance Research (CMRR), Department of Radiology, University of Minnesota, Minneapolis, MN, USA
| | - Aude Oliva
- Computer Science and Artificial Intelligence Laboratory, MIT, Cambridge, MA, USA
| | - Radoslaw Cichy
- Department of Education and Psychology, Freie Universität Berlin, Berlin, Germany
| |
Collapse
|
3
|
Lee Masson H, Chang L, Isik L. Multidimensional neural representations of social features during movie viewing. Soc Cogn Affect Neurosci 2024; 19:nsae030. [PMID: 38722755 PMCID: PMC11130526 DOI: 10.1093/scan/nsae030] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/21/2023] [Revised: 03/05/2024] [Accepted: 05/03/2024] [Indexed: 05/29/2024] Open
Abstract
The social world is dynamic and contextually embedded. Yet, most studies utilize simple stimuli that do not capture the complexity of everyday social episodes. To address this, we implemented a movie viewing paradigm and investigated how everyday social episodes are processed in the brain. Participants watched one of two movies during an MRI scan. Neural patterns from brain regions involved in social perception, mentalization, action observation and sensory processing were extracted. Representational similarity analysis results revealed that several labeled social features (including social interaction, mentalization, the actions of others, characters talking about themselves, talking about others and talking about objects) were represented in the superior temporal gyrus (STG) and middle temporal gyrus (MTG). The mentalization feature was also represented throughout the theory of mind network, and characters talking about others engaged the temporoparietal junction (TPJ), suggesting that listeners may spontaneously infer the mental state of those being talked about. In contrast, we did not observe the action representations in the frontoparietal regions of the action observation network. The current findings indicate that STG and MTG serve as key regions for social processing, and that listening to characters talk about others elicits spontaneous mental state inference in TPJ during natural movie viewing.
Collapse
Affiliation(s)
| | - Lucy Chang
- Department of Cognitive Science, Johns Hopkins University, Baltimore 21218, USA
| | - Leyla Isik
- Department of Cognitive Science, Johns Hopkins University, Baltimore 21218, USA
| |
Collapse
|
4
|
Kim HJ, Lux BK, Lee E, Finn ES, Woo CW. Brain decoding of spontaneous thought: Predictive modeling of self-relevance and valence using personal narratives. Proc Natl Acad Sci U S A 2024; 121:e2401959121. [PMID: 38547065 PMCID: PMC10998624 DOI: 10.1073/pnas.2401959121] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/01/2024] [Accepted: 02/20/2024] [Indexed: 04/02/2024] Open
Abstract
The contents and dynamics of spontaneous thought are important factors for personality traits and mental health. However, assessing spontaneous thoughts is challenging due to their unconstrained nature, and directing participants' attention to report their thoughts may fundamentally alter them. Here, we aimed to decode two key content dimensions of spontaneous thought-self-relevance and valence-directly from brain activity. To train functional MRI-based predictive models, we used individually generated personal stories as stimuli in a story-reading task to mimic narrative-like spontaneous thoughts (n = 49). We then tested these models on multiple test datasets (total n = 199). The default mode, ventral attention, and frontoparietal networks played key roles in the predictions, with the anterior insula and midcingulate cortex contributing to self-relevance prediction and the left temporoparietal junction and dorsomedial prefrontal cortex contributing to valence prediction. Overall, this study presents brain models of internal thoughts and emotions, highlighting the potential for the brain decoding of spontaneous thought.
Collapse
Affiliation(s)
- Hong Ji Kim
- Center for Neuroscience Imaging Research, Institute for Basic Science, Suwon16419, South Korea
- Department of Biomedical Engineering, Sungkyunkwan University, Suwon16419, South Korea
- Department of Intelligent Precision Healthcare Convergence, Sungkyunkwan University, Suwon16419, South Korea
| | - Byeol Kim Lux
- Center for Neuroscience Imaging Research, Institute for Basic Science, Suwon16419, South Korea
- Department of Biomedical Engineering, Sungkyunkwan University, Suwon16419, South Korea
- Department of Psychological and Brain Sciences, Dartmouth College, NH03755
| | - Eunjin Lee
- Center for Neuroscience Imaging Research, Institute for Basic Science, Suwon16419, South Korea
- Department of Biomedical Engineering, Sungkyunkwan University, Suwon16419, South Korea
- Department of Intelligent Precision Healthcare Convergence, Sungkyunkwan University, Suwon16419, South Korea
| | - Emily S. Finn
- Department of Psychological and Brain Sciences, Dartmouth College, NH03755
| | - Choong-Wan Woo
- Center for Neuroscience Imaging Research, Institute for Basic Science, Suwon16419, South Korea
- Department of Biomedical Engineering, Sungkyunkwan University, Suwon16419, South Korea
- Department of Intelligent Precision Healthcare Convergence, Sungkyunkwan University, Suwon16419, South Korea
- Life-inspired Neural Network for Prediction and Optimization Research Group, Suwon16419, South Korea
| |
Collapse
|
5
|
Thye M, Hoffman P, Mirman D. The neural basis of naturalistic semantic and social cognition. Sci Rep 2024; 14:6796. [PMID: 38514738 PMCID: PMC10957894 DOI: 10.1038/s41598-024-56897-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/24/2022] [Accepted: 03/11/2024] [Indexed: 03/23/2024] Open
Abstract
Decoding social environments and engaging meaningfully with other people are critical aspects of human cognition. Multiple cognitive systems, including social and semantic cognition, work alongside each other to support these processes. This study investigated shared processing between social and semantic systems using neuroimaging data collected during movie-viewing, which captures the multimodal environment in which social knowledge is exchanged. Semantic and social content from movie events (event-level) and movie transcripts (word-level) were used in parametric modulation analyses to test (1) the degree to which semantic and social information is processed within each respective network and (2) engagement of the same cross-network regions or the same domain-general hub located within the semantic network during semantic and social processing. Semantic word and event-level content engaged the same fronto-temporo-parietal network and a portion of the semantic hub in the anterior temporal lobe (ATL). Social word and event-level content engaged the supplementary motor area and right angular gyrus within the social network, but only social words engaged the domain-general semantic hub in left ATL. There was evidence of shared processing between the social and semantic systems in the dorsolateral portion of right ATL which was engaged by word and event-level semantic and social content. Overlap between the semantic and social word and event results was highly variable within and across participants, with the most consistent loci of overlap occurring in left inferior frontal, bilateral precentral and supramarginal gyri for social and semantic words and in bilateral superior temporal gyrus extending from ATL posteriorly into supramarginal gyri for social and semantic events. These results indicate a complex pattern of shared and distinct regions for social and semantic cognition during naturalistic processing. PROTOCOL REGISTRATION: The stage 1 protocol for this Registered Report was accepted in principle on October 11, 2022. The protocol, as accepted by the journal, can be found at: https://doi.org/10.17605/OSF.IO/ACWQY .
Collapse
Affiliation(s)
- Melissa Thye
- Department of Psychology, University of Edinburgh, Edinburgh, EH8 9JZ, UK.
| | - Paul Hoffman
- Department of Psychology, University of Edinburgh, Edinburgh, EH8 9JZ, UK
| | - Daniel Mirman
- Department of Psychology, University of Edinburgh, Edinburgh, EH8 9JZ, UK
| |
Collapse
|
6
|
Sherafati A, Bajracharya A, Jones MS, Speh E, Munsi M, Lin CHP, Fishell AK, Hershey T, Eggebrecht AT, Culver JP, Peelle JE. A high-density diffuse optical tomography dataset of naturalistic viewing. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2024:2023.11.07.565473. [PMID: 37986896 PMCID: PMC10659362 DOI: 10.1101/2023.11.07.565473] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/22/2023]
Abstract
Traditional laboratory tasks offer tight experimental control but lack the richness of our everyday human experience. As a result many cognitive neuroscientists have been motivated to adopt experimental paradigms that are more natural, such as stories and movies. Here we describe data collected from 58 healthy adult participants (aged 18-76 years) who viewed 10 minutes of a movie (The Good, the Bad, and the Ugly, 1966). Most (36) participants viewed the clip more than once, resulting in 106 sessions of data. Cortical responses were mapped using high-density diffuse optical tomography (first- through fourth nearest neighbor separations of 1.3, 3.0, 3.9, and 4.7 cm), covering large portions of superficial occipital, temporal, parietal, and frontal lobes. Consistency of measured activity across subjects was quantified using intersubject correlation analysis. Data are provided in both channel format (SNIRF) and projected to standard space (NIfTI), using an atlas-based light model. These data are suitable for methods exploration as well as investigating a wide variety of cognitive phenomena.
Collapse
Affiliation(s)
| | | | - Michael S Jones
- Department of Otolaryngology, Washington University in St. Louis
| | - Emma Speh
- Department of Radiology, Washington University in St. Louis
| | - Monalisa Munsi
- Department of Radiology, Washington University in St. Louis
| | - Chen-Hao P Lin
- Department of Physics, Washington University in St. Louis
| | | | - Tamara Hershey
- Department of Psychiatry, Washington University in St. Louis
- Department of Radiology, Washington University in St. Louis
| | | | | | - Jonathan E Peelle
- Center for Cognitive and Brain Health, Northeastern University
- Department of Communication Sciences and Disorders, Northeastern University
- Department of Psychology, Northeastern University
| |
Collapse
|
7
|
Gong Z, Zhou M, Dai Y, Wen Y, Liu Y, Zhen Z. A large-scale fMRI dataset for the visual processing of naturalistic scenes. Sci Data 2023; 10:559. [PMID: 37612327 PMCID: PMC10447576 DOI: 10.1038/s41597-023-02471-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/27/2023] [Accepted: 08/14/2023] [Indexed: 08/25/2023] Open
Abstract
One ultimate goal of visual neuroscience is to understand how the brain processes visual stimuli encountered in the natural environment. Achieving this goal requires records of brain responses under massive amounts of naturalistic stimuli. Although the scientific community has put a lot of effort into collecting large-scale functional magnetic resonance imaging (fMRI) data under naturalistic stimuli, more naturalistic fMRI datasets are still urgently needed. We present here the Natural Object Dataset (NOD), a large-scale fMRI dataset containing responses to 57,120 naturalistic images from 30 participants. NOD strives for a balance between sampling variation between individuals and sampling variation between stimuli. This enables NOD to be utilized not only for determining whether an observation is generalizable across many individuals, but also for testing whether a response pattern is generalized to a variety of naturalistic stimuli. We anticipate that the NOD together with existing naturalistic neuroimaging datasets will serve as a new impetus for our understanding of the visual processing of naturalistic stimuli.
Collapse
Affiliation(s)
- Zhengxin Gong
- Beijing Key Laboratory of Applied Experimental Psychology, Faculty of Psychology, Beijing Normal University, Beijing, 100875, China
| | - Ming Zhou
- State Key Laboratory of Cognitive Neuroscience and Learning & IDG/McGovern Institute for Brain Research, Beijing Normal University, Beijing, 100875, China
| | - Yuxuan Dai
- Beijing Key Laboratory of Applied Experimental Psychology, Faculty of Psychology, Beijing Normal University, Beijing, 100875, China
| | - Yushan Wen
- Beijing Key Laboratory of Applied Experimental Psychology, Faculty of Psychology, Beijing Normal University, Beijing, 100875, China
| | - Youyi Liu
- State Key Laboratory of Cognitive Neuroscience and Learning & IDG/McGovern Institute for Brain Research, Beijing Normal University, Beijing, 100875, China.
| | - Zonglei Zhen
- Beijing Key Laboratory of Applied Experimental Psychology, Faculty of Psychology, Beijing Normal University, Beijing, 100875, China.
- State Key Laboratory of Cognitive Neuroscience and Learning & IDG/McGovern Institute for Brain Research, Beijing Normal University, Beijing, 100875, China.
| |
Collapse
|
8
|
LeBel A, Wagner L, Jain S, Adhikari-Desai A, Gupta B, Morgenthal A, Tang J, Xu L, Huth AG. A natural language fMRI dataset for voxelwise encoding models. Sci Data 2023; 10:555. [PMID: 37612332 PMCID: PMC10447563 DOI: 10.1038/s41597-023-02437-z] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/26/2022] [Accepted: 08/02/2023] [Indexed: 08/25/2023] Open
Abstract
Speech comprehension is a complex process that draws on humans' abilities to extract lexical information, parse syntax, and form semantic understanding. These sub-processes have traditionally been studied using separate neuroimaging experiments that attempt to isolate specific effects of interest. More recently it has become possible to study all stages of language comprehension in a single neuroimaging experiment using narrative natural language stimuli. The resulting data are richly varied at every level, enabling analyses that can probe everything from spectral representations to high-level representations of semantic meaning. We provide a dataset containing BOLD fMRI responses recorded while 8 participants each listened to 27 complete, natural, narrative stories (~6 hours). This dataset includes pre-processed and raw MRIs, as well as hand-constructed 3D cortical surfaces for each participant. To address the challenges of analyzing naturalistic data, this dataset is accompanied by a python library containing basic code for creating voxelwise encoding models. Altogether, this dataset provides a large and novel resource for understanding speech and language processing in the human brain.
Collapse
Affiliation(s)
- Amanda LeBel
- Helen Wills Neuroscience Institute, University of California, Berkeley, Berkeley, CA, 94704, USA
| | - Lauren Wagner
- Department of Psychiatry and Biobehavioral Sciences, University of California, Los Angeles, CA, 90095, USA
| | - Shailee Jain
- Department of Computer Science, The University of Texas at Austin, Austin, TX, 78712, USA
| | - Aneesh Adhikari-Desai
- Department of Computer Science, The University of Texas at Austin, Austin, TX, 78712, USA
- Department of Neuroscience, The University of Texas at Austin, Austin, TX, 78712, USA
| | - Bhavin Gupta
- Department of Computer Science, The University of Texas at Austin, Austin, TX, 78712, USA
| | - Allyson Morgenthal
- Department of Neuroscience, The University of Texas at Austin, Austin, TX, 78712, USA
| | - Jerry Tang
- Department of Computer Science, The University of Texas at Austin, Austin, TX, 78712, USA
| | - Lixiang Xu
- Department of Physics, The University of Texas at Austin, Austin, TX, 78712, USA
| | - Alexander G Huth
- Department of Computer Science, The University of Texas at Austin, Austin, TX, 78712, USA.
- Department of Neuroscience, The University of Texas at Austin, Austin, TX, 78712, USA.
| |
Collapse
|
9
|
Clark IA, Maguire EA. Release of cognitive and multimodal MRI data including real-world tasks and hippocampal subfield segmentations. Sci Data 2023; 10:540. [PMID: 37587129 PMCID: PMC10432478 DOI: 10.1038/s41597-023-02449-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/02/2023] [Accepted: 08/07/2023] [Indexed: 08/18/2023] Open
Abstract
We share data from N = 217 healthy adults (mean age 29 years, range 20-41; 109 females, 108 males) who underwent extensive cognitive assessment and neuroimaging to examine the neural basis of individual differences, with a particular focus on a brain structure called the hippocampus. Cognitive data were collected using a wide array of questionnaires, naturalistic tests that examined imagination, autobiographical memory recall and spatial navigation, traditional laboratory-based tests such as recalling word pairs, and comprehensive characterisation of the strategies used to perform the cognitive tests. 3 Tesla MRI data were also acquired and include multi-parameter mapping to examine tissue microstructure, diffusion-weighted MRI, T2-weighted high-resolution partial volume structural MRI scans (with the masks of hippocampal subfields manually segmented from these scans), whole brain resting state functional MRI scans and partial volume high resolution resting state functional MRI scans. This rich dataset will be of value to cognitive and clinical neuroscientists researching individual differences, real-world cognition, brain-behaviour associations, hippocampal subfields and more. All data are freely available on Dryad.
Collapse
Affiliation(s)
- Ian A Clark
- Wellcome Centre for Human Neuroimaging, Department of Imaging Neuroscience, UCL Queen Square Institute of Neurology, University College London, London, UK
| | - Eleanor A Maguire
- Wellcome Centre for Human Neuroimaging, Department of Imaging Neuroscience, UCL Queen Square Institute of Neurology, University College London, London, UK.
| |
Collapse
|
10
|
Thye M, Hoffman P, Mirman D. The words that little by little revealed everything: Neural response to lexical-semantic content during narrative comprehension. Neuroimage 2023; 276:120204. [PMID: 37257674 DOI: 10.1016/j.neuroimage.2023.120204] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/09/2022] [Revised: 04/19/2023] [Accepted: 05/27/2023] [Indexed: 06/02/2023] Open
Abstract
The ease with which narratives are understood belies the complexity of the information being conveyed and the cognitive processes that support comprehension. The meanings of the words must be rapidly accessed and integrated with the reader's mental representation of the overarching, unfolding scenario. A broad, bilateral brain network is engaged by this process, but it is not clear how words that vary on specific semantic dimensions, such as ambiguity, emotion, or socialness, engage the semantic, semantic control, or social cognition systems. In the present study, data from 48 participants who listened to The Little Prince audiobook during MRI scanning were selected from the Le Petit Prince dataset. The lexical and semantic content within the narrative was quantified from the transcript words with factor scores capturing Word Length, Semantic Flexibility, Emotional Strength, and Social Impact. These scores, along with word quantity variables, were used to investigate where these predictors co-vary with activation across the brain. In contrast to studies of isolated word processing, large networks were found to co-vary with the lexical and semantic content within the narrative. An increase in semantic content engaged the ventral portion of ventrolateral ATL, consistent with its role as a semantic hub. Decreased semantic content engaged temporal pole and inferior parietal lobule, which may reflect semantic integration. The semantic control network was engaged by words with low Semantic Flexibility, perhaps due to the demand required to process infrequent, less semantically diverse language. Activation in ATL co-varied with an increase in Social Impact, which is consistent with the claim that social knowledge is housed within the neural architecture of the semantic system. These results suggest that current models of language processing may present an impoverished estimate of the neural systems that coordinate to support narrative comprehension, and, by extension, real-world language processing.
Collapse
Affiliation(s)
- Melissa Thye
- School of Philosophy, Psychology & Language Sciences, University of Edinburgh, Edinburgh EH8 9JZ, United Kingdom.
| | - Paul Hoffman
- School of Philosophy, Psychology & Language Sciences, University of Edinburgh, Edinburgh EH8 9JZ, United Kingdom
| | - Daniel Mirman
- School of Philosophy, Psychology & Language Sciences, University of Edinburgh, Edinburgh EH8 9JZ, United Kingdom
| |
Collapse
|
11
|
Zhou M, Gong Z, Dai Y, Wen Y, Liu Y, Zhen Z. A large-scale fMRI dataset for human action recognition. Sci Data 2023; 10:415. [PMID: 37369643 DOI: 10.1038/s41597-023-02325-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/13/2023] [Accepted: 06/21/2023] [Indexed: 06/29/2023] Open
Abstract
Human action recognition is a critical capability for our survival, allowing us to interact easily with the environment and others in everyday life. Although the neural basis of action recognition has been widely studied using a few action categories from simple contexts as stimuli, how the human brain recognizes diverse human actions in real-world environments still needs to be explored. Here, we present the Human Action Dataset (HAD), a large-scale functional magnetic resonance imaging (fMRI) dataset for human action recognition. HAD contains fMRI responses to 21,600 video clips from 30 participants. The video clips encompass 180 human action categories and offer a comprehensive coverage of complex activities in daily life. We demonstrate that the data are reliable within and across participants and, notably, capture rich representation information of the observed human actions. This extensive dataset, with its vast number of action categories and exemplars, has the potential to deepen our understanding of human action recognition in natural environments.
Collapse
Affiliation(s)
- Ming Zhou
- State Key Laboratory of Cognitive Neuroscience and Learning & IDG/McGovern Institute for Brain Research, Beijing Normal University, Beijing, 100875, China
| | - Zhengxin Gong
- Beijing Key Laboratory of Applied Experimental Psychology, Faculty of Psychology, Beijing Normal University, Beijing, 100875, China
| | - Yuxuan Dai
- Beijing Key Laboratory of Applied Experimental Psychology, Faculty of Psychology, Beijing Normal University, Beijing, 100875, China
| | - Yushan Wen
- Beijing Key Laboratory of Applied Experimental Psychology, Faculty of Psychology, Beijing Normal University, Beijing, 100875, China
| | - Youyi Liu
- State Key Laboratory of Cognitive Neuroscience and Learning & IDG/McGovern Institute for Brain Research, Beijing Normal University, Beijing, 100875, China
| | - Zonglei Zhen
- Beijing Key Laboratory of Applied Experimental Psychology, Faculty of Psychology, Beijing Normal University, Beijing, 100875, China.
| |
Collapse
|
12
|
Mahrukh R, Shakil S, Malik AS. Sentiments analysis of fMRI using automatically generated stimuli labels under naturalistic paradigm. Sci Rep 2023; 13:7267. [PMID: 37142654 PMCID: PMC10160115 DOI: 10.1038/s41598-023-33734-7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/13/2023] [Accepted: 04/18/2023] [Indexed: 05/06/2023] Open
Abstract
Our emotions and sentiments are influenced by naturalistic stimuli such as the movies we watch and the songs we listen to, accompanied by changes in our brain activation. Comprehension of these brain-activation dynamics can assist in identification of any associated neurological condition such as stress and depression, leading towards making informed decision about suitable stimuli. A large number of open-access functional magnetic resonance imaging (fMRI) datasets collected under naturalistic conditions can be used for classification/prediction studies. However, these datasets do not provide emotion/sentiment labels, which limits their use in supervised learning studies. Manual labeling by subjects can generate these labels, however, this method is subjective and biased. In this study, we are proposing another approach of generating automatic labels from the naturalistic stimulus itself. We are using sentiment analyzers (VADER, TextBlob, and Flair) from natural language processing to generate labels using movie subtitles. Subtitles generated labels are used as the class labels for positive, negative, and neutral sentiments for classification of brain fMRI images. Support vector machine, random forest, decision tree, and deep neural network classifiers are used. We are getting reasonably good classification accuracy (42-84%) for imbalanced data, which is increased (55-99%) for balanced data.
Collapse
Affiliation(s)
| | - Sadia Shakil
- Institute of Space Technology, Islamabad, Pakistan.
- Faculty of Information Technology, Brno University of Technology, Brno, Czech Republic.
- Turner Institute for Brain and Mental Health, Monash University, Melbourne, Australia.
| | - Aamir Saeed Malik
- Faculty of Information Technology, Brno University of Technology, Brno, Czech Republic.
| |
Collapse
|
13
|
Grall C, Equita J, Finn ES. Neural unscrambling of temporal information during a nonlinear narrative. Cereb Cortex 2023:7031158. [PMID: 36752641 DOI: 10.1093/cercor/bhad015] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/05/2022] [Revised: 01/12/2023] [Accepted: 01/13/2023] [Indexed: 02/09/2023] Open
Abstract
Although we must experience our lives chronologically, storytellers often manipulate the order in which they relay events. How the brain processes temporal information while encoding a nonlinear narrative remains unclear. Here, we use functional magnetic resonance imaging during movie watching to investigate which brain regions are sensitive to information about time in a narrative and test whether the representation of temporal context across a narrative is more influenced by the order in which events are presented or their underlying chronological sequence. Results indicate that medial parietal regions are sensitive to cued jumps through time over and above other changes in context (i.e., location). Moreover, when processing non-chronological narrative information, the precuneus and posterior cingulate engage in on-the-fly temporal unscrambling to represent information chronologically. Specifically, days that are closer together in chronological time are represented more similarly regardless of when they are presented in the movie, and this representation is consistent across participants. Additional analyses reveal a strong spatial signature associated with higher magnitude jumps through time. These findings are consistent with prior theorizing on medial parietal regions as central to maintaining and updating narrative situation models, and suggest the priority of chronological information when encoding narrative events.
Collapse
Affiliation(s)
- Clare Grall
- Department of Psychological and Brain Sciences, Dartmouth College, 6207 Moore Hall, Hanover, NH 03755, United States
| | - Josefa Equita
- Department of Psychological and Brain Sciences, Dartmouth College, 6207 Moore Hall, Hanover, NH 03755, United States
| | - Emily S Finn
- Department of Psychological and Brain Sciences, Dartmouth College, 6207 Moore Hall, Hanover, NH 03755, United States
| |
Collapse
|
14
|
De Rosa AP, Esposito F, Valsasina P, d'Ambrosio A, Bisecco A, Rocca MA, Tommasin S, Marzi C, De Stefano N, Battaglini M, Pantano P, Cirillo M, Tedeschi G, Filippi M, Gallo A. Resting-state functional MRI in multicenter studies on multiple sclerosis: a report on raw data quality and functional connectivity features from the Italian Neuroimaging Network Initiative. J Neurol 2023; 270:1047-1066. [PMID: 36350401 PMCID: PMC9886598 DOI: 10.1007/s00415-022-11479-z] [Citation(s) in RCA: 4] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/25/2022] [Revised: 11/03/2022] [Accepted: 11/04/2022] [Indexed: 11/11/2022]
Abstract
The Italian Neuroimaging Network Initiative (INNI) is an expanding repository of brain MRI data from multiple sclerosis (MS) patients recruited at four Italian MRI research sites. We describe the raw data quality of resting-state functional MRI (RS-fMRI) time-series in INNI and the inter-site variability in functional connectivity (FC) features after unified automated data preprocessing. MRI datasets from 489 MS patients and 246 healthy control (HC) subjects were retrieved from the INNI database. Raw data quality metrics included temporal signal-to-noise ratio (tSNR), spatial smoothness (FWHM), framewise displacement (FD), and differential variation in signals (DVARS). Automated preprocessing integrated white-matter lesion segmentation (SAMSEG) into a standard fMRI pipeline (fMRIPrep). FC features were calculated on pre-processed data and harmonized between sites (Combat) prior to assessing general MS-related alterations. Across centers (both groups), median tSNR and FWHM ranged from 47 to 84 and from 2.0 to 2.5, and median FD and DVARS ranged from 0.08 to 0.24 and from 1.06 to 1.22. After preprocessing, only global FC-related features were significantly correlated with FD or DVARS. Across large-scale networks, age/sex/FD-adjusted and harmonized FC features exhibited both inter-site and site-specific inter-group effects. Significant general reductions were obtained for somatomotor and limbic networks in MS patients (vs. HC). The implemented procedures provide technical information on raw data quality and outcome of fully automated preprocessing that might serve as reference in future RS-fMRI studies within INNI. The unified pipeline introduced little bias across sites and appears suitable for multisite FC analyses on harmonized network estimates.
Collapse
Affiliation(s)
- Alessandro Pasquale De Rosa
- Department of Advanced Medical and Surgical Sciences, University of Campania "Luigi Vanvitelli", Piazza Luigi Miraglia, 2, 80138, Naples, Italy
| | - Fabrizio Esposito
- Department of Advanced Medical and Surgical Sciences, University of Campania "Luigi Vanvitelli", Piazza Luigi Miraglia, 2, 80138, Naples, Italy.
| | - Paola Valsasina
- Neuroimaging Research Unit, Division of Neuroscience, IRCCS San Raffaele Scientific Institute, Via Olgettina 60, 20132, Milan, Italy
| | - Alessandro d'Ambrosio
- Department of Advanced Medical and Surgical Sciences, University of Campania "Luigi Vanvitelli", Piazza Luigi Miraglia, 2, 80138, Naples, Italy
| | - Alvino Bisecco
- Department of Advanced Medical and Surgical Sciences, University of Campania "Luigi Vanvitelli", Piazza Luigi Miraglia, 2, 80138, Naples, Italy
| | - Maria A Rocca
- Neuroimaging Research Unit, Division of Neuroscience, IRCCS San Raffaele Scientific Institute, Via Olgettina 60, 20132, Milan, Italy
- Neurology Unit, IRCCS San Raffaele Scientific Institute, Via Olgettina 60, 20132, Milan, Italy
- Vita-Salute San Raffaele University, Via Olgettina 58, 20132, Milan, Italy
| | - Silvia Tommasin
- Department of Human Neurosciences, Sapienza University of Rome, Viale Dell'Università, 30, 00185, Rome, Italy
| | - Chiara Marzi
- Institute of Applied Physics "Nello Cararra" (IFAC), National Research Council (CNR), Via Madonna del Piano, 10, Sesto Fiorentino, 50019, Florence, Italy
| | - Nicola De Stefano
- Department of Medicine, Surgery and Neuroscience, University of Siena, Siena, Italy
| | - Marco Battaglini
- Department of Medicine, Surgery and Neuroscience, University of Siena, Siena, Italy
| | - Patrizia Pantano
- Department of Human Neurosciences, Sapienza University of Rome, Viale Dell'Università, 30, 00185, Rome, Italy
| | - Mario Cirillo
- Department of Advanced Medical and Surgical Sciences, University of Campania "Luigi Vanvitelli", Piazza Luigi Miraglia, 2, 80138, Naples, Italy
| | - Gioacchino Tedeschi
- Department of Advanced Medical and Surgical Sciences, University of Campania "Luigi Vanvitelli", Piazza Luigi Miraglia, 2, 80138, Naples, Italy
| | - Massimo Filippi
- Neuroimaging Research Unit, Division of Neuroscience, IRCCS San Raffaele Scientific Institute, Via Olgettina 60, 20132, Milan, Italy
- Neurology Unit, IRCCS San Raffaele Scientific Institute, Via Olgettina 60, 20132, Milan, Italy
- Neurorehabilitation Unit, IRCCS San Raffaele Scientific Institute, Via Olgettina 60, 20132, Milan, Italy
- Neurophysiology Service, IRCCS San Raffaele Scientific Institute, Via Olgettina 60, 20132, Milan, Italy
- Vita-Salute San Raffaele University, Via Olgettina 58, 20132, Milan, Italy
| | - Antonio Gallo
- Department of Advanced Medical and Surgical Sciences, University of Campania "Luigi Vanvitelli", Piazza Luigi Miraglia, 2, 80138, Naples, Italy
| |
Collapse
|
15
|
Santos-Lobato BL, Rocha JSDS, Rocha LC. Case report: Cerebellar sparing in juvenile Huntington's disease. Front Neurol 2023; 13:1089193. [PMID: 36712421 PMCID: PMC9874289 DOI: 10.3389/fneur.2022.1089193] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/04/2022] [Accepted: 12/28/2022] [Indexed: 01/13/2023] Open
Abstract
Juvenile Huntington's disease is an early-onset variant of Huntington's disease, generally associated with large CAG repeats and distinct clinical symptoms. The role of the cerebellum in Huntington's disease has been reevaluated, based on the presence of ataxia and findings on the impact of the disease on cerebellar volume. Recent studies showed a hyperconnectivity between the cerebellum and the basal ganglia in premanifest children with expanded CAG repeats, as well as an enlargement of the cerebellum in adolescence-onset Huntington's disease. We report a 21-year-old Brazilian female with Huntington's disease (age at disease onset 16 years) with Parkinsonism and no ataxic features. There was no reduction of cerebellar volume over 3 years of follow-up, despite the brain atrophy in other regions and clinical worsening. Furthermore, the cerebellar volume of the patient was similar to age- and sex-matched controls. These findings support the existence of compensatory mechanisms involving the cerebellum in individuals with a moderate-to-high number of CAG repeats (50-100 copies) in the early stages of life.
Collapse
Affiliation(s)
- Bruno Lopes Santos-Lobato
- Laboratory of Experimental Neuropathology, Federal University of Pará, Belém, PA, Brazil,Hospital Ophir Loyola, Belém, PA, Brazil,*Correspondence: Bruno Lopes Santos-Lobato ✉
| | | | | |
Collapse
|
16
|
Schmälzle R, Huskey R. Integrating media content analysis, reception analysis, and media effects studies. Front Neurosci 2023; 17:1155750. [PMID: 37179563 PMCID: PMC10173883 DOI: 10.3389/fnins.2023.1155750] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/31/2023] [Accepted: 03/28/2023] [Indexed: 05/15/2023] Open
Abstract
Every day, the world of media is at our fingertips, whether it is watching movies, listening to the radio, or browsing online media. On average, people spend over 8 h per day consuming messages from the mass media, amounting to a total lifetime dose of more than 20 years in which conceptual content stimulates our brains. Effects from this flood of information range from short-term attention bursts (e.g., by breaking news features or viral 'memes') to life-long memories (e.g., of one's favorite childhood movie), and from micro-level impacts on an individual's memory, attitudes, and behaviors to macro-level effects on nations or generations. The modern study of media's influence on society dates back to the 1940s. This body of mass communication scholarship has largely asked, "what is media's effect on the individual?" Around the time of the cognitive revolution, media psychologists began to ask, "what cognitive processes are involved in media processing?" More recently, neuroimaging researchers started using real-life media as stimuli to examine perception and cognition under more natural conditions. Such research asks: "what can media tell us about brain function?" With some exceptions, these bodies of scholarship often talk past each other. An integration offers new insights into the neurocognitive mechanisms through which media affect single individuals and entire audiences. However, this endeavor faces the same challenges as all interdisciplinary approaches: Researchers with different backgrounds have different levels of expertise, goals, and foci. For instance, neuroimaging researchers label media stimuli as "naturalistic" although they are in many ways rather artificial. Similarly, media experts are typically unfamiliar with the brain. Neither media creators nor neuroscientifically oriented researchers approach media effects from a social scientific perspective, which is the domain of yet another species. In this article, we provide an overview of approaches and traditions to studying media, and we review the emerging literature that aims to connect these streams. We introduce an organizing scheme that connects the causal paths from media content → brain responses → media effects and discuss network control theory as a promising framework to integrate media content, reception, and effects analyses.
Collapse
Affiliation(s)
- Ralf Schmälzle
- Department of Communication, Michigan State University, East Lansing, MI, United States
- *Correspondence: Ralf Schmälzle,
| | - Richard Huskey
- Department of Communication, University of California, Davis, Davis, CA, United States
- Cognitive Science Program, University of California, Davis, Davis, CA, United States
- Center for Mind and Brain, University of California, Davis, Davis, CA, United States
| |
Collapse
|
17
|
Wang S, Zhang X, Zhang J, Zong C. A synchronized multimodal neuroimaging dataset for studying brain language processing. Sci Data 2022; 9:590. [PMID: 36180444 PMCID: PMC9525723 DOI: 10.1038/s41597-022-01708-5] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/15/2022] [Accepted: 08/22/2022] [Indexed: 11/15/2022] Open
Abstract
We present a synchronized multimodal neuroimaging dataset for studying brain language processing (SMN4Lang) that contains functional magnetic resonance imaging (fMRI) and magnetoencephalography (MEG) data on the same 12 healthy volunteers while the volunteers listened to 6 hours of naturalistic stories, as well as high-resolution structural (T1, T2), diffusion MRI and resting-state fMRI data for each participant. We also provide rich linguistic annotations for the stimuli, including word frequencies, syntactic tree structures, time-aligned characters and words, and various types of word and character embeddings. Quality assessment indicators verify that this is a high-quality neuroimaging dataset. Such synchronized data is separately collected by the same group of participants first listening to story materials in fMRI and then in MEG which are well suited to studying the dynamic processing of language comprehension, such as the time and location of different linguistic features encoded in the brain. In addition, this dataset, comprising a large vocabulary from stories with various topics, can serve as a brain benchmark to evaluate and improve computational language models. Measurement(s) | functional brain measurement • Magnetoencephalography | Technology Type(s) | Functional Magnetic Resonance Imaging • Magnetoencephalography | Factor Type(s) | naturalistic stimuli listening | Sample Characteristic - Organism | humanbeings |
Collapse
Affiliation(s)
- Shaonan Wang
- National Laboratory of Pattern Recognition, Institute of Automation, CAS, Beijing, China. .,School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China.
| | - Xiaohan Zhang
- National Laboratory of Pattern Recognition, Institute of Automation, CAS, Beijing, China.,School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China
| | - Jiajun Zhang
- National Laboratory of Pattern Recognition, Institute of Automation, CAS, Beijing, China.,School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China
| | - Chengqing Zong
- National Laboratory of Pattern Recognition, Institute of Automation, CAS, Beijing, China.,School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China
| |
Collapse
|
18
|
de la Vega A, Rocca R, Blair RW, Markiewicz CJ, Mentch J, Kent JD, Herholz P, Ghosh SS, Poldrack RA, Yarkoni T. Neuroscout, a unified platform for generalizable and reproducible fMRI research. eLife 2022; 11:79277. [PMID: 36040302 PMCID: PMC9489206 DOI: 10.7554/elife.79277] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/06/2022] [Accepted: 08/27/2022] [Indexed: 11/28/2022] Open
Abstract
Functional magnetic resonance imaging (fMRI) has revolutionized cognitive neuroscience, but methodological barriers limit the generalizability of findings from the lab to the real world. Here, we present Neuroscout, an end-to-end platform for analysis of naturalistic fMRI data designed to facilitate the adoption of robust and generalizable research practices. Neuroscout leverages state-of-the-art machine learning models to automatically annotate stimuli from dozens of fMRI studies using naturalistic stimuli—such as movies and narratives—allowing researchers to easily test neuroscientific hypotheses across multiple ecologically-valid datasets. In addition, Neuroscout builds on a robust ecosystem of open tools and standards to provide an easy-to-use analysis builder and a fully automated execution engine that reduce the burden of reproducible research. Through a series of meta-analytic case studies, we validate the automatic feature extraction approach and demonstrate its potential to support more robust fMRI research. Owing to its ease of use and a high degree of automation, Neuroscout makes it possible to overcome modeling challenges commonly arising in naturalistic analysis and to easily scale analyses within and across datasets, democratizing generalizable fMRI research.
Collapse
Affiliation(s)
- Alejandro de la Vega
- Department of Psychology, The University of Texas at Austin, Austin, United States
| | - Roberta Rocca
- Interacting Minds Centre, Aarhus University, Aarhus, Denmark
| | - Ross W Blair
- Department of Psychology, Stanford University, Stanford, United States
| | | | - Jeff Mentch
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, United States
| | - James D Kent
- Department of Psychology, The University of Texas at Austin, Austin, United States
| | - Peer Herholz
- Montreal Neurological Institute, McGill University, Montreal, Canada
| | - Satrajit S Ghosh
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, United States
| | | | - Tal Yarkoni
- Department of Psychology, The University of Texas at Austin, Austin, United States
| |
Collapse
|
19
|
Nicholls VI, Alsbury-Nealy B, Krugliak A, Clarke A. Context effects on object recognition in real-world environments: A study protocol. Wellcome Open Res 2022. [DOI: 10.12688/wellcomeopenres.17856.1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/20/2022] Open
Abstract
Background: The environments that we live in impact on our ability to recognise objects, with recognition being facilitated when objects appear in expected locations (congruent) compared to unexpected locations (incongruent). However, these findings are based on experiments where the object is isolated from its environment. Moreover, it is not clear which components of the recognition process are impacted by the environment. In this experiment, we seek to examine the impact real world environments have on object recognition. Specifically, we will use mobile electroencephalography (mEEG) and augmented reality (AR) to investigate how the visual and semantic processing aspects of object recognition are changed by the environment. Methods: We will use AR to place congruent and incongruent virtual objects around indoor and outdoor environments. During the experiment a total of 34 participants will walk around the environments and find these objects while we record their eye movements and neural signals. We will perform two primary analyses. First, we will analyse the event-related potential (ERP) data using paired samples t-tests in the N300/400 time windows in an attempt to replicate congruency effects on the N300/400. Second, we will use representational similarity analysis (RSA) and computational models of vision and semantics to determine how visual and semantic processes are changed by congruency. Conclusions: Based on previous literature, we hypothesise that scene-object congruence would facilitate object recognition. For ERPs, we predict a congruency effect in the N300/N400, and for RSA we predict that higher level visual and semantic information will be represented earlier for congruent scenes than incongruent scenes. By collecting mEEG data while participants are exploring a real-world environment, we will be able to determine the impact of a natural context on object recognition, and the different processing stages of object recognition.
Collapse
|
20
|
Liu X, Dai Y, Xie H, Zhen Z. A studyforrest extension, MEG recordings while watching the audio-visual movie "Forrest Gump". Sci Data 2022; 9:206. [PMID: 35562378 PMCID: PMC9106652 DOI: 10.1038/s41597-022-01299-1] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/02/2021] [Accepted: 03/30/2022] [Indexed: 01/01/2023] Open
Abstract
Naturalistic stimuli, such as movies, are being increasingly used to map brain function because of their high ecological validity. The pioneering studyforrest and other naturalistic neuroimaging projects have provided free access to multiple movie-watching functional magnetic resonance imaging (fMRI) datasets to prompt the community for naturalistic experimental paradigms. However, sluggish blood-oxygenation-level-dependent fMRI signals are incapable of resolving neuronal activity with the temporal resolution at which it unfolds. Instead, magnetoencephalography (MEG) measures changes in the magnetic field produced by neuronal activity and is able to capture rich dynamics of the brain at the millisecond level while watching naturalistic movies. Herein, we present the first public prolonged MEG dataset collected from 11 participants while watching the 2 h long audio-visual movie "Forrest Gump". Minimally preprocessed data was also provided to facilitate the use of the dataset. As a studyforrest extension, we envision that this dataset, together with fMRI data from the studyforrest project, will serve as a foundation for exploring the neural dynamics of various cognitive functions in real-world contexts.
Collapse
Affiliation(s)
- Xingyu Liu
- Beijing Key Laboratory of Applied Experimental Psychology, National Demonstration Center for Experimental Psychology Education, Faculty of Psychology, Beijing Normal University, Beijing, China
| | - Yuxuan Dai
- Beijing Key Laboratory of Applied Experimental Psychology, National Demonstration Center for Experimental Psychology Education, Faculty of Psychology, Beijing Normal University, Beijing, China
| | - Hailun Xie
- Beijing Key Laboratory of Applied Experimental Psychology, National Demonstration Center for Experimental Psychology Education, Faculty of Psychology, Beijing Normal University, Beijing, China
| | - Zonglei Zhen
- Beijing Key Laboratory of Applied Experimental Psychology, National Demonstration Center for Experimental Psychology Education, Faculty of Psychology, Beijing Normal University, Beijing, China.
- State Key Laboratory of Cognitive Neuroscience and Learning & IDG/McGovern Institute for Brain Research, Beijing Normal University, Beijing, China.
| |
Collapse
|
21
|
Berezutskaya J, Vansteensel MJ, Aarnoutse EJ, Freudenburg ZV, Piantoni G, Branco MP, Ramsey NF. Open multimodal iEEG-fMRI dataset from naturalistic stimulation with a short audiovisual film. Sci Data 2022; 9:91. [PMID: 35314718 PMCID: PMC8938409 DOI: 10.1038/s41597-022-01173-0] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/10/2021] [Accepted: 01/24/2022] [Indexed: 12/19/2022] Open
Abstract
Intracranial human recordings are a valuable and rare resource of information about the brain. Making such data publicly available not only helps tackle reproducibility issues in science, it helps make more use of these valuable data. This is especially true for data collected using naturalistic tasks. Here, we describe a dataset collected from a large group of human subjects while they watched a short audiovisual film. The dataset has several unique features. First, it includes a large amount of intracranial electroencephalography (iEEG) data (51 participants, age range of 5-55 years, who all performed the same task). Second, it includes functional magnetic resonance imaging (fMRI) recordings (30 participants, age range of 7-47) during the same task. Eighteen participants performed both iEEG and fMRI versions of the task, non-simultaneously. Third, the data were acquired using a rich audiovisual stimulus, for which we provide detailed speech and video annotations. This dataset can be used to study neural mechanisms of multimodal perception and language comprehension, and similarity of neural signals across brain recording modalities.
Collapse
Affiliation(s)
- Julia Berezutskaya
- Brain Center, Department of Neurology and Neurosurgery, University Medical Center Utrecht, Utrecht, the Netherlands.
- Donders Institute for Brain, Cognition and Behaviour, Radboud University, Nijmegen, the Netherlands.
| | - Mariska J Vansteensel
- Brain Center, Department of Neurology and Neurosurgery, University Medical Center Utrecht, Utrecht, the Netherlands
| | - Erik J Aarnoutse
- Brain Center, Department of Neurology and Neurosurgery, University Medical Center Utrecht, Utrecht, the Netherlands
| | - Zachary V Freudenburg
- Brain Center, Department of Neurology and Neurosurgery, University Medical Center Utrecht, Utrecht, the Netherlands
| | - Giovanni Piantoni
- Brain Center, Department of Neurology and Neurosurgery, University Medical Center Utrecht, Utrecht, the Netherlands
| | - Mariana P Branco
- Brain Center, Department of Neurology and Neurosurgery, University Medical Center Utrecht, Utrecht, the Netherlands
| | - Nick F Ramsey
- Brain Center, Department of Neurology and Neurosurgery, University Medical Center Utrecht, Utrecht, the Netherlands
| |
Collapse
|
22
|
Camacho MC, Williams EM, Balser D, Kamojjala R, Sekar N, Steinberger D, Yarlagadda S, Perlman SB, Barch DM. EmoCodes: a Standardized Coding System for Socio-emotional Content in Complex Video Stimuli. AFFECTIVE SCIENCE 2022; 3:168-181. [PMID: 36046099 PMCID: PMC9383008 DOI: 10.1007/s42761-021-00100-7] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/19/2021] [Accepted: 12/21/2021] [Indexed: 06/10/2023]
Abstract
UNLABELLED Social information processing is vital for inferring emotional states in others, yet affective neuroscience has only begun to scratch the surface of how we represent emotional information in the brain. Most previous affective neuroscience work has used isolated stimuli such as static images of affective faces or scenes to probe affective processing. While this work has provided rich insight to the initial stages of emotion processing (encoding cues), activation to isolated stimuli provides limited insight into later phases of emotion processing such as interpretation of cues or interactions between cues and established cognitive schemas. Recent work has highlighted the potential value of using complex video stimuli to probe socio-emotional processing, highlighting the need to develop standardized video coding schemas as this exciting field expands. Toward that end, we present a standardized and open-source coding system for complex videos, two fully coded videos, and a video and code processing Python library. The EmoCodes manual coding system provides an externally validated and replicable system for coding complex cartoon stimuli, with future plans to validate the system for other video types. The emocodes Python library provides automated tools for extracting low-level features from video files as well as tools for summarizing and analyzing the manual codes for suitability of use in neuroimaging analysis. Materials can be freely accessed at https://emocodes.org/. These tools represent an important step toward replicable and standardized study of socio-emotional processing using complex video stimuli. SUPPLEMENTARY INFORMATION The online version contains supplementary material available at 10.1007/s42761-021-00100-7.
Collapse
Affiliation(s)
- M. Catalina Camacho
- Department of Psychological and Brain Sciences, Washington University in St. Louis, One Brookings Drive, St. Louis, MO 63130 USA
| | - Elizabeth M. Williams
- Department of Psychological and Brain Sciences, Washington University in St. Louis, One Brookings Drive, St. Louis, MO 63130 USA
| | - Dori Balser
- Department of Psychological and Brain Sciences, Washington University in St. Louis, One Brookings Drive, St. Louis, MO 63130 USA
| | - Ruchika Kamojjala
- Department of Psychological and Brain Sciences, Washington University in St. Louis, One Brookings Drive, St. Louis, MO 63130 USA
| | - Nikhil Sekar
- Department of Psychological and Brain Sciences, Washington University in St. Louis, One Brookings Drive, St. Louis, MO 63130 USA
| | - David Steinberger
- Department of Psychological and Brain Sciences, Washington University in St. Louis, One Brookings Drive, St. Louis, MO 63130 USA
| | - Sishir Yarlagadda
- Department of Psychological and Brain Sciences, Washington University in St. Louis, One Brookings Drive, St. Louis, MO 63130 USA
| | - Susan B. Perlman
- Department of Psychiatry, Washington University in St. Louis, 4444 Forest Park Drive, MO 63110 St. Louis, USA
| | - Deanna M. Barch
- Department of Psychological and Brain Sciences, Washington University in St. Louis, One Brookings Drive, St. Louis, MO 63130 USA
| |
Collapse
|
23
|
Kirk PA, Robinson OJ, Skipper JI. Anxiety and amygdala connectivity during movie-watching. Neuropsychologia 2022; 169:108194. [PMID: 35245529 PMCID: PMC8987737 DOI: 10.1016/j.neuropsychologia.2022.108194] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/03/2021] [Revised: 11/08/2021] [Accepted: 02/27/2022] [Indexed: 12/13/2022]
Abstract
Rodent and human studies have implicated an amygdala-prefrontal circuit during threat processing. One possibility is that while amygdala activity underlies core features of anxiety (e.g. detection of salient information), prefrontal cortices (i.e. dorsomedial prefrontal/anterior cingulate cortex) entrain its responsiveness. To date, this has been established in tightly controlled paradigms (predominantly using static face perception tasks) but has not been extended to more naturalistic settings. Consequently, using ‘movie fMRI’—in which participants watch ecologically-rich movie stimuli rather than constrained cognitive tasks—we sought to test whether individual differences in anxiety correlate with the degree of face-dependent amygdala-prefrontal coupling in two independent samples. Analyses suggested increased face-dependent superior parietal activation and decreased speech-dependent auditory cortex activation as a function of anxiety. However, we failed to find evidence for anxiety-dependent connectivity, neither in our stimulus-dependent or -independent analyses. Our findings suggest that work using experimentally constrained tasks may not replicate in more ecologically valid settings and, moreover, highlight the importance of testing the generalizability of neuroimaging findings outside of the original context. Using ‘movie fMRI’, we tested whether trait anxiety correlates with face-dependent amygdala-prefrontal coupling. We observed altered superior parietal activation to faces and auditory cortex activation to speech as a function of anxiety. We failed to find evidence for anxiety-dependent amygdala-dmPFC connectivity in stimulus-dependent or -independent analyses. Our findings highlight the importance of testing the generalizability of neuroimaging findings outside of the original context.
Collapse
Affiliation(s)
- Peter A Kirk
- UCL Institute of Cognitive Neuroscience, UK; UCL Experimental Psychology, UK.
| | - Oliver J Robinson
- UCL Institute of Cognitive Neuroscience, UK; UCL Clinical, Educational and Health Psychology, UK
| | | |
Collapse
|
24
|
Towards real-world neuroscience using mobile EEG and augmented reality. Sci Rep 2022; 12:2291. [PMID: 35145166 PMCID: PMC8831466 DOI: 10.1038/s41598-022-06296-3] [Citation(s) in RCA: 10] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/21/2021] [Accepted: 01/25/2022] [Indexed: 01/10/2023] Open
Abstract
Our visual environment impacts multiple aspects of cognition including perception, attention and memory, yet most studies traditionally remove or control the external environment. As a result, we have a limited understanding of neurocognitive processes beyond the controlled lab environment. Here, we aim to study neural processes in real-world environments, while also maintaining a degree of control over perception. To achieve this, we combined mobile EEG (mEEG) and augmented reality (AR), which allows us to place virtual objects into the real world. We validated this AR and mEEG approach using a well-characterised cognitive response-the face inversion effect. Participants viewed upright and inverted faces in three EEG tasks (1) a lab-based computer task, (2) walking through an indoor environment while seeing face photographs, and (3) walking through an indoor environment while seeing virtual faces. We find greater low frequency EEG activity for inverted compared to upright faces in all experimental tasks, demonstrating that cognitively relevant signals can be extracted from mEEG and AR paradigms. This was established in both an epoch-based analysis aligned to face events, and a GLM-based approach that incorporates continuous EEG signals and face perception states. Together, this research helps pave the way to exploring neurocognitive processes in real-world environments while maintaining experimental control using AR.
Collapse
|
25
|
Scan Once, Analyse Many: Using Large Open-Access Neuroimaging Datasets to Understand the Brain. Neuroinformatics 2022; 20:109-137. [PMID: 33974213 PMCID: PMC8111663 DOI: 10.1007/s12021-021-09519-6] [Citation(s) in RCA: 9] [Impact Index Per Article: 4.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 03/07/2021] [Indexed: 02/06/2023]
Abstract
We are now in a time of readily available brain imaging data. Not only are researchers now sharing data more than ever before, but additionally large-scale data collecting initiatives are underway with the vision that many future researchers will use the data for secondary analyses. Here I provide an overview of available datasets and some example use cases. Example use cases include examining individual differences, more robust findings, reproducibility-both in public input data and availability as a replication sample, and methods development. I further discuss a variety of considerations associated with using existing data and the opportunities associated with large datasets. Suggestions for further readings on general neuroimaging and topic-specific discussions are also provided.
Collapse
|
26
|
Allen EJ, St-Yves G, Wu Y, Breedlove JL, Prince JS, Dowdle LT, Nau M, Caron B, Pestilli F, Charest I, Hutchinson JB, Naselaris T, Kay K. A massive 7T fMRI dataset to bridge cognitive neuroscience and artificial intelligence. Nat Neurosci 2022; 25:116-126. [PMID: 34916659 DOI: 10.1038/s41593-021-00962-x] [Citation(s) in RCA: 81] [Impact Index Per Article: 40.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/01/2021] [Accepted: 10/12/2021] [Indexed: 11/09/2022]
Abstract
Extensive sampling of neural activity during rich cognitive phenomena is critical for robust understanding of brain function. Here we present the Natural Scenes Dataset (NSD), in which high-resolution functional magnetic resonance imaging responses to tens of thousands of richly annotated natural scenes were measured while participants performed a continuous recognition task. To optimize data quality, we developed and applied novel estimation and denoising techniques. Simple visual inspections of the NSD data reveal clear representational transformations along the ventral visual pathway. Further exemplifying the inferential power of the dataset, we used NSD to build and train deep neural network models that predict brain activity more accurately than state-of-the-art models from computer vision. NSD also includes substantial resting-state and diffusion data, enabling network neuroscience perspectives to constrain and enhance models of perception and memory. Given its unprecedented scale, quality and breadth, NSD opens new avenues of inquiry in cognitive neuroscience and artificial intelligence.
Collapse
Affiliation(s)
- Emily J Allen
- Center for Magnetic Resonance Research (CMRR), Department of Radiology, University of Minnesota, Minneapolis, MN, USA
- Department of Psychology, University of Minnesota, Minneapolis, MN, USA
| | - Ghislain St-Yves
- Department of Neuroscience, Medical University of South Carolina, Charleston, SC, USA
- Department of Neuroscience, University of Minnesota, Minneapolis, MN, USA
| | - Yihan Wu
- Graduate Program in Cognitive Science, University of Minnesota, Minneapolis, MN, USA
| | - Jesse L Breedlove
- Department of Neuroscience, Medical University of South Carolina, Charleston, SC, USA
- Department of Psychology, University of Minnesota, Minneapolis, MN, USA
| | - Jacob S Prince
- Department of Psychology, Carnegie Mellon University, Pittsburgh, PA, USA
- Department of Psychology, Harvard University, Cambridge, MA, USA
| | - Logan T Dowdle
- Department of Neuroscience, Center for Magnetic Resonance Research (CMRR), University of Minnesota, Minneapolis, MN, USA
- Department of Neurosurgery, Center for Magnetic Resonance Research (CMRR), University of Minnesota, Minneapolis, MN, USA
| | - Matthias Nau
- National Institute of Mental Health (NIMH), Bethesda MD, USA
| | - Brad Caron
- Program in Neuroscience, Indiana University, Bloomington IN, USA
- Program in Vision Science, Indiana University, Bloomington IN, USA
| | - Franco Pestilli
- Department of Psychology, University of Texas at Austin, Austin, TX, USA
- Center for Perceptual Systems, University of Texas at Austin, Austin, TX, USA
- Institute for Neuroscience, University of Texas at Austin, Austin, TX, USA
| | - Ian Charest
- Center for Human Brain Health, School of Psychology, University of Birmingham, Birmingham, UK
- cerebrUM, Département de Psychologie, Université de Montréal, Montréal QC, Canada
| | | | - Thomas Naselaris
- Department of Neuroscience, Medical University of South Carolina, Charleston, SC, USA
- Department of Neuroscience, University of Minnesota, Minneapolis, MN, USA
| | - Kendrick Kay
- Center for Magnetic Resonance Research (CMRR), Department of Radiology, University of Minnesota, Minneapolis, MN, USA.
| |
Collapse
|
27
|
Functional selectivity for social interaction perception in the human superior temporal sulcus during natural viewing. Neuroimage 2021; 245:118741. [PMID: 34800663 DOI: 10.1016/j.neuroimage.2021.118741] [Citation(s) in RCA: 18] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/08/2021] [Revised: 09/15/2021] [Accepted: 11/16/2021] [Indexed: 11/22/2022] Open
Abstract
Recognizing others' social interactions is a crucial human ability. Using simple stimuli, previous studies have shown that social interactions are selectively processed in the superior temporal sulcus (STS), but prior work with movies has suggested that social interactions are processed in the medial prefrontal cortex (mPFC), part of the theory of mind network. It remains unknown to what extent social interaction selectivity is observed in real world stimuli when controlling for other covarying perceptual and social information, such as faces, voices, and theory of mind. The current study utilizes a functional magnetic resonance imaging (fMRI) movie paradigm and advanced machine learning methods to uncover the brain mechanisms uniquely underlying naturalistic social interaction perception. We analyzed two publicly available fMRI datasets, collected while both male and female human participants (n = 17 and 18) watched two different commercial movies in the MRI scanner. By performing voxel-wise encoding and variance partitioning analyses, we found that broad social-affective features predict neural responses in social brain regions, including the STS and mPFC. However, only the STS showed robust and unique selectivity specifically to social interactions, independent from other covarying features. This selectivity was observed across two separate fMRI datasets. These findings suggest that naturalistic social interaction perception recruits dedicated neural circuity in the STS, separate from the theory of mind network, and is a critical dimension of human social understanding.
Collapse
|
28
|
Skipper JI, Aliko S, Brown S, Jo YJ, Lo S, Molimpakis E, Lametti DR. Reorganization of the Neurobiology of Language After Sentence Overlearning. Cereb Cortex 2021; 32:2447-2468. [PMID: 34585723 PMCID: PMC9157312 DOI: 10.1093/cercor/bhab354] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/29/2021] [Revised: 08/27/2021] [Accepted: 08/29/2021] [Indexed: 11/14/2022] Open
Abstract
It is assumed that there are a static set of "language regions" in the brain. Yet, language comprehension engages regions well beyond these, and patients regularly produce familiar "formulaic" expressions when language regions are severely damaged. These suggest that the neurobiology of language is not fixed but varies with experiences, like the extent of word sequence learning. We hypothesized that perceiving overlearned sentences is supported by speech production and not putative language regions. Participants underwent 2 sessions of behavioral testing and functional magnetic resonance imaging (fMRI). During the intervening 15 days, they repeated 2 sentences 30 times each, twice a day. In both fMRI sessions, they "passively" listened to those sentences, novel sentences, and produced sentences. Behaviorally, evidence for overlearning included a 2.1-s decrease in reaction times to predict the final word in overlearned sentences. This corresponded to the recruitment of sensorimotor regions involved in sentence production, inactivation of temporal and inferior frontal regions involved in novel sentence listening, and a 45% change in global network organization. Thus, there was a profound whole-brain reorganization following sentence overlearning, out of "language" and into sensorimotor regions. The latter are generally preserved in aphasia and Alzheimer's disease, perhaps explaining residual abilities with formulaic expressions in both.
Collapse
Affiliation(s)
| | - Sarah Aliko
- Experimental Psychology, University College London, London, UK.,London Interdisciplinary Biosciences Consortium, University College London, London, UK
| | - Stephen Brown
- Natural Sciences, University College London, London, UK
| | - Yoon Ju Jo
- Experimental Psychology, University College London, London, UK
| | - Serena Lo
- Speech and Language Sciences, University College London, London, UK
| | - Emilia Molimpakis
- Wellcome Centre for Human Neuroimaging, University College London, London, UK
| | - Daniel R Lametti
- Experimental Psychology, University College London, London, UK.,Department of Psychology, Acadia University, Nova Scotia, Canada
| |
Collapse
|
29
|
Nastase SA, Liu YF, Hillman H, Zadbood A, Hasenfratz L, Keshavarzian N, Chen J, Honey CJ, Yeshurun Y, Regev M, Nguyen M, Chang CHC, Baldassano C, Lositsky O, Simony E, Chow MA, Leong YC, Brooks PP, Micciche E, Choe G, Goldstein A, Vanderwal T, Halchenko YO, Norman KA, Hasson U. The "Narratives" fMRI dataset for evaluating models of naturalistic language comprehension. Sci Data 2021; 8:250. [PMID: 34584100 PMCID: PMC8479122 DOI: 10.1038/s41597-021-01033-3] [Citation(s) in RCA: 23] [Impact Index Per Article: 7.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/04/2021] [Accepted: 08/18/2021] [Indexed: 02/08/2023] Open
Abstract
The "Narratives" collection aggregates a variety of functional MRI datasets collected while human subjects listened to naturalistic spoken stories. The current release includes 345 subjects, 891 functional scans, and 27 diverse stories of varying duration totaling ~4.6 hours of unique stimuli (~43,000 words). This data collection is well-suited for naturalistic neuroimaging analysis, and is intended to serve as a benchmark for models of language and narrative comprehension. We provide standardized MRI data accompanied by rich metadata, preprocessed versions of the data ready for immediate use, and the spoken story stimuli with time-stamped phoneme- and word-level transcripts. All code and data are publicly available with full provenance in keeping with current best practices in transparent and reproducible neuroimaging.
Collapse
Affiliation(s)
- Samuel A Nastase
- Princeton Neuroscience Institute and Department of Psychology, Princeton University, Princeton, NJ, USA.
| | - Yun-Fei Liu
- Department of Psychological and Brain Sciences, Johns Hopkins University, Baltimore, MD, USA
| | - Hanna Hillman
- Princeton Neuroscience Institute and Department of Psychology, Princeton University, Princeton, NJ, USA
| | - Asieh Zadbood
- Princeton Neuroscience Institute and Department of Psychology, Princeton University, Princeton, NJ, USA
| | - Liat Hasenfratz
- Princeton Neuroscience Institute and Department of Psychology, Princeton University, Princeton, NJ, USA
| | - Neggin Keshavarzian
- Princeton Neuroscience Institute and Department of Psychology, Princeton University, Princeton, NJ, USA
| | - Janice Chen
- Department of Psychological and Brain Sciences, Johns Hopkins University, Baltimore, MD, USA
| | - Christopher J Honey
- Department of Psychological and Brain Sciences, Johns Hopkins University, Baltimore, MD, USA
| | - Yaara Yeshurun
- School of Psychological Sciences, Tel Aviv University, Tel Aviv, Israel
| | - Mor Regev
- Montreal Neurological Institute, McGill University, Montreal, QC, Canada
| | - Mai Nguyen
- Princeton Neuroscience Institute and Department of Psychology, Princeton University, Princeton, NJ, USA
| | - Claire H C Chang
- Princeton Neuroscience Institute and Department of Psychology, Princeton University, Princeton, NJ, USA
| | | | - Olga Lositsky
- Department of Cognitive, Linguistic and Psychological Sciences, Brown University, Providence, RI, USA
| | - Erez Simony
- Faculty of Electrical Engineering, Holon Institute of Technology, Holon, Israel
- Department of Neurobiology, Weizmann Institute of Science, Rehovot, Israel
| | | | - Yuan Chang Leong
- Helen Wills Neuroscience Institute, University of California, Berkeley, CA, USA
| | - Paula P Brooks
- Princeton Neuroscience Institute and Department of Psychology, Princeton University, Princeton, NJ, USA
| | - Emily Micciche
- Peabody College, Vanderbilt University, Nashville, TN, USA
| | - Gina Choe
- Princeton Neuroscience Institute and Department of Psychology, Princeton University, Princeton, NJ, USA
| | - Ariel Goldstein
- Princeton Neuroscience Institute and Department of Psychology, Princeton University, Princeton, NJ, USA
| | - Tamara Vanderwal
- Department of Psychiatry, University of British Columbia, and BC Children's Hospital Research Institute, Vancouver, BC, Canada
| | - Yaroslav O Halchenko
- Department of Psychological and Brain Sciences and Department of Computer Science, Dartmouth College, Hanover, NH, USA
| | - Kenneth A Norman
- Princeton Neuroscience Institute and Department of Psychology, Princeton University, Princeton, NJ, USA
| | - Uri Hasson
- Princeton Neuroscience Institute and Department of Psychology, Princeton University, Princeton, NJ, USA
| |
Collapse
|
30
|
Zhang Y, Kim JH, Brang D, Liu Z. Naturalistic Stimuli: A Paradigm for Multi-Scale Functional Characterization of the Human Brain. CURRENT OPINION IN BIOMEDICAL ENGINEERING 2021; 19:100298. [PMID: 34423178 PMCID: PMC8376216 DOI: 10.1016/j.cobme.2021.100298] [Citation(s) in RCA: 12] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/23/2022]
Abstract
Movies, audio stories, and virtual reality are increasingly used as stimuli for functional brain imaging. Such naturalistic paradigms are in sharp contrast to the tradition of experimental reductionism in neuroscience research. Being complex, dynamic, and diverse, naturalistic stimuli set up a more ecologically relevant condition and induce highly reproducible brain responses across a wide range of spatiotemporal scales. Here, we review recent technical advances and scientific findings on imaging the brain under naturalistic stimuli. Then we elaborate on the premise of using naturalistic paradigms for multi-scale, multi-modal, and high-throughput functional characterization of the human brain. We further highlight the growing potential of using deep learning models to infer neural information processing from brain responses to naturalistic stimuli. Lastly, we advocate large-scale collaborations to combine brain imaging and recording data across experiments, subjects, and labs that use the same set of naturalistic stimuli.
Collapse
Affiliation(s)
- Yizhen Zhang
- Department of Electrical Engineering and Computer Science, University of Michigan
| | - Jung-Hoon Kim
- Department of Biomedical Engineering, University of Michigan
- Weldon School of Biomedical Engineering, Purdue University
| | - David Brang
- Department of Psychology, University of Michigan
| | - Zhongming Liu
- Department of Electrical Engineering and Computer Science, University of Michigan
- Department of Biomedical Engineering, University of Michigan
| |
Collapse
|
31
|
|
32
|
Song H, Rosenberg MD. Predicting attention across time and contexts with functional brain connectivity. Curr Opin Behav Sci 2021. [DOI: 10.1016/j.cobeha.2020.12.007] [Citation(s) in RCA: 10] [Impact Index Per Article: 3.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/14/2022]
|
33
|
Topazian HM, Gumbo A, Brandt K, Kayange M, Smith JS, Edwards JK, Goel V, Mvalo T, Emch M, Pettifor AE, Juliano JJ, Hoffman I. Effectiveness of a national mass distribution campaign of long-lasting insecticide-treated nets and indoor residual spraying on clinical malaria in Malawi, 2018-2020. BMJ Glob Health 2021; 6:bmjgh-2021-005447. [PMID: 33947708 PMCID: PMC8098915 DOI: 10.1136/bmjgh-2021-005447] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/18/2021] [Revised: 04/06/2021] [Accepted: 04/16/2021] [Indexed: 11/16/2022] Open
Abstract
Introduction Malawi’s malaria burden is primarily assessed via cross-sectional national household surveys. However, malaria is spatially and temporally heterogenous and no analyses have been performed at a subdistrict level throughout the course of a year. The WHO recommends mass distribution of long-lasting insecticide-treated bed nets (LLINs) every 3 years, but a national longitudinal evaluation has never been conducted in Malawi to determine LLIN effectiveness lifespans. Methods Using District Health Information Software 2 (DHIS2) health facility data, available from January 2018 to June 2020, we assessed malaria risk before and after a mass distribution campaign, stratifying by age group and comparing risk differences (RDs) by LLIN type or annual application of indoor residual spraying (IRS). Results 711 health facilities contributed 20 962 facility reports over 30 months. After national distribution of 10.7 million LLINs and IRS in limited settings, malaria risk decreased from 25.6 to 16.7 cases per 100 people from 2018 to 2019 high transmission seasons, and rebounded to 23.2 in 2020, resulting in significant RDs of −8.9 in 2019 and −2.4 in 2020 as compared with 2018. Piperonyl butoxide (PBO)-treated LLINs were more effective than pyrethroid-treated LLINs, with adjusted RDs of −2.3 (95% CI −2.7 to −1.9) and −1.5 (95% CI −2.0 to −1.0) comparing 2019 and 2020 high transmission seasons to 2018. Use of IRS sustained protection with adjusted RDs of −1.4 (95% CI −2.0 to −0.9) and −2.8% (95% CI −3.5 to −2.2) relative to pyrethroid-treated LLINs. Overall, 12 of 28 districts (42.9%) experienced increases in malaria risk in from 2018 to 2020. Conclusion LLINs in Malawi have a limited effectiveness lifespan and IRS and PBO-treated LLINs perform better than pyrethroid-treated LLINs, perhaps due to net repurposing and insecticide-resistance. DHIS2 provides a compelling framework in which to examine localised malaria trends and evaluate ongoing interventions.
Collapse
Affiliation(s)
- Hillary M Topazian
- Department of Epidemiology, University of North Carolina at Chapel Hill Gillings School of Global Public Health, Chapel Hill, North Carolina, USA
| | - Austin Gumbo
- National Malaria Control Programme, Malawi Ministry of Health, Lilongwe, Malawi
| | - Katerina Brandt
- Department of Geography, University of North Carolina at Chapel Hill Graduate School, Chapel Hill, North Carolina, USA.,Carolina Population Center, University of North Carolina at Chapel Hill, Chapel Hill, North Carolina, USA
| | - Michael Kayange
- National Malaria Control Programme, Malawi Ministry of Health, Lilongwe, Malawi
| | - Jennifer S Smith
- Department of Epidemiology, University of North Carolina at Chapel Hill Gillings School of Global Public Health, Chapel Hill, North Carolina, USA
| | - Jessie K Edwards
- Department of Epidemiology, University of North Carolina at Chapel Hill Gillings School of Global Public Health, Chapel Hill, North Carolina, USA
| | - Varun Goel
- Department of Geography, University of North Carolina at Chapel Hill Graduate School, Chapel Hill, North Carolina, USA.,Carolina Population Center, University of North Carolina at Chapel Hill, Chapel Hill, North Carolina, USA
| | - Tisungane Mvalo
- University of North Carolina Project-Malawi, Lilongwe, Malawi.,Department of Pediatrics, University of North Carolina at Chapel Hill School of Medicine, Chapel Hill, North Carolina, USA
| | - Michael Emch
- Department of Geography, University of North Carolina at Chapel Hill Graduate School, Chapel Hill, North Carolina, USA.,Carolina Population Center, University of North Carolina at Chapel Hill, Chapel Hill, North Carolina, USA
| | - Audrey E Pettifor
- Department of Epidemiology, University of North Carolina at Chapel Hill Gillings School of Global Public Health, Chapel Hill, North Carolina, USA.,Carolina Population Center, University of North Carolina at Chapel Hill, Chapel Hill, North Carolina, USA
| | - Jonathan J Juliano
- Division of Infectious Diseases, University of North Carolina at Chapel Hill School of Medicine, Chapel Hill, North Carolina, USA
| | - Irving Hoffman
- University of North Carolina Project-Malawi, Lilongwe, Malawi.,Institute for Global Health and Infectious Diseases, University of North Carolina at Chapel Hill School of Medicine, Chapel Hill, North Carolina, USA
| |
Collapse
|
34
|
Saarimäki H. Naturalistic Stimuli in Affective Neuroimaging: A Review. Front Hum Neurosci 2021; 15:675068. [PMID: 34220474 PMCID: PMC8245682 DOI: 10.3389/fnhum.2021.675068] [Citation(s) in RCA: 10] [Impact Index Per Article: 3.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/02/2021] [Accepted: 05/17/2021] [Indexed: 11/13/2022] Open
Abstract
Naturalistic stimuli such as movies, music, and spoken and written stories elicit strong emotions and allow brain imaging of emotions in close-to-real-life conditions. Emotions are multi-component phenomena: relevant stimuli lead to automatic changes in multiple functional components including perception, physiology, behavior, and conscious experiences. Brain activity during naturalistic stimuli reflects all these changes, suggesting that parsing emotion-related processing during such complex stimulation is not a straightforward task. Here, I review affective neuroimaging studies that have employed naturalistic stimuli to study emotional processing, focusing especially on experienced emotions. I argue that to investigate emotions with naturalistic stimuli, we need to define and extract emotion features from both the stimulus and the observer.
Collapse
Affiliation(s)
- Heini Saarimäki
- Human Information Processing Laboratory, Faculty of Social Sciences, Tampere University, Tampere, Finland
| |
Collapse
|
35
|
Häusler CO, Hanke M. A studyforrest extension, an annotation of spoken language in the German dubbed movie "Forrest Gump" and its audio-description. F1000Res 2021; 10:54. [PMID: 33732435 PMCID: PMC7921887 DOI: 10.12688/f1000research.27621.1] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Accepted: 01/12/2021] [Indexed: 11/20/2022] Open
Abstract
Here we present an annotation of speech in the audio-visual movie "Forrest Gump" and its audio-description for a visually impaired audience, as an addition to a large public functional brain imaging dataset ( studyforrest.org). The annotation provides information about the exact timing of each of the more than 2500 spoken sentences, 16,000 words (including 202 non-speech vocalizations), 66,000 phonemes, and their corresponding speaker. Additionally, for every word, we provide lemmatization, a simple part-of-speech-tagging (15 grammatical categories), a detailed part-of-speech tagging (43 grammatical categories), syntactic dependencies, and a semantic analysis based on word embedding which represents each word in a 300-dimensional semantic space. To validate the dataset's quality, we build a model of hemodynamic brain activity based on information drawn from the annotation. Results suggest that the annotation's content and quality enable independent researchers to create models of brain activity correlating with a variety of linguistic aspects under conditions of near-real-life complexity.
Collapse
Affiliation(s)
- Christian Olaf Häusler
- Institute of Neuroscience and Medicine, Brain & Behaviour (INM-7), Research Center Jülich, Jülich, Nordrhein-Westfalen, 52425, Germany
- Institute of Systems Neuroscience, Medical Faculty, Heinrich Heine University, Düsseldorf, Nordrhein-Westfalen, 40225, Germany
| | - Michael Hanke
- Institute of Neuroscience and Medicine, Brain & Behaviour (INM-7), Research Center Jülich, Jülich, Nordrhein-Westfalen, 52425, Germany
- Institute of Systems Neuroscience, Medical Faculty, Heinrich Heine University, Düsseldorf, Nordrhein-Westfalen, 40225, Germany
| |
Collapse
|
36
|
Visconti di Oleggio Castello M, Chauhan V, Jiahui G, Gobbini MI. An fMRI dataset in response to "The Grand Budapest Hotel", a socially-rich, naturalistic movie. Sci Data 2020; 7:383. [PMID: 33177526 PMCID: PMC7658985 DOI: 10.1038/s41597-020-00735-4] [Citation(s) in RCA: 17] [Impact Index Per Article: 4.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/28/2020] [Accepted: 10/27/2020] [Indexed: 11/18/2022] Open
Abstract
Naturalistic stimuli evoke strong, consistent, and information-rich patterns of brain activity, and engage large extents of the human brain. They allow researchers to compare highly similar brain responses across subjects, and to study how complex representations are encoded in brain activity. Here, we describe and share a dataset where 25 subjects watched part of the feature film "The Grand Budapest Hotel" by Wes Anderson. The movie has a large cast with many famous actors. Throughout the story, the camera shots highlight faces and expressions, which are fundamental to understand the complex narrative of the movie. This movie was chosen to sample brain activity specifically related to social interactions and face processing. This dataset provides researchers with fMRI data that can be used to explore social cognitive processes and face processing, adding to the existing neuroimaging datasets that sample brain activity with naturalistic movies.
Collapse
Affiliation(s)
| | - Vassiki Chauhan
- Center for Cognitive Neuroscience, Dartmouth College, Hanover, USA
| | - Guo Jiahui
- Center for Cognitive Neuroscience, Dartmouth College, Hanover, USA
| | - M Ida Gobbini
- Cognitive Science Program, Dartmouth College, Hanover, USA.
- Dipartimento di Medicina Specialistica, Diagnostica e Sperimentale, University of Bologna, Bologna, Italy.
| |
Collapse
|