1
|
Gong Z, Zhou M, Dai Y, Wen Y, Liu Y, Zhen Z. A large-scale fMRI dataset for the visual processing of naturalistic scenes. Sci Data 2023; 10:559. [PMID: 37612327 PMCID: PMC10447576 DOI: 10.1038/s41597-023-02471-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/27/2023] [Accepted: 08/14/2023] [Indexed: 08/25/2023] Open
Abstract
One ultimate goal of visual neuroscience is to understand how the brain processes visual stimuli encountered in the natural environment. Achieving this goal requires records of brain responses under massive amounts of naturalistic stimuli. Although the scientific community has put a lot of effort into collecting large-scale functional magnetic resonance imaging (fMRI) data under naturalistic stimuli, more naturalistic fMRI datasets are still urgently needed. We present here the Natural Object Dataset (NOD), a large-scale fMRI dataset containing responses to 57,120 naturalistic images from 30 participants. NOD strives for a balance between sampling variation between individuals and sampling variation between stimuli. This enables NOD to be utilized not only for determining whether an observation is generalizable across many individuals, but also for testing whether a response pattern is generalized to a variety of naturalistic stimuli. We anticipate that the NOD together with existing naturalistic neuroimaging datasets will serve as a new impetus for our understanding of the visual processing of naturalistic stimuli.
Collapse
Affiliation(s)
- Zhengxin Gong
- Beijing Key Laboratory of Applied Experimental Psychology, Faculty of Psychology, Beijing Normal University, Beijing, 100875, China
| | - Ming Zhou
- State Key Laboratory of Cognitive Neuroscience and Learning & IDG/McGovern Institute for Brain Research, Beijing Normal University, Beijing, 100875, China
| | - Yuxuan Dai
- Beijing Key Laboratory of Applied Experimental Psychology, Faculty of Psychology, Beijing Normal University, Beijing, 100875, China
| | - Yushan Wen
- Beijing Key Laboratory of Applied Experimental Psychology, Faculty of Psychology, Beijing Normal University, Beijing, 100875, China
| | - Youyi Liu
- State Key Laboratory of Cognitive Neuroscience and Learning & IDG/McGovern Institute for Brain Research, Beijing Normal University, Beijing, 100875, China.
| | - Zonglei Zhen
- Beijing Key Laboratory of Applied Experimental Psychology, Faculty of Psychology, Beijing Normal University, Beijing, 100875, China.
- State Key Laboratory of Cognitive Neuroscience and Learning & IDG/McGovern Institute for Brain Research, Beijing Normal University, Beijing, 100875, China.
| |
Collapse
|
2
|
Shahdloo M, Çelik E, Urgen BA, Gallant JL, Çukur T. Task-Dependent Warping of Semantic Representations during Search for Visual Action Categories. J Neurosci 2022; 42:6782-6799. [PMID: 35863889 PMCID: PMC9436022 DOI: 10.1523/jneurosci.1372-21.2022] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/01/2021] [Revised: 06/29/2022] [Accepted: 07/06/2022] [Indexed: 11/21/2022] Open
Abstract
Object and action perception in cluttered dynamic natural scenes relies on efficient allocation of limited brain resources to prioritize the attended targets over distractors. It has been suggested that during visual search for objects, distributed semantic representation of hundreds of object categories is warped to expand the representation of targets. Yet, little is known about whether and where in the brain visual search for action categories modulates semantic representations. To address this fundamental question, we studied brain activity recorded from five subjects (one female) via functional magnetic resonance imaging while they viewed natural movies and searched for either communication or locomotion actions. We find that attention directed to action categories elicits tuning shifts that warp semantic representations broadly across neocortex and that these shifts interact with intrinsic selectivity of cortical voxels for target actions. These results suggest that attention serves to facilitate task performance during social interactions by dynamically shifting semantic selectivity toward target actions and that tuning shifts are a general feature of conceptual representations in the brain.SIGNIFICANCE STATEMENT The ability to swiftly perceive the actions and intentions of others is a crucial skill for humans that relies on efficient allocation of limited brain resources to prioritize the attended targets over distractors. However, little is known about the nature of high-level semantic representations during natural visual search for action categories. Here, we provide the first evidence showing that attention significantly warps semantic representations by inducing tuning shifts in single cortical voxels, broadly spread across occipitotemporal, parietal, prefrontal, and cingulate cortices. This dynamic attentional mechanism can facilitate action perception by efficiently allocating neural resources to accentuate the representation of task-relevant action categories.
Collapse
Affiliation(s)
- Mo Shahdloo
- Wellcome Centre for Integrative Neuroimaging, Department of Experimental Psychology, University of Oxford, Oxford OX3 9DU, United Kingdom
- National Magnetic Resonance Research Centre, Bilkent University, 06800 Ankara, Turkey
- Departments of Electrical and Electronics Engineering and
| | - Emin Çelik
- National Magnetic Resonance Research Centre, Bilkent University, 06800 Ankara, Turkey
- Neuroscience Program, Aysel Sabuncu Brain Research Centre, Bilkent University, 06800 Ankara, Turkey
| | - Burcu A Urgen
- National Magnetic Resonance Research Centre, Bilkent University, 06800 Ankara, Turkey
- Psychology, Bilkent University, 06800 Ankara, Turkey
- Neuroscience Program, Aysel Sabuncu Brain Research Centre, Bilkent University, 06800 Ankara, Turkey
| | - Jack L Gallant
- Helen Wills Neuroscience Institute, University of California, Berkeley, Berkeley, California 94720
| | - Tolga Çukur
- National Magnetic Resonance Research Centre, Bilkent University, 06800 Ankara, Turkey
- Departments of Electrical and Electronics Engineering and
- Neuroscience Program, Aysel Sabuncu Brain Research Centre, Bilkent University, 06800 Ankara, Turkey
- Helen Wills Neuroscience Institute, University of California, Berkeley, Berkeley, California 94720
| |
Collapse
|
3
|
de la Vega A, Rocca R, Blair RW, Markiewicz CJ, Mentch J, Kent JD, Herholz P, Ghosh SS, Poldrack RA, Yarkoni T. Neuroscout, a unified platform for generalizable and reproducible fMRI research. eLife 2022; 11:79277. [PMID: 36040302 PMCID: PMC9489206 DOI: 10.7554/elife.79277] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/06/2022] [Accepted: 08/27/2022] [Indexed: 11/28/2022] Open
Abstract
Functional magnetic resonance imaging (fMRI) has revolutionized cognitive neuroscience, but methodological barriers limit the generalizability of findings from the lab to the real world. Here, we present Neuroscout, an end-to-end platform for analysis of naturalistic fMRI data designed to facilitate the adoption of robust and generalizable research practices. Neuroscout leverages state-of-the-art machine learning models to automatically annotate stimuli from dozens of fMRI studies using naturalistic stimuli—such as movies and narratives—allowing researchers to easily test neuroscientific hypotheses across multiple ecologically-valid datasets. In addition, Neuroscout builds on a robust ecosystem of open tools and standards to provide an easy-to-use analysis builder and a fully automated execution engine that reduce the burden of reproducible research. Through a series of meta-analytic case studies, we validate the automatic feature extraction approach and demonstrate its potential to support more robust fMRI research. Owing to its ease of use and a high degree of automation, Neuroscout makes it possible to overcome modeling challenges commonly arising in naturalistic analysis and to easily scale analyses within and across datasets, democratizing generalizable fMRI research.
Collapse
Affiliation(s)
- Alejandro de la Vega
- Department of Psychology, The University of Texas at Austin, Austin, United States
| | - Roberta Rocca
- Interacting Minds Centre, Aarhus University, Aarhus, Denmark
| | - Ross W Blair
- Department of Psychology, Stanford University, Stanford, United States
| | | | - Jeff Mentch
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, United States
| | - James D Kent
- Department of Psychology, The University of Texas at Austin, Austin, United States
| | - Peer Herholz
- Montreal Neurological Institute, McGill University, Montreal, Canada
| | - Satrajit S Ghosh
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, United States
| | | | - Tal Yarkoni
- Department of Psychology, The University of Texas at Austin, Austin, United States
| |
Collapse
|
4
|
Nastase SA, Liu YF, Hillman H, Zadbood A, Hasenfratz L, Keshavarzian N, Chen J, Honey CJ, Yeshurun Y, Regev M, Nguyen M, Chang CHC, Baldassano C, Lositsky O, Simony E, Chow MA, Leong YC, Brooks PP, Micciche E, Choe G, Goldstein A, Vanderwal T, Halchenko YO, Norman KA, Hasson U. The "Narratives" fMRI dataset for evaluating models of naturalistic language comprehension. Sci Data 2021; 8:250. [PMID: 34584100 PMCID: PMC8479122 DOI: 10.1038/s41597-021-01033-3] [Citation(s) in RCA: 23] [Impact Index Per Article: 7.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/04/2021] [Accepted: 08/18/2021] [Indexed: 02/08/2023] Open
Abstract
The "Narratives" collection aggregates a variety of functional MRI datasets collected while human subjects listened to naturalistic spoken stories. The current release includes 345 subjects, 891 functional scans, and 27 diverse stories of varying duration totaling ~4.6 hours of unique stimuli (~43,000 words). This data collection is well-suited for naturalistic neuroimaging analysis, and is intended to serve as a benchmark for models of language and narrative comprehension. We provide standardized MRI data accompanied by rich metadata, preprocessed versions of the data ready for immediate use, and the spoken story stimuli with time-stamped phoneme- and word-level transcripts. All code and data are publicly available with full provenance in keeping with current best practices in transparent and reproducible neuroimaging.
Collapse
Affiliation(s)
- Samuel A Nastase
- Princeton Neuroscience Institute and Department of Psychology, Princeton University, Princeton, NJ, USA.
| | - Yun-Fei Liu
- Department of Psychological and Brain Sciences, Johns Hopkins University, Baltimore, MD, USA
| | - Hanna Hillman
- Princeton Neuroscience Institute and Department of Psychology, Princeton University, Princeton, NJ, USA
| | - Asieh Zadbood
- Princeton Neuroscience Institute and Department of Psychology, Princeton University, Princeton, NJ, USA
| | - Liat Hasenfratz
- Princeton Neuroscience Institute and Department of Psychology, Princeton University, Princeton, NJ, USA
| | - Neggin Keshavarzian
- Princeton Neuroscience Institute and Department of Psychology, Princeton University, Princeton, NJ, USA
| | - Janice Chen
- Department of Psychological and Brain Sciences, Johns Hopkins University, Baltimore, MD, USA
| | - Christopher J Honey
- Department of Psychological and Brain Sciences, Johns Hopkins University, Baltimore, MD, USA
| | - Yaara Yeshurun
- School of Psychological Sciences, Tel Aviv University, Tel Aviv, Israel
| | - Mor Regev
- Montreal Neurological Institute, McGill University, Montreal, QC, Canada
| | - Mai Nguyen
- Princeton Neuroscience Institute and Department of Psychology, Princeton University, Princeton, NJ, USA
| | - Claire H C Chang
- Princeton Neuroscience Institute and Department of Psychology, Princeton University, Princeton, NJ, USA
| | | | - Olga Lositsky
- Department of Cognitive, Linguistic and Psychological Sciences, Brown University, Providence, RI, USA
| | - Erez Simony
- Faculty of Electrical Engineering, Holon Institute of Technology, Holon, Israel
- Department of Neurobiology, Weizmann Institute of Science, Rehovot, Israel
| | | | - Yuan Chang Leong
- Helen Wills Neuroscience Institute, University of California, Berkeley, CA, USA
| | - Paula P Brooks
- Princeton Neuroscience Institute and Department of Psychology, Princeton University, Princeton, NJ, USA
| | - Emily Micciche
- Peabody College, Vanderbilt University, Nashville, TN, USA
| | - Gina Choe
- Princeton Neuroscience Institute and Department of Psychology, Princeton University, Princeton, NJ, USA
| | - Ariel Goldstein
- Princeton Neuroscience Institute and Department of Psychology, Princeton University, Princeton, NJ, USA
| | - Tamara Vanderwal
- Department of Psychiatry, University of British Columbia, and BC Children's Hospital Research Institute, Vancouver, BC, Canada
| | - Yaroslav O Halchenko
- Department of Psychological and Brain Sciences and Department of Computer Science, Dartmouth College, Hanover, NH, USA
| | - Kenneth A Norman
- Princeton Neuroscience Institute and Department of Psychology, Princeton University, Princeton, NJ, USA
| | - Uri Hasson
- Princeton Neuroscience Institute and Department of Psychology, Princeton University, Princeton, NJ, USA
| |
Collapse
|
5
|
Visconti di Oleggio Castello M, Chauhan V, Jiahui G, Gobbini MI. An fMRI dataset in response to "The Grand Budapest Hotel", a socially-rich, naturalistic movie. Sci Data 2020; 7:383. [PMID: 33177526 PMCID: PMC7658985 DOI: 10.1038/s41597-020-00735-4] [Citation(s) in RCA: 17] [Impact Index Per Article: 4.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/28/2020] [Accepted: 10/27/2020] [Indexed: 11/18/2022] Open
Abstract
Naturalistic stimuli evoke strong, consistent, and information-rich patterns of brain activity, and engage large extents of the human brain. They allow researchers to compare highly similar brain responses across subjects, and to study how complex representations are encoded in brain activity. Here, we describe and share a dataset where 25 subjects watched part of the feature film "The Grand Budapest Hotel" by Wes Anderson. The movie has a large cast with many famous actors. Throughout the story, the camera shots highlight faces and expressions, which are fundamental to understand the complex narrative of the movie. This movie was chosen to sample brain activity specifically related to social interactions and face processing. This dataset provides researchers with fMRI data that can be used to explore social cognitive processes and face processing, adding to the existing neuroimaging datasets that sample brain activity with naturalistic movies.
Collapse
Affiliation(s)
| | - Vassiki Chauhan
- Center for Cognitive Neuroscience, Dartmouth College, Hanover, USA
| | - Guo Jiahui
- Center for Cognitive Neuroscience, Dartmouth College, Hanover, USA
| | - M Ida Gobbini
- Cognitive Science Program, Dartmouth College, Hanover, USA.
- Dipartimento di Medicina Specialistica, Diagnostica e Sperimentale, University of Bologna, Bologna, Italy.
| |
Collapse
|
6
|
Aliko S, Huang J, Gheorghiu F, Meliss S, Skipper JI. A naturalistic neuroimaging database for understanding the brain using ecological stimuli. Sci Data 2020; 7:347. [PMID: 33051448 PMCID: PMC7555491 DOI: 10.1038/s41597-020-00680-2] [Citation(s) in RCA: 30] [Impact Index Per Article: 7.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/26/2020] [Accepted: 09/16/2020] [Indexed: 12/22/2022] Open
Abstract
Neuroimaging has advanced our understanding of human psychology using reductionist stimuli that often do not resemble information the brain naturally encounters. It has improved our understanding of the network organization of the brain mostly through analyses of 'resting-state' data for which the functions of networks cannot be verifiably labelled. We make a 'Naturalistic Neuroimaging Database' (NNDb v1.0) publically available to allow for a more complete understanding of the brain under more ecological conditions during which networks can be labelled. Eighty-six participants underwent behavioural testing and watched one of 10 full-length movies while functional magnetic resonance imaging was acquired. Resulting timeseries data are shown to be of high quality, with good signal-to-noise ratio, few outliers and low movement. Data-driven functional analyses provide further evidence of data quality. They also demonstrate accurate timeseries/movie alignment and how movie annotations might be used to label networks. The NNDb can be used to answer questions previously unaddressed with standard neuroimaging approaches, progressing our knowledge of how the brain works in the real world.
Collapse
Affiliation(s)
- Sarah Aliko
- London Interdisciplinary Biosciences Consortium, University College London, London, UK.
- Experimental Psychology, University College London, London, UK.
| | - Jiawen Huang
- Experimental Psychology, University College London, London, UK
| | | | - Stefanie Meliss
- Experimental Psychology, University College London, London, UK
- School of Psychology and Clinical Language Sciences, University of Reading, Reading, UK
| | | |
Collapse
|
7
|
DuPre E, Hanke M, Poline JB. Nature abhors a paywall: How open science can realize the potential of naturalistic stimuli. Neuroimage 2020; 216:116330. [PMID: 31704292 PMCID: PMC7198323 DOI: 10.1016/j.neuroimage.2019.116330] [Citation(s) in RCA: 13] [Impact Index Per Article: 3.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/03/2019] [Revised: 10/31/2019] [Accepted: 10/31/2019] [Indexed: 11/26/2022] Open
Abstract
Naturalistic stimuli show significant potential to inform behavioral, cognitive, and clinical neuroscience. To date, this impact is still limited by the relative inaccessibility of both generated neuroimaging data as well as the supporting naturalistic stimuli. In this perspective, we highlight currently available naturalistic datasets and technical solutions such as DataLad that continue to advance our ability to share this data. We also review scientific and sociological challenges in selecting naturalistic stimuli for reproducible research. Overall, we encourage researchers to share their naturalistic datasets to the full extent possible under local copyright law.
Collapse
Affiliation(s)
| | - Michael Hanke
- Institute of Neuroscience and Medicine, Brain & Behaviour (INM-7), Research Centre Jülich, Jülich, Germany; Institute of Systems Neuroscience, Medical Faculty, Heinrich Heine University Düsseldorf, Düsseldorf, Germany
| | | |
Collapse
|
8
|
Wen Z, Yu T, Yang X, Li Y. Goal-Directed Processing of Naturalistic Stimuli Modulates Large-Scale Functional Connectivity. Front Neurosci 2019; 12:1003. [PMID: 30760971 PMCID: PMC6361838 DOI: 10.3389/fnins.2018.01003] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.2] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/20/2018] [Accepted: 12/12/2018] [Indexed: 11/25/2022] Open
Abstract
Humans selectively process external information according to their internal goals. Previous studies have found that cortical activity and interactions between specific cortical areas such as frontal-parietal regions are modulated by behavioral goals. However, these results are largely based on simple stimuli and task rules in laboratory settings. Here, we investigated how top-down goals modulate whole-brain functional connectivity (FC) under naturalistic conditions. Analyses were conducted on a publicly available functional magnetic resonance imaging (fMRI) dataset (OpenfMRI database, accession number: ds000233) collected on twelve participants who made either behavioral or taxonomic judgments of behaving animals containing in naturalistic video clips. The task-evoked FC patterns of the participants were extracted using a novel inter-subject functional correlation (ISFC) method that increases the signal-to-noise ratio for detecting task-induced inter-regional correlation compared with standard FC analysis. Using multivariate pattern analysis (MVPA) methods, we successfully predicted the task goals of the participants with ISFC patterns but not with standard FC patterns, suggests that the ISFC method may be an efficient tool for exploring subtle network differences between brain states. We further examined the predictive power of several canonical brain networks and found that many within-network and across-network ISFC measures supported task goals classification. Our findings suggest that goal-directed processing of naturalistic stimuli systematically modulates large-scale brain networks but is not limited to the local neural activity or connectivity of specific regions.
Collapse
Affiliation(s)
- Zhenfu Wen
- Center for Brain Computer Interfaces and Brain Information Processing, South China University of Technology, Guangzhou, China.,Guangzhou Key Laboratory of Brain Computer Interaction and Application, Guangzhou, China
| | - Tianyou Yu
- Center for Brain Computer Interfaces and Brain Information Processing, South China University of Technology, Guangzhou, China.,Guangzhou Key Laboratory of Brain Computer Interaction and Application, Guangzhou, China
| | - Xinbin Yang
- Department of Surgical Thoracic Oncology, Affiliated Cancer Hospital and Institute of Guangzhou Medical University, Guangzhou, China
| | - Yuanqing Li
- Center for Brain Computer Interfaces and Brain Information Processing, South China University of Technology, Guangzhou, China.,Guangzhou Key Laboratory of Brain Computer Interaction and Application, Guangzhou, China
| |
Collapse
|