1
|
Pooja R, Ghosh P, Sreekumar V. Towards an ecologically valid naturalistic cognitive neuroscience of memory and event cognition. Neuropsychologia 2024; 203:108970. [PMID: 39147361 DOI: 10.1016/j.neuropsychologia.2024.108970] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/31/2023] [Revised: 07/31/2024] [Accepted: 08/08/2024] [Indexed: 08/17/2024]
Abstract
The landscape of human memory and event cognition research has witnessed a transformative journey toward the use of naturalistic contexts and tasks. In this review, we track this progression from abrupt, artificial stimuli used in extensively controlled laboratory experiments to more naturalistic tasks and stimuli that present a more faithful representation of the real world. We argue that in order to improve ecological validity, naturalistic study designs must consider the complexity of the cognitive phenomenon being studied. Then, we review the current state of "naturalistic" event segmentation studies and critically assess frequently employed movie stimuli. We evaluate recently developed tools like lifelogging and other extended reality technologies to help address the challenges we identified with existing naturalistic approaches. We conclude by offering some guidelines that can be used to design ecologically valid cognitive neuroscience studies of memory and event cognition.
Collapse
Affiliation(s)
- Raju Pooja
- Cognitive Science Lab, International Institute of Information Technology, Hyderabad, India
| | - Pritha Ghosh
- Cognitive Science Lab, International Institute of Information Technology, Hyderabad, India
| | - Vishnu Sreekumar
- Cognitive Science Lab, International Institute of Information Technology, Hyderabad, India.
| |
Collapse
|
2
|
Feilong M, Jiahui G, Gobbini MI, Haxby JV. A cortical surface template for human neuroscience. Nat Methods 2024:10.1038/s41592-024-02346-y. [PMID: 39014074 DOI: 10.1038/s41592-024-02346-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/12/2023] [Accepted: 06/06/2024] [Indexed: 07/18/2024]
Abstract
Neuroimaging data analysis relies on normalization to standard anatomical templates to resolve macroanatomical differences across brains. Existing human cortical surface templates sample locations unevenly because of distortions introduced by inflation of the folded cortex into a standard shape. Here we present the onavg template, which affords uniform sampling of the cortex. We created the onavg template based on openly available high-quality structural scans of 1,031 brains-25 times more than existing cortical templates. We optimized the vertex locations based on cortical anatomy, achieving an even distribution. We observed consistently higher multivariate pattern classification accuracies and representational geometry inter-participant correlations based on onavg than on other templates, and onavg only needs three-quarters as much data to achieve the same performance compared with other templates. The optimized sampling also reduces CPU time across algorithms by 1.3-22.4% due to less variation in the number of vertices in each searchlight.
Collapse
Affiliation(s)
- Ma Feilong
- Center for Cognitive Neuroscience, Dartmouth College, Hanover, NH, USA.
| | - Guo Jiahui
- Center for Cognitive Neuroscience, Dartmouth College, Hanover, NH, USA
- School of Behavioral and Brain Sciences, University of Texas at Dallas, Richardson, TX, USA
| | - Maria Ida Gobbini
- Department of Medical and Surgical Sciences, University of Bologna, Bologna, Italy
- IRCCS Istituto delle Scienze Neurologiche di Bologna, Bologna, Italy
| | - James V Haxby
- Center for Cognitive Neuroscience, Dartmouth College, Hanover, NH, USA.
| |
Collapse
|
3
|
Virk T, Letendre T, Pathman T. The convergence of naturalistic paradigms and cognitive neuroscience methods to investigate memory and its development. Neuropsychologia 2024; 196:108779. [PMID: 38154592 DOI: 10.1016/j.neuropsychologia.2023.108779] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/14/2023] [Revised: 12/12/2023] [Accepted: 12/23/2023] [Indexed: 12/30/2023]
Abstract
Studies that involve lab-based stimuli (e.g., words, pictures) are fundamental in the memory literature. At the same time, there is growing acknowledgment that memory processes assessed in the lab may not be analogous to how memory operates in the real world. Naturalistic paradigms can bridge this gap and over the decades a growing proportion of memory research has involved more naturalistic events. However, there is significant variation in the types of naturalistic studies used to study memory and its development, each with various advantages and limitations. Further, there are notable gaps in how often different types of naturalistic approaches have been combined with cognitive neuroscience methods (e.g., fMRI, EEG) to elucidate the neural processes and substrates involved in memory encoding and retrieval in the real world. Here we summarize and discuss what we identify as progressively more naturalistic methodologies used in the memory literature (movie, virtual reality, staged-events inside and outside of the lab, photo-taking, and naturally occurring event studies). Our goal is to describe each approach's benefits (e.g., naturalistic quality, feasibility), limitations (e.g., viability of neuroimaging method for event encoding versus event retrieval), and discuss possible future directions with each approach. We focus on child studies, when available, but also highlight past adult studies. Although there is a growing body of child memory research, naturalistic approaches combined with cognitive neuroscience methodologies in this domain remain sparse. Overall, this viewpoint article reviews how we can study memory through the lens of developmental cognitive neuroscience, while utilizing naturalistic and real-world events.
Collapse
|
4
|
Keles U, Dubois J, Le KJM, Tyszka JM, Kahn DA, Reed CM, Chung JM, Mamelak AN, Adolphs R, Rutishauser U. Multimodal single-neuron, intracranial EEG, and fMRI brain responses during movie watching in human patients. Sci Data 2024; 11:214. [PMID: 38365977 PMCID: PMC10873379 DOI: 10.1038/s41597-024-03029-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/27/2023] [Accepted: 01/31/2024] [Indexed: 02/18/2024] Open
Abstract
We present a multimodal dataset of intracranial recordings, fMRI, and eye tracking in 20 participants during movie watching. Recordings consist of single neurons, local field potential, and intracranial EEG activity acquired from depth electrodes targeting the amygdala, hippocampus, and medial frontal cortex implanted for monitoring of epileptic seizures. Participants watched an 8-min long excerpt from the video "Bang! You're Dead" and performed a recognition memory test for movie content. 3 T fMRI activity was recorded prior to surgery in 11 of these participants while performing the same task. This NWB- and BIDS-formatted dataset includes spike times, field potential activity, behavior, eye tracking, electrode locations, demographics, and functional and structural MRI scans. For technical validation, we provide signal quality metrics, assess eye tracking quality, behavior, the tuning of cells and high-frequency broadband power field potentials to familiarity and event boundaries, and show brain-wide inter-subject correlations for fMRI. This dataset will facilitate the investigation of brain activity during movie watching, recognition memory, and the neural basis of the fMRI-BOLD signal.
Collapse
Affiliation(s)
- Umit Keles
- Department of Neurosurgery, Cedars-Sinai Medical Center, Los Angeles, CA, USA
- Division of the Humanities and Social Sciences, California Institute of Technology, Pasadena, CA, USA
| | - Julien Dubois
- Department of Neurosurgery, Cedars-Sinai Medical Center, Los Angeles, CA, USA
| | - Kevin J M Le
- Division of Biology and Biological Engineering, California Institute of Technology, Pasadena, CA, USA
| | - J Michael Tyszka
- Division of the Humanities and Social Sciences, California Institute of Technology, Pasadena, CA, USA
| | - David A Kahn
- Division of the Humanities and Social Sciences, California Institute of Technology, Pasadena, CA, USA
| | - Chrystal M Reed
- Department of Neurology, Cedars-Sinai Medical Center, Los Angeles, CA, USA
| | - Jeffrey M Chung
- Department of Neurology, Cedars-Sinai Medical Center, Los Angeles, CA, USA
| | - Adam N Mamelak
- Department of Neurosurgery, Cedars-Sinai Medical Center, Los Angeles, CA, USA
| | - Ralph Adolphs
- Division of the Humanities and Social Sciences, California Institute of Technology, Pasadena, CA, USA.
- Division of Biology and Biological Engineering, California Institute of Technology, Pasadena, CA, USA.
| | - Ueli Rutishauser
- Department of Neurosurgery, Cedars-Sinai Medical Center, Los Angeles, CA, USA.
- Division of Biology and Biological Engineering, California Institute of Technology, Pasadena, CA, USA.
- Department of Neurology, Cedars-Sinai Medical Center, Los Angeles, CA, USA.
- Center for Neural Science and Medicine, Department of Biomedical Sciences, Cedars-Sinai Medical Center, Los Angeles, CA, USA.
| |
Collapse
|
5
|
Jiahui G, Feilong M, Nastase SA, Haxby JV, Gobbini MI. Cross-movie prediction of individualized functional topography. eLife 2023; 12:e86037. [PMID: 37994909 PMCID: PMC10666932 DOI: 10.7554/elife.86037] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/09/2023] [Accepted: 11/09/2023] [Indexed: 11/24/2023] Open
Abstract
Participant-specific, functionally defined brain areas are usually mapped with functional localizers and estimated by making contrasts between responses to single categories of input. Naturalistic stimuli engage multiple brain systems in parallel, provide more ecologically plausible estimates of real-world statistics, and are friendly to special populations. The current study shows that cortical functional topographies in individual participants can be estimated with high fidelity from naturalistic stimuli. Importantly, we demonstrate that robust, individualized estimates can be obtained even when participants watched different movies, were scanned with different parameters/scanners, and were sampled from different institutes across the world. Our results create a foundation for future studies that allow researchers to estimate a broad range of functional topographies based on naturalistic movies and a normative database, making it possible to integrate high-level cognitive functions across datasets from laboratories worldwide.
Collapse
Affiliation(s)
- Guo Jiahui
- Center for Cognitive Neuroscience, Dartmouth CollegeHanoverUnited States
| | - Ma Feilong
- Center for Cognitive Neuroscience, Dartmouth CollegeHanoverUnited States
| | - Samuel A Nastase
- Princeton Neuroscience Institute, Princeton UniversityPrincetonUnited States
| | - James V Haxby
- Center for Cognitive Neuroscience, Dartmouth CollegeHanoverUnited States
| | - M Ida Gobbini
- Department of Medical and Surgical Sciences (DIMEC), University of BolognaBolognaItaly
- IRCCS, Istituto delle Scienze Neurologiche di BolognaBolognaItaly
| |
Collapse
|
6
|
Shimon-Raz O, Yeshurun Y, Ulmer-Yaniv A, Levinkron A, Salomon R, Feldman R. Attachment Reminders Trigger Widespread Synchrony across Multiple Brains. J Neurosci 2023; 43:7213-7225. [PMID: 37813569 PMCID: PMC10601370 DOI: 10.1523/jneurosci.0026-23.2023] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/05/2023] [Revised: 08/15/2023] [Accepted: 09/04/2023] [Indexed: 10/17/2023] Open
Abstract
Infant stimuli elicit widespread neural and behavioral response in human adults, and such massive allocation of resources attests to the evolutionary significance of the primary attachment. Here, we examined whether attachment reminders also trigger cross-brain concordance and generate greater neural uniformity, as indicated by intersubject correlation. Human mothers were imaged twice in oxytocin/placebo administration design, and stimuli included four ecological videos of a standard unfamiliar mother and infant: two infant/mother alone (Alone) and two mother-infant dyadic contexts (Social). Theory-driven analysis measured cross-brain synchrony in preregistered nodes of the parental caregiving network (PCN), which integrates subcortical structures underpinning mammalian mothering with cortical areas implicated in simulation, mentalization, and emotion regulation, and data-driven analysis assessed brain-wide concordance using whole-brain parcellation. Results demonstrated widespread cross-brain synchrony in both the PCN and across the neuroaxis, from primary sensory/somatosensory areas, through insular-cingulate regions, to temporal and prefrontal cortices. The Social context yielded significantly more cross-brain concordance, with PCNs striatum, parahippocampal gyrus, superior temporal sulcus, ACC, and PFC displaying cross-brain synchrony only to mother-infant social cues. Moment-by-moment fluctuations in mother-infant social synchrony, ranging from episodes of low synchrony to tightly coordinated positive bouts, were tracked online by cross-brain concordance in the preregistered ACC. Findings indicate that social attachment stimuli, representing evolutionary-salient universal cues that require no verbal narrative, trigger substantial interbrain concordance and suggest that the mother-infant bond, an icon standing at the heart of human civilization, may function to glue brains into a unified experience and bind humans into social groups.SIGNIFICANCE STATEMENT Infant stimuli elicit widespread neural response in human adults, attesting to their evolutionary significance, but do they also trigger cross-brain concordance and induce neural uniformity among perceivers? We measured cross-brain synchrony to ecological mother-infant videos. We used theory-driven analysis, measuring cross-brain concordance in the parenting network, and data-driven analysis, assessing brain-wide concordance using whole-brain parcellation. Attachment cues triggered widespread cross-brain concordance in both the parenting network and across the neuroaxis. Moment-by-moment fluctuations in behavioral synchrony were tracked online by cross-brain variability in ACC. Attachment reminders bind humans' brains into a unitary experience and stimuli characterized by social synchrony enhance neural similarity among participants, describing one mechanism by which attachment bonds provide the neural template for the consolidation of social groups.
Collapse
Affiliation(s)
| | - Yaara Yeshurun
- School of Psychological Sciences, Tel Aviv University, Tel Aviv, 69978, Israel
- Sagol School of Neuroscience, Tel Aviv University, Tel Aviv, 69978, Israel
| | | | - Ayelet Levinkron
- Sagol School of Neuroscience, Tel Aviv University, Tel Aviv, 69978, Israel
| | - Roy Salomon
- Department of Cognitive Sciences, University of Haifa, Haifa, 3498838, Israel
| | | |
Collapse
|
7
|
Jiahui G, Feilong M, Visconti di Oleggio Castello M, Nastase SA, Haxby JV, Gobbini MI. Modeling naturalistic face processing in humans with deep convolutional neural networks. Proc Natl Acad Sci U S A 2023; 120:e2304085120. [PMID: 37847731 PMCID: PMC10614847 DOI: 10.1073/pnas.2304085120] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/16/2023] [Accepted: 09/11/2023] [Indexed: 10/19/2023] Open
Abstract
Deep convolutional neural networks (DCNNs) trained for face identification can rival and even exceed human-level performance. The ways in which the internal face representations in DCNNs relate to human cognitive representations and brain activity are not well understood. Nearly all previous studies focused on static face image processing with rapid display times and ignored the processing of naturalistic, dynamic information. To address this gap, we developed the largest naturalistic dynamic face stimulus set in human neuroimaging research (700+ naturalistic video clips of unfamiliar faces). We used this naturalistic dataset to compare representational geometries estimated from DCNNs, behavioral responses, and brain responses. We found that DCNN representational geometries were consistent across architectures, cognitive representational geometries were consistent across raters in a behavioral arrangement task, and neural representational geometries in face areas were consistent across brains. Representational geometries in late, fully connected DCNN layers, which are optimized for individuation, were much more weakly correlated with cognitive and neural geometries than were geometries in late-intermediate layers. The late-intermediate face-DCNN layers successfully matched cognitive representational geometries, as measured with a behavioral arrangement task that primarily reflected categorical attributes, and correlated with neural representational geometries in known face-selective topographies. Our study suggests that current DCNNs successfully capture neural cognitive processes for categorical attributes of faces but less accurately capture individuation and dynamic features.
Collapse
Affiliation(s)
- Guo Jiahui
- Center for Cognitive Neuroscience, Dartmouth College, Hanover, NH03755
| | - Ma Feilong
- Center for Cognitive Neuroscience, Dartmouth College, Hanover, NH03755
| | | | - Samuel A. Nastase
- Princeton Neuroscience Institute, Princeton University, Princeton, NJ08544
| | - James V. Haxby
- Center for Cognitive Neuroscience, Dartmouth College, Hanover, NH03755
| | - M. Ida Gobbini
- Department of Medical and Surgical Sciences, University of Bologna, Bologna40138, Italy
- Istituti di Ricovero e Cura a Carattere Scientifico, Istituto delle Scienze Neurologiche di Bologna, Bologna40139, Italia
| |
Collapse
|
8
|
Gong Z, Zhou M, Dai Y, Wen Y, Liu Y, Zhen Z. A large-scale fMRI dataset for the visual processing of naturalistic scenes. Sci Data 2023; 10:559. [PMID: 37612327 PMCID: PMC10447576 DOI: 10.1038/s41597-023-02471-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/27/2023] [Accepted: 08/14/2023] [Indexed: 08/25/2023] Open
Abstract
One ultimate goal of visual neuroscience is to understand how the brain processes visual stimuli encountered in the natural environment. Achieving this goal requires records of brain responses under massive amounts of naturalistic stimuli. Although the scientific community has put a lot of effort into collecting large-scale functional magnetic resonance imaging (fMRI) data under naturalistic stimuli, more naturalistic fMRI datasets are still urgently needed. We present here the Natural Object Dataset (NOD), a large-scale fMRI dataset containing responses to 57,120 naturalistic images from 30 participants. NOD strives for a balance between sampling variation between individuals and sampling variation between stimuli. This enables NOD to be utilized not only for determining whether an observation is generalizable across many individuals, but also for testing whether a response pattern is generalized to a variety of naturalistic stimuli. We anticipate that the NOD together with existing naturalistic neuroimaging datasets will serve as a new impetus for our understanding of the visual processing of naturalistic stimuli.
Collapse
Affiliation(s)
- Zhengxin Gong
- Beijing Key Laboratory of Applied Experimental Psychology, Faculty of Psychology, Beijing Normal University, Beijing, 100875, China
| | - Ming Zhou
- State Key Laboratory of Cognitive Neuroscience and Learning & IDG/McGovern Institute for Brain Research, Beijing Normal University, Beijing, 100875, China
| | - Yuxuan Dai
- Beijing Key Laboratory of Applied Experimental Psychology, Faculty of Psychology, Beijing Normal University, Beijing, 100875, China
| | - Yushan Wen
- Beijing Key Laboratory of Applied Experimental Psychology, Faculty of Psychology, Beijing Normal University, Beijing, 100875, China
| | - Youyi Liu
- State Key Laboratory of Cognitive Neuroscience and Learning & IDG/McGovern Institute for Brain Research, Beijing Normal University, Beijing, 100875, China.
| | - Zonglei Zhen
- Beijing Key Laboratory of Applied Experimental Psychology, Faculty of Psychology, Beijing Normal University, Beijing, 100875, China.
- State Key Laboratory of Cognitive Neuroscience and Learning & IDG/McGovern Institute for Brain Research, Beijing Normal University, Beijing, 100875, China.
| |
Collapse
|
9
|
LeBel A, Wagner L, Jain S, Adhikari-Desai A, Gupta B, Morgenthal A, Tang J, Xu L, Huth AG. A natural language fMRI dataset for voxelwise encoding models. Sci Data 2023; 10:555. [PMID: 37612332 PMCID: PMC10447563 DOI: 10.1038/s41597-023-02437-z] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/26/2022] [Accepted: 08/02/2023] [Indexed: 08/25/2023] Open
Abstract
Speech comprehension is a complex process that draws on humans' abilities to extract lexical information, parse syntax, and form semantic understanding. These sub-processes have traditionally been studied using separate neuroimaging experiments that attempt to isolate specific effects of interest. More recently it has become possible to study all stages of language comprehension in a single neuroimaging experiment using narrative natural language stimuli. The resulting data are richly varied at every level, enabling analyses that can probe everything from spectral representations to high-level representations of semantic meaning. We provide a dataset containing BOLD fMRI responses recorded while 8 participants each listened to 27 complete, natural, narrative stories (~6 hours). This dataset includes pre-processed and raw MRIs, as well as hand-constructed 3D cortical surfaces for each participant. To address the challenges of analyzing naturalistic data, this dataset is accompanied by a python library containing basic code for creating voxelwise encoding models. Altogether, this dataset provides a large and novel resource for understanding speech and language processing in the human brain.
Collapse
Affiliation(s)
- Amanda LeBel
- Helen Wills Neuroscience Institute, University of California, Berkeley, Berkeley, CA, 94704, USA
| | - Lauren Wagner
- Department of Psychiatry and Biobehavioral Sciences, University of California, Los Angeles, CA, 90095, USA
| | - Shailee Jain
- Department of Computer Science, The University of Texas at Austin, Austin, TX, 78712, USA
| | - Aneesh Adhikari-Desai
- Department of Computer Science, The University of Texas at Austin, Austin, TX, 78712, USA
- Department of Neuroscience, The University of Texas at Austin, Austin, TX, 78712, USA
| | - Bhavin Gupta
- Department of Computer Science, The University of Texas at Austin, Austin, TX, 78712, USA
| | - Allyson Morgenthal
- Department of Neuroscience, The University of Texas at Austin, Austin, TX, 78712, USA
| | - Jerry Tang
- Department of Computer Science, The University of Texas at Austin, Austin, TX, 78712, USA
| | - Lixiang Xu
- Department of Physics, The University of Texas at Austin, Austin, TX, 78712, USA
| | - Alexander G Huth
- Department of Computer Science, The University of Texas at Austin, Austin, TX, 78712, USA.
- Department of Neuroscience, The University of Texas at Austin, Austin, TX, 78712, USA.
| |
Collapse
|
10
|
Zhou M, Gong Z, Dai Y, Wen Y, Liu Y, Zhen Z. A large-scale fMRI dataset for human action recognition. Sci Data 2023; 10:415. [PMID: 37369643 DOI: 10.1038/s41597-023-02325-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/13/2023] [Accepted: 06/21/2023] [Indexed: 06/29/2023] Open
Abstract
Human action recognition is a critical capability for our survival, allowing us to interact easily with the environment and others in everyday life. Although the neural basis of action recognition has been widely studied using a few action categories from simple contexts as stimuli, how the human brain recognizes diverse human actions in real-world environments still needs to be explored. Here, we present the Human Action Dataset (HAD), a large-scale functional magnetic resonance imaging (fMRI) dataset for human action recognition. HAD contains fMRI responses to 21,600 video clips from 30 participants. The video clips encompass 180 human action categories and offer a comprehensive coverage of complex activities in daily life. We demonstrate that the data are reliable within and across participants and, notably, capture rich representation information of the observed human actions. This extensive dataset, with its vast number of action categories and exemplars, has the potential to deepen our understanding of human action recognition in natural environments.
Collapse
Affiliation(s)
- Ming Zhou
- State Key Laboratory of Cognitive Neuroscience and Learning & IDG/McGovern Institute for Brain Research, Beijing Normal University, Beijing, 100875, China
| | - Zhengxin Gong
- Beijing Key Laboratory of Applied Experimental Psychology, Faculty of Psychology, Beijing Normal University, Beijing, 100875, China
| | - Yuxuan Dai
- Beijing Key Laboratory of Applied Experimental Psychology, Faculty of Psychology, Beijing Normal University, Beijing, 100875, China
| | - Yushan Wen
- Beijing Key Laboratory of Applied Experimental Psychology, Faculty of Psychology, Beijing Normal University, Beijing, 100875, China
| | - Youyi Liu
- State Key Laboratory of Cognitive Neuroscience and Learning & IDG/McGovern Institute for Brain Research, Beijing Normal University, Beijing, 100875, China
| | - Zonglei Zhen
- Beijing Key Laboratory of Applied Experimental Psychology, Faculty of Psychology, Beijing Normal University, Beijing, 100875, China.
| |
Collapse
|
11
|
Wang S, Zhang X, Zhang J, Zong C. A synchronized multimodal neuroimaging dataset for studying brain language processing. Sci Data 2022; 9:590. [PMID: 36180444 PMCID: PMC9525723 DOI: 10.1038/s41597-022-01708-5] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/15/2022] [Accepted: 08/22/2022] [Indexed: 11/15/2022] Open
Abstract
We present a synchronized multimodal neuroimaging dataset for studying brain language processing (SMN4Lang) that contains functional magnetic resonance imaging (fMRI) and magnetoencephalography (MEG) data on the same 12 healthy volunteers while the volunteers listened to 6 hours of naturalistic stories, as well as high-resolution structural (T1, T2), diffusion MRI and resting-state fMRI data for each participant. We also provide rich linguistic annotations for the stimuli, including word frequencies, syntactic tree structures, time-aligned characters and words, and various types of word and character embeddings. Quality assessment indicators verify that this is a high-quality neuroimaging dataset. Such synchronized data is separately collected by the same group of participants first listening to story materials in fMRI and then in MEG which are well suited to studying the dynamic processing of language comprehension, such as the time and location of different linguistic features encoded in the brain. In addition, this dataset, comprising a large vocabulary from stories with various topics, can serve as a brain benchmark to evaluate and improve computational language models. Measurement(s) | functional brain measurement • Magnetoencephalography | Technology Type(s) | Functional Magnetic Resonance Imaging • Magnetoencephalography | Factor Type(s) | naturalistic stimuli listening | Sample Characteristic - Organism | humanbeings |
Collapse
Affiliation(s)
- Shaonan Wang
- National Laboratory of Pattern Recognition, Institute of Automation, CAS, Beijing, China. .,School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China.
| | - Xiaohan Zhang
- National Laboratory of Pattern Recognition, Institute of Automation, CAS, Beijing, China.,School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China
| | - Jiajun Zhang
- National Laboratory of Pattern Recognition, Institute of Automation, CAS, Beijing, China.,School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China
| | - Chengqing Zong
- National Laboratory of Pattern Recognition, Institute of Automation, CAS, Beijing, China.,School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China
| |
Collapse
|
12
|
de la Vega A, Rocca R, Blair RW, Markiewicz CJ, Mentch J, Kent JD, Herholz P, Ghosh SS, Poldrack RA, Yarkoni T. Neuroscout, a unified platform for generalizable and reproducible fMRI research. eLife 2022; 11:79277. [PMID: 36040302 PMCID: PMC9489206 DOI: 10.7554/elife.79277] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/06/2022] [Accepted: 08/27/2022] [Indexed: 11/28/2022] Open
Abstract
Functional magnetic resonance imaging (fMRI) has revolutionized cognitive neuroscience, but methodological barriers limit the generalizability of findings from the lab to the real world. Here, we present Neuroscout, an end-to-end platform for analysis of naturalistic fMRI data designed to facilitate the adoption of robust and generalizable research practices. Neuroscout leverages state-of-the-art machine learning models to automatically annotate stimuli from dozens of fMRI studies using naturalistic stimuli—such as movies and narratives—allowing researchers to easily test neuroscientific hypotheses across multiple ecologically-valid datasets. In addition, Neuroscout builds on a robust ecosystem of open tools and standards to provide an easy-to-use analysis builder and a fully automated execution engine that reduce the burden of reproducible research. Through a series of meta-analytic case studies, we validate the automatic feature extraction approach and demonstrate its potential to support more robust fMRI research. Owing to its ease of use and a high degree of automation, Neuroscout makes it possible to overcome modeling challenges commonly arising in naturalistic analysis and to easily scale analyses within and across datasets, democratizing generalizable fMRI research.
Collapse
Affiliation(s)
- Alejandro de la Vega
- Department of Psychology, The University of Texas at Austin, Austin, United States
| | - Roberta Rocca
- Interacting Minds Centre, Aarhus University, Aarhus, Denmark
| | - Ross W Blair
- Department of Psychology, Stanford University, Stanford, United States
| | | | - Jeff Mentch
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, United States
| | - James D Kent
- Department of Psychology, The University of Texas at Austin, Austin, United States
| | - Peer Herholz
- Montreal Neurological Institute, McGill University, Montreal, Canada
| | - Satrajit S Ghosh
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, United States
| | | | - Tal Yarkoni
- Department of Psychology, The University of Texas at Austin, Austin, United States
| |
Collapse
|
13
|
A studyforrest extension, MEG recordings while watching the audio-visual movie "Forrest Gump". Sci Data 2022; 9:206. [PMID: 35562378 PMCID: PMC9106652 DOI: 10.1038/s41597-022-01299-1] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/02/2021] [Accepted: 03/30/2022] [Indexed: 01/01/2023] Open
Abstract
Naturalistic stimuli, such as movies, are being increasingly used to map brain function because of their high ecological validity. The pioneering studyforrest and other naturalistic neuroimaging projects have provided free access to multiple movie-watching functional magnetic resonance imaging (fMRI) datasets to prompt the community for naturalistic experimental paradigms. However, sluggish blood-oxygenation-level-dependent fMRI signals are incapable of resolving neuronal activity with the temporal resolution at which it unfolds. Instead, magnetoencephalography (MEG) measures changes in the magnetic field produced by neuronal activity and is able to capture rich dynamics of the brain at the millisecond level while watching naturalistic movies. Herein, we present the first public prolonged MEG dataset collected from 11 participants while watching the 2 h long audio-visual movie “Forrest Gump”. Minimally preprocessed data was also provided to facilitate the use of the dataset. As a studyforrest extension, we envision that this dataset, together with fMRI data from the studyforrest project, will serve as a foundation for exploring the neural dynamics of various cognitive functions in real-world contexts. Measurement(s) | Brain activity measurement • Brain structure | Technology Type(s) | Magnetoencephalography • Magnetic Resonance Imaging | Factor Type(s) | Audiovisual movie | Sample Characteristic - Organism | Homo sapiens |
Collapse
|
14
|
The Dual Mechanisms of Cognitive Control dataset, a theoretically-guided within-subject task fMRI battery. Sci Data 2022; 9:114. [PMID: 35351911 PMCID: PMC8964804 DOI: 10.1038/s41597-022-01226-4] [Citation(s) in RCA: 6] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/20/2021] [Accepted: 02/17/2022] [Indexed: 12/30/2022] Open
Abstract
Cognitive control is a critical higher mental function, which is subject to considerable individual variation, and is impaired in a range of mental health disorders. We describe here the initial release of Dual Mechanisms of Cognitive Control (DMCC) project data, the DMCC55B dataset, with 55 healthy unrelated young adult participants. Each participant performed four well-established cognitive control tasks (AX-CPT, Cued Task-Switching, Sternberg Working Memory, and Stroop) while undergoing functional MRI scanning. The dataset includes a range of state and trait self-report questionnaires, as well as behavioural tasks assessing individual differences in cognitive ability. The DMCC project is on-going and features additional components (e.g., related participants, manipulations of cognitive control mode, resting state fMRI, longitudinal testing) that will be publicly released following study completion. This DMCC55B subset is released early with the aim of encouraging wider use and greater benefit to the scientific community. The DMCC55B dataset is suitable for benchmarking and methods exploration, as well as analyses of task performance and individual differences.
Collapse
|
15
|
Shared neural codes for visual and semantic information about familiar faces in a common representational space. Proc Natl Acad Sci U S A 2021; 118:2110474118. [PMID: 34732577 PMCID: PMC8609335 DOI: 10.1073/pnas.2110474118] [Citation(s) in RCA: 9] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 09/23/2021] [Indexed: 11/18/2022] Open
Abstract
Our brain processes faces of close others differently than faces of visually familiar individuals. While both types of faces activate similar visual areas, faces of close others activate areas involved in processing social and semantic information. Here, we used between-subject linear classifiers trained on hyperaligned brain data to investigate the neural code for visual and semantic information about familiar others. The identity of both visually and personally familiar faces could be decoded across participants from brain activity in visual areas. Instead, only the identity of personally familiar faces could be decoded in areas involved in social cognition. Our results suggest that individually distinctive information associated with familiar faces is embedded in a neural code that is shared across brains. Processes evoked by seeing a personally familiar face encompass recognition of visual appearance and activation of social and person knowledge. Whereas visual appearance is the same for all viewers, social and person knowledge may be more idiosyncratic. Using between-subject multivariate decoding of hyperaligned functional magnetic resonance imaging data, we investigated whether representations of personally familiar faces in different parts of the distributed neural system for face perception are shared across individuals who know the same people. We found that the identities of both personally familiar and merely visually familiar faces were decoded accurately across brains in the core system for visual processing, but only the identities of personally familiar faces could be decoded across brains in the extended system for processing nonvisual information associated with faces. Our results show that personal interactions with the same individuals lead to shared neural representations of both the seen and unseen features that distinguish their identities.
Collapse
|
16
|
Nastase SA, Liu YF, Hillman H, Zadbood A, Hasenfratz L, Keshavarzian N, Chen J, Honey CJ, Yeshurun Y, Regev M, Nguyen M, Chang CHC, Baldassano C, Lositsky O, Simony E, Chow MA, Leong YC, Brooks PP, Micciche E, Choe G, Goldstein A, Vanderwal T, Halchenko YO, Norman KA, Hasson U. The "Narratives" fMRI dataset for evaluating models of naturalistic language comprehension. Sci Data 2021; 8:250. [PMID: 34584100 PMCID: PMC8479122 DOI: 10.1038/s41597-021-01033-3] [Citation(s) in RCA: 23] [Impact Index Per Article: 7.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/04/2021] [Accepted: 08/18/2021] [Indexed: 02/08/2023] Open
Abstract
The "Narratives" collection aggregates a variety of functional MRI datasets collected while human subjects listened to naturalistic spoken stories. The current release includes 345 subjects, 891 functional scans, and 27 diverse stories of varying duration totaling ~4.6 hours of unique stimuli (~43,000 words). This data collection is well-suited for naturalistic neuroimaging analysis, and is intended to serve as a benchmark for models of language and narrative comprehension. We provide standardized MRI data accompanied by rich metadata, preprocessed versions of the data ready for immediate use, and the spoken story stimuli with time-stamped phoneme- and word-level transcripts. All code and data are publicly available with full provenance in keeping with current best practices in transparent and reproducible neuroimaging.
Collapse
Affiliation(s)
- Samuel A Nastase
- Princeton Neuroscience Institute and Department of Psychology, Princeton University, Princeton, NJ, USA.
| | - Yun-Fei Liu
- Department of Psychological and Brain Sciences, Johns Hopkins University, Baltimore, MD, USA
| | - Hanna Hillman
- Princeton Neuroscience Institute and Department of Psychology, Princeton University, Princeton, NJ, USA
| | - Asieh Zadbood
- Princeton Neuroscience Institute and Department of Psychology, Princeton University, Princeton, NJ, USA
| | - Liat Hasenfratz
- Princeton Neuroscience Institute and Department of Psychology, Princeton University, Princeton, NJ, USA
| | - Neggin Keshavarzian
- Princeton Neuroscience Institute and Department of Psychology, Princeton University, Princeton, NJ, USA
| | - Janice Chen
- Department of Psychological and Brain Sciences, Johns Hopkins University, Baltimore, MD, USA
| | - Christopher J Honey
- Department of Psychological and Brain Sciences, Johns Hopkins University, Baltimore, MD, USA
| | - Yaara Yeshurun
- School of Psychological Sciences, Tel Aviv University, Tel Aviv, Israel
| | - Mor Regev
- Montreal Neurological Institute, McGill University, Montreal, QC, Canada
| | - Mai Nguyen
- Princeton Neuroscience Institute and Department of Psychology, Princeton University, Princeton, NJ, USA
| | - Claire H C Chang
- Princeton Neuroscience Institute and Department of Psychology, Princeton University, Princeton, NJ, USA
| | | | - Olga Lositsky
- Department of Cognitive, Linguistic and Psychological Sciences, Brown University, Providence, RI, USA
| | - Erez Simony
- Faculty of Electrical Engineering, Holon Institute of Technology, Holon, Israel
- Department of Neurobiology, Weizmann Institute of Science, Rehovot, Israel
| | | | - Yuan Chang Leong
- Helen Wills Neuroscience Institute, University of California, Berkeley, CA, USA
| | - Paula P Brooks
- Princeton Neuroscience Institute and Department of Psychology, Princeton University, Princeton, NJ, USA
| | - Emily Micciche
- Peabody College, Vanderbilt University, Nashville, TN, USA
| | - Gina Choe
- Princeton Neuroscience Institute and Department of Psychology, Princeton University, Princeton, NJ, USA
| | - Ariel Goldstein
- Princeton Neuroscience Institute and Department of Psychology, Princeton University, Princeton, NJ, USA
| | - Tamara Vanderwal
- Department of Psychiatry, University of British Columbia, and BC Children's Hospital Research Institute, Vancouver, BC, Canada
| | - Yaroslav O Halchenko
- Department of Psychological and Brain Sciences and Department of Computer Science, Dartmouth College, Hanover, NH, USA
| | - Kenneth A Norman
- Princeton Neuroscience Institute and Department of Psychology, Princeton University, Princeton, NJ, USA
| | - Uri Hasson
- Princeton Neuroscience Institute and Department of Psychology, Princeton University, Princeton, NJ, USA
| |
Collapse
|
17
|
Park S, Kim DW, Han CH, Im CH. Estimation of Emotional Arousal Changes of a Group of Individuals During Movie Screening Using Steady-State Visual-Evoked Potential. Front Neuroinform 2021; 15:731236. [PMID: 34566617 PMCID: PMC8457524 DOI: 10.3389/fninf.2021.731236] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/26/2021] [Accepted: 08/10/2021] [Indexed: 11/13/2022] Open
Abstract
Neurocinematics is an emerging discipline in neuroscience, which aims to provide new filmmaking techniques by analyzing the brain activities of a group of audiences. Several neurocinematics studies attempted to track temporal changes in mental states during movie screening; however, it is still needed to develop efficient and robust electroencephalography (EEG) features for tracking brain states precisely over a long period. This study proposes a novel method for estimating emotional arousal changes in a group of individuals during movie screening by employing steady-state visual evoked potential (SSVEP), which is a widely used EEG response elicited by the presentation of periodic visual stimuli. Previous studies have reported that the emotional arousal of each individual modulates the strength of SSVEP responses. Based on this phenomenon, movie clips were superimposed on a background, eliciting an SSVEP response with a specific frequency. Two emotionally arousing movie clips were presented to six healthy male participants, while EEG signals were recorded from the occipital channels. We then investigated whether the movie scenes that elicited higher SSVEP responses coincided well with those rated as the most impressive scenes by 37 viewers in a separate experimental session. Our results showed that the SSVEP response averaged across six participants could accurately predict the overall impressiveness of each movie, evaluated with a much larger group of individuals.
Collapse
Affiliation(s)
- Seonghun Park
- Computational Neuroengineering Laboratory, Department of Biomedical Engineering, Hanyang University, Seoul, South Korea.,Department of Electronic Engineering, Hanyang University, Seoul, South Korea
| | - Do-Won Kim
- Department of Biomedical Engineering, Chonnam National University, Yeosu, South Korea.,School of Healthcare and Biomedical Engineering, Chonnam National University, Yeosu, South Korea
| | - Chang-Hee Han
- Computational Neuroengineering Laboratory, Department of Biomedical Engineering, Hanyang University, Seoul, South Korea.,Department of Software Engineering, Dongseo University, Pusan, South Korea
| | - Chang-Hwan Im
- Computational Neuroengineering Laboratory, Department of Biomedical Engineering, Hanyang University, Seoul, South Korea.,Department of Electronic Engineering, Hanyang University, Seoul, South Korea
| |
Collapse
|
18
|
Hybrid hyperalignment: A single high-dimensional model of shared information embedded in cortical patterns of response and functional connectivity. Neuroimage 2021; 233:117975. [PMID: 33762217 PMCID: PMC8273921 DOI: 10.1016/j.neuroimage.2021.117975] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/25/2020] [Revised: 02/20/2021] [Accepted: 03/14/2021] [Indexed: 11/26/2022] Open
Abstract
Shared information content is represented across brains in idiosyncratic functional topographies. Hyperalignment addresses these idiosyncrasies by using neural responses to project individuals’ brain data into a common model space while maintaining the geometric relationships between distinct patterns of activity or connectivity. The dimensions of this common model capture functional profiles that are shared across individuals such as cortical response profiles collected during a common time-locked stimulus presentation (e.g. movie viewing) or functional connectivity profiles. Hyperalignment can use either response-based or connectivity-based input data to derive transformations that project individuals’ neural data from anatomical space into the common model space. Previously, only response or connectivity profiles were used in the derivation of these transformations. In this study, we developed a new hyperalignment algorithm, hybrid hyperalignment, that derives transformations based on both response-based and connectivity-based information. We used three different movie-viewing fMRI datasets to test the performance of our new algorithm. Hybrid hyperalignment derives a single common model space that aligns response-based information as well as or better than response hyperalignment while simultaneously aligning connectivity-based information better than connectivity hyperalignment. These results suggest that a single common information space can encode both shared cortical response and functional connectivity profiles across individuals.
Collapse
|