1
|
Adomaitis L, Grinbaum A. Neurotechnologies, Ethics, and the Limits of Free Will. Integr Psychol Behav Sci 2024; 58:894-907. [PMID: 38388982 DOI: 10.1007/s12124-024-09830-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 02/19/2024] [Indexed: 02/24/2024]
Abstract
This article delves into the implications of neurotechnologies for the philosophical debates surrounding free will and moral responsibility. Tracing the concept from ancient religious and philosophical roots, we discuss how recent neurotechnological advancements (e.g. optogenetics, fMRI and machine learning, predictive diagnostics, et al.) challenge traditional notions of autonomy. Although neurotechnologies aim to enhance autonomy in the strict sense - as self-determination - they risk reducing or changing the broader notion of autonomy, which involves personal authenticity. We also submit that, in a world with an altered or limited concept of free will, humans should still be held accountable for actions executed through their bodies. By examining the dynamic between choice and responsibility, we emphasize the shift in technology ethics, moral philosophy, and the broader legal landscape in response to the advancement of neurotechnologies. By bringing the neurotechnological innovations into the world, neuroscientists not only change the technological landscape but also partake in long-standing moral narratives about freedom, justice, and responsibility.
Collapse
|
2
|
Fantoni M, Federici A, Camponogara I, Handjaras G, Martinelli A, Bednaya E, Ricciardi E, Pavani F, Bottari D. The impact of face masks on face-to-face neural tracking of speech: Auditory and visual obstacles. Heliyon 2024; 10:e34860. [PMID: 39157360 PMCID: PMC11328033 DOI: 10.1016/j.heliyon.2024.e34860] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/29/2024] [Revised: 07/17/2024] [Accepted: 07/17/2024] [Indexed: 08/20/2024] Open
Abstract
Face masks provide fundamental protection against the transmission of respiratory viruses but hamper communication. We estimated auditory and visual obstacles generated by face masks on communication by measuring the neural tracking of speech. To this end, we recorded the EEG while participants were exposed to naturalistic audio-visual speech, embedded in 5-talker noise, in three contexts: (i) no-mask (audio-visual information was fully available), (ii) virtual mask (occluded lips, but intact audio), and (iii) real mask (occluded lips and degraded audio). Neural tracking of lip movements and of the sound envelope of speech was measured through backward modeling, that is, by reconstructing stimulus properties from neural activity. Behaviorally, face masks increased perceived listening difficulty and phonological errors in speech content retrieval. At the neural level, we observed that the occlusion of the mouth abolished lip tracking and dampened neural tracking of the speech envelope at the earliest processing stages. By contrast, degraded acoustic information related to face mask filtering altered neural tracking of speech envelope at later processing stages. Finally, a consistent link emerged between the increment of perceived listening difficulty and the drop in reconstruction performance of speech envelope when attending to a speaker wearing a face mask. Results clearly dissociated the visual and auditory impact of face masks on the neural tracking of speech. While the visual obstacle related to face masks hampered the ability to predict and integrate audio-visual speech, the auditory filter generated by face masks impacted neural processing stages typically associated with auditory selective attention. The link between perceived difficulty and neural tracking drop also provides evidence of the impact of face masks on the metacognitive levels subtending face-to-face communication.
Collapse
Affiliation(s)
- M. Fantoni
- MoMiLab, IMT School for Advanced Studies Lucca, Lucca, Italy
| | - A. Federici
- MoMiLab, IMT School for Advanced Studies Lucca, Lucca, Italy
| | | | - G. Handjaras
- MoMiLab, IMT School for Advanced Studies Lucca, Lucca, Italy
| | | | - E. Bednaya
- MoMiLab, IMT School for Advanced Studies Lucca, Lucca, Italy
| | - E. Ricciardi
- MoMiLab, IMT School for Advanced Studies Lucca, Lucca, Italy
| | - F. Pavani
- Centro Interdipartimentale Mente/Cervello–CIMEC, University of Trento, Italy
- Centro Interuniversitario di Ricerca “Cognizione Linguaggio e Sordità”–CIRCLeS, University of Trento, Italy
| | - D. Bottari
- MoMiLab, IMT School for Advanced Studies Lucca, Lucca, Italy
| |
Collapse
|
3
|
Lin R, Naselaris T, Kay K, Wehbe L. Stacked regressions and structured variance partitioning for interpretable brain maps. Neuroimage 2024; 298:120772. [PMID: 39117095 DOI: 10.1016/j.neuroimage.2024.120772] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/26/2023] [Revised: 07/26/2024] [Accepted: 08/02/2024] [Indexed: 08/10/2024] Open
Abstract
Relating brain activity associated with a complex stimulus to different properties of that stimulus is a powerful approach for constructing functional brain maps. However, when stimuli are naturalistic, their properties are often correlated (e.g., visual and semantic features of natural images, or different layers of a convolutional neural network that are used as features of images). Correlated properties can act as confounders for each other and complicate the interpretability of brain maps, and can impact the robustness of statistical estimators. Here, we present an approach for brain mapping based on two proposed methods: stacking different encoding models and structured variance partitioning. Our stacking algorithm combines encoding models that each uses as input a feature space that describes a different stimulus attribute. The algorithm learns to predict the activity of a voxel as a linear combination of the outputs of different encoding models. We show that the resulting combined model can predict held-out brain activity better or at least as well as the individual encoding models. Further, the weights of the linear combination are readily interpretable; they show the importance of each feature space for predicting a voxel. We then build on our stacking models to introduce structured variance partitioning, a new type of variance partitioning that takes into account the known relationships between features. Our approach constrains the size of the hypothesis space and allows us to ask targeted questions about the similarity between feature spaces and brain regions even in the presence of correlations between the feature spaces. We validate our approach in simulation, showcase its brain mapping potential on fMRI data, and release a Python package. Our methods can be useful for researchers interested in aligning brain activity with different layers of a neural network, or with other types of correlated feature spaces.
Collapse
Affiliation(s)
- Ruogu Lin
- Computational Biology Department, Carnegie Mellon University, Pittsburgh, PA 15213, United States of America
| | - Thomas Naselaris
- Department of Neuroscience, University of Minnesota, Minneapolis, MN 55455, United States of America; Center for Magnetic Resonance Research (CMRR), Department of Radiology, University of Minnesota, Minneapolis, MN 55455, United States of America
| | - Kendrick Kay
- Center for Magnetic Resonance Research (CMRR), Department of Radiology, University of Minnesota, Minneapolis, MN 55455, United States of America
| | - Leila Wehbe
- Neuroscience Institute, Carnegie Mellon University, Pittsburgh, PA 15213, United States of America; Machine Learning Department, Carnegie Mellon University, Pittsburgh, PA 15213, United States of America.
| |
Collapse
|
4
|
Shimaoka D, Wong YT, Rosa MG, Seow N, Price C. Naturalistic Movies and Encoding, Analysis Define Areal Borders in, Marmoset Third-Tier Visual Cortex. Prog Neurobiol 2024:102657. [PMID: 39103115 DOI: 10.1016/j.pneurobio.2024.102657] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/18/2024] [Revised: 06/24/2024] [Accepted: 07/30/2024] [Indexed: 08/07/2024]
Abstract
Accurate definition of the borders of cortical visual areas is essential for the study.of neuronal processes leading to perception. However, data used for definition of.areal boundaries have suffered from issues related to resolution, uniform.coverage, or suitability for objective analysis, leading to ambiguity. Here, we.present a novel approach that combines widefield optical imaging, presentation.of naturalistic movies, and encoding model analysis, to objectively define borders.in the primate extrastriate cortex. We applied this method to test conflicting.hypotheses about the third-tier visual cortex, where areal boundaries have.remained controversial. We demonstrate pronounced tuning preferences in the.third-tier areas, and an organizational structure in which the dorsomedial area.(DM) contains representations of both the upper and lower contralateral.quadrants, and is located immediate anterior to V2. High-density.electrophysiological recordings with a Neuropixels probe confirm these findings.Our encoding-model approach offers a powerful, objective way to disambiguate.areal boundaries….
Collapse
Affiliation(s)
- Daisuke Shimaoka
- Department of Physiology and Neuroscience Program, Biomedicine Discovery, Institute, Monash University, Clayton, Australia
| | - Yan Tat Wong
- Department of Physiology and Neuroscience Program, Biomedicine Discovery, Institute, Monash University, Clayton, Australia; Electrical and Computer Systems Engineering, Monash University, Clayton, Australia
| | - Marcello Gp Rosa
- Department of Physiology and Neuroscience Program, Biomedicine Discovery, Institute, Monash University, Clayton, Australia
| | - Nicholas Seow
- Department of Physiology and Neuroscience Program, Biomedicine Discovery, Institute, Monash University, Clayton, Australia; Electrical and Computer Systems Engineering, Monash University, Clayton, Australia
| | - Chiang Price
- Department of Physiology and Neuroscience Program, Biomedicine Discovery, Institute, Monash University, Clayton, Australia
| |
Collapse
|
5
|
Zada Z, Goldstein A, Michelmann S, Simony E, Price A, Hasenfratz L, Barham E, Zadbood A, Doyle W, Friedman D, Dugan P, Melloni L, Devore S, Flinker A, Devinsky O, Nastase SA, Hasson U. A shared model-based linguistic space for transmitting our thoughts from brain to brain in natural conversations. Neuron 2024:S0896-6273(24)00460-4. [PMID: 39096896 DOI: 10.1016/j.neuron.2024.06.025] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/03/2023] [Revised: 03/26/2024] [Accepted: 06/25/2024] [Indexed: 08/05/2024]
Abstract
Effective communication hinges on a mutual understanding of word meaning in different contexts. We recorded brain activity using electrocorticography during spontaneous, face-to-face conversations in five pairs of epilepsy patients. We developed a model-based coupling framework that aligns brain activity in both speaker and listener to a shared embedding space from a large language model (LLM). The context-sensitive LLM embeddings allow us to track the exchange of linguistic information, word by word, from one brain to another in natural conversations. Linguistic content emerges in the speaker's brain before word articulation and rapidly re-emerges in the listener's brain after word articulation. The contextual embeddings better capture word-by-word neural alignment between speaker and listener than syntactic and articulatory models. Our findings indicate that the contextual embeddings learned by LLMs can serve as an explicit numerical model of the shared, context-rich meaning space humans use to communicate their thoughts to one another.
Collapse
Affiliation(s)
- Zaid Zada
- Princeton Neuroscience Institute and Department of Psychology, Princeton University, Princeton, NJ 08544, USA.
| | - Ariel Goldstein
- Princeton Neuroscience Institute and Department of Psychology, Princeton University, Princeton, NJ 08544, USA; Department of Cognitive and Brain Sciences and Business School, Hebrew University, Jerusalem 9190501, Israel
| | - Sebastian Michelmann
- Princeton Neuroscience Institute and Department of Psychology, Princeton University, Princeton, NJ 08544, USA
| | - Erez Simony
- Princeton Neuroscience Institute and Department of Psychology, Princeton University, Princeton, NJ 08544, USA; Faculty of Engineering, Holon Institute of Technology, Holon 5810201, Israel
| | - Amy Price
- Princeton Neuroscience Institute and Department of Psychology, Princeton University, Princeton, NJ 08544, USA
| | - Liat Hasenfratz
- Princeton Neuroscience Institute and Department of Psychology, Princeton University, Princeton, NJ 08544, USA
| | - Emily Barham
- Princeton Neuroscience Institute and Department of Psychology, Princeton University, Princeton, NJ 08544, USA
| | - Asieh Zadbood
- Princeton Neuroscience Institute and Department of Psychology, Princeton University, Princeton, NJ 08544, USA; Department of Psychology, Columbia University, New York, NY 10027, USA
| | - Werner Doyle
- Grossman School of Medicine, New York University, New York, NY 10016, USA
| | - Daniel Friedman
- Grossman School of Medicine, New York University, New York, NY 10016, USA
| | - Patricia Dugan
- Grossman School of Medicine, New York University, New York, NY 10016, USA
| | - Lucia Melloni
- Grossman School of Medicine, New York University, New York, NY 10016, USA
| | - Sasha Devore
- Grossman School of Medicine, New York University, New York, NY 10016, USA
| | - Adeen Flinker
- Grossman School of Medicine, New York University, New York, NY 10016, USA; Tandon School of Engineering, New York University, New York, NY 10016, USA
| | - Orrin Devinsky
- Grossman School of Medicine, New York University, New York, NY 10016, USA
| | - Samuel A Nastase
- Princeton Neuroscience Institute and Department of Psychology, Princeton University, Princeton, NJ 08544, USA
| | - Uri Hasson
- Princeton Neuroscience Institute and Department of Psychology, Princeton University, Princeton, NJ 08544, USA
| |
Collapse
|
6
|
Simonelli F, Handjaras G, Benuzzi F, Bernardi G, Leo A, Duzzi D, Cecchetti L, Nichelli PF, Porro CA, Pietrini P, Ricciardi E, Lui F. Sensitivity and specificity of the action observation network to kinematics, target object, and gesture meaning. Hum Brain Mapp 2024; 45:e26762. [PMID: 39037079 PMCID: PMC11261593 DOI: 10.1002/hbm.26762] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/31/2023] [Revised: 05/23/2024] [Accepted: 06/02/2024] [Indexed: 07/23/2024] Open
Abstract
Hierarchical models have been proposed to explain how the brain encodes actions, whereby different areas represent different features, such as gesture kinematics, target object, action goal, and meaning. The visual processing of action-related information is distributed over a well-known network of brain regions spanning separate anatomical areas, attuned to specific stimulus properties, and referred to as action observation network (AON). To determine the brain organization of these features, we measured representational geometries during the observation of a large set of transitive and intransitive gestures in two independent functional magnetic resonance imaging experiments. We provided evidence for a partial dissociation between kinematics, object characteristics, and action meaning in the occipito-parietal, ventro-temporal, and lateral occipito-temporal cortex, respectively. Importantly, most of the AON showed low specificity to all the explored features, and representational spaces sharing similar information content were spread across the cortex without being anatomically adjacent. Overall, our results support the notion that the AON relies on overlapping and distributed coding and may act as a unique representational space instead of mapping features in a modular and segregated manner.
Collapse
Affiliation(s)
| | | | - Francesca Benuzzi
- Department of Biomedical, Metabolic and Neural Sciences and Center for Neuroscience and NeurotechnologyUniversity of Modena and Reggio EmiliaModenaItaly
| | | | - Andrea Leo
- IMT School for Advanced Studies LuccaLuccaItaly
| | - Davide Duzzi
- Department of Biomedical, Metabolic and Neural Sciences and Center for Neuroscience and NeurotechnologyUniversity of Modena and Reggio EmiliaModenaItaly
| | | | - Paolo F. Nichelli
- Department of Biomedical, Metabolic and Neural Sciences and Center for Neuroscience and NeurotechnologyUniversity of Modena and Reggio EmiliaModenaItaly
| | - Carlo A. Porro
- Department of Biomedical, Metabolic and Neural Sciences and Center for Neuroscience and NeurotechnologyUniversity of Modena and Reggio EmiliaModenaItaly
| | | | | | - Fausta Lui
- Department of Biomedical, Metabolic and Neural Sciences and Center for Neuroscience and NeurotechnologyUniversity of Modena and Reggio EmiliaModenaItaly
| |
Collapse
|
7
|
Chen Y, Beech P, Yin Z, Jia S, Zhang J, Yu Z, Liu JK. Decoding dynamic visual scenes across the brain hierarchy. PLoS Comput Biol 2024; 20:e1012297. [PMID: 39093861 PMCID: PMC11324145 DOI: 10.1371/journal.pcbi.1012297] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/12/2023] [Revised: 08/14/2024] [Accepted: 07/03/2024] [Indexed: 08/04/2024] Open
Abstract
Understanding the computational mechanisms that underlie the encoding and decoding of environmental stimuli is a crucial investigation in neuroscience. Central to this pursuit is the exploration of how the brain represents visual information across its hierarchical architecture. A prominent challenge resides in discerning the neural underpinnings of the processing of dynamic natural visual scenes. Although considerable research efforts have been made to characterize individual components of the visual pathway, a systematic understanding of the distinctive neural coding associated with visual stimuli, as they traverse this hierarchical landscape, remains elusive. In this study, we leverage the comprehensive Allen Visual Coding-Neuropixels dataset and utilize the capabilities of deep learning neural network models to study neural coding in response to dynamic natural visual scenes across an expansive array of brain regions. Our study reveals that our decoding model adeptly deciphers visual scenes from neural spiking patterns exhibited within each distinct brain area. A compelling observation arises from the comparative analysis of decoding performances, which manifests as a notable encoding proficiency within the visual cortex and subcortical nuclei, in contrast to a relatively reduced encoding activity within hippocampal neurons. Strikingly, our results unveil a robust correlation between our decoding metrics and well-established anatomical and functional hierarchy indexes. These findings corroborate existing knowledge in visual coding related to artificial visual stimuli and illuminate the functional role of these deeper brain regions using dynamic stimuli. Consequently, our results suggest a novel perspective on the utility of decoding neural network models as a metric for quantifying the encoding quality of dynamic natural visual scenes represented by neural responses, thereby advancing our comprehension of visual coding within the complex hierarchy of the brain.
Collapse
Affiliation(s)
- Ye Chen
- School of Computer Science, Peking University, Beijing, China
- Institute for Artificial Intelligence, Peking University, Beijing, China
| | - Peter Beech
- School of Computing, University of Leeds, Leeds, United Kingdom
| | - Ziwei Yin
- School of Computer Science, Centre for Human Brain Health, University of Birmingham, Birmingham, United Kingdom
| | - Shanshan Jia
- School of Computer Science, Peking University, Beijing, China
- Institute for Artificial Intelligence, Peking University, Beijing, China
| | - Jiayi Zhang
- Institutes of Brain Science, State Key Laboratory of Medical Neurobiology, MOE Frontiers Center for Brain Science and Institute for Medical and Engineering Innovation, Eye & ENT Hospital, Fudan University, Shanghai, China
| | - Zhaofei Yu
- School of Computer Science, Peking University, Beijing, China
- Institute for Artificial Intelligence, Peking University, Beijing, China
| | - Jian K. Liu
- School of Computing, University of Leeds, Leeds, United Kingdom
- School of Computer Science, Centre for Human Brain Health, University of Birmingham, Birmingham, United Kingdom
| |
Collapse
|
8
|
Lahner B, Dwivedi K, Iamshchinina P, Graumann M, Lascelles A, Roig G, Gifford AT, Pan B, Jin S, Ratan Murty NA, Kay K, Oliva A, Cichy R. Modeling short visual events through the BOLD moments video fMRI dataset and metadata. Nat Commun 2024; 15:6241. [PMID: 39048577 PMCID: PMC11269733 DOI: 10.1038/s41467-024-50310-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/14/2023] [Accepted: 07/04/2024] [Indexed: 07/27/2024] Open
Abstract
Studying the neural basis of human dynamic visual perception requires extensive experimental data to evaluate the large swathes of functionally diverse brain neural networks driven by perceiving visual events. Here, we introduce the BOLD Moments Dataset (BMD), a repository of whole-brain fMRI responses to over 1000 short (3 s) naturalistic video clips of visual events across ten human subjects. We use the videos' extensive metadata to show how the brain represents word- and sentence-level descriptions of visual events and identify correlates of video memorability scores extending into the parietal cortex. Furthermore, we reveal a match in hierarchical processing between cortical regions of interest and video-computable deep neural networks, and we showcase that BMD successfully captures temporal dynamics of visual events at second resolution. With its rich metadata, BMD offers new perspectives and accelerates research on the human brain basis of visual event perception.
Collapse
Affiliation(s)
- Benjamin Lahner
- Computer Science and Artificial Intelligence Laboratory, MIT, Cambridge, MA, USA.
| | - Kshitij Dwivedi
- Department of Education and Psychology, Freie Universität Berlin, Berlin, Germany
- Department of Computer Science, Goethe University Frankfurt, Frankfurt am Main, Germany
| | - Polina Iamshchinina
- Department of Education and Psychology, Freie Universität Berlin, Berlin, Germany
| | - Monika Graumann
- Department of Education and Psychology, Freie Universität Berlin, Berlin, Germany
| | - Alex Lascelles
- Computer Science and Artificial Intelligence Laboratory, MIT, Cambridge, MA, USA
| | - Gemma Roig
- Department of Computer Science, Goethe University Frankfurt, Frankfurt am Main, Germany
- The Hessian Center for AI (hessian.AI), Darmstadt, Germany
| | | | - Bowen Pan
- Computer Science and Artificial Intelligence Laboratory, MIT, Cambridge, MA, USA
| | - SouYoung Jin
- Computer Science and Artificial Intelligence Laboratory, MIT, Cambridge, MA, USA
| | - N Apurva Ratan Murty
- Department of Brain and Cognitive Science, MIT, Cambridge, MA, USA
- School of Psychology, Georgia Institute of Technology, Atlanta, GA, USA
| | - Kendrick Kay
- Center for Magnetic Resonance Research (CMRR), Department of Radiology, University of Minnesota, Minneapolis, MN, USA
| | - Aude Oliva
- Computer Science and Artificial Intelligence Laboratory, MIT, Cambridge, MA, USA
| | - Radoslaw Cichy
- Department of Education and Psychology, Freie Universität Berlin, Berlin, Germany
| |
Collapse
|
9
|
Shreesha L, Levin M. Stress sharing as cognitive glue for collective intelligences: A computational model of stress as a coordinator for morphogenesis. Biochem Biophys Res Commun 2024; 731:150396. [PMID: 39018974 DOI: 10.1016/j.bbrc.2024.150396] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/18/2024] [Revised: 07/03/2024] [Accepted: 07/11/2024] [Indexed: 07/19/2024]
Abstract
Individual cells have numerous competencies in physiological and metabolic spaces. However, multicellular collectives can reliably navigate anatomical morphospace towards much larger, reliable endpoints. Understanding the robustness and control properties of this process is critical for evolutionary developmental biology, bioengineering, and regenerative medicine. One mechanism that has been proposed for enabling individual cells to coordinate toward specific morphological outcomes is the sharing of stress (where stress is a physiological parameter that reflects the current amount of error in the context of a homeostatic loop). Here, we construct and analyze a multiscale agent-based model of morphogenesis in which we quantitatively examine the impact of stress sharing on the ability to reach target morphology. We found that stress sharing improves the morphogenetic efficiency of multicellular collectives; populations with stress sharing reached anatomical targets faster. Moreover, stress sharing influenced the future fate of distant cells in the multi-cellular collective, enhancing cells' movement and their radius of influence, consistent with the hypothesis that stress sharing works to increase cohesiveness of collectives. During development, anatomical goal states could not be inferred from observation of stress states, revealing the limitations of knowledge of goals by an extern observer outside the system itself. Taken together, our analyses support an important role for stress sharing in natural and engineered systems that seek robust large-scale behaviors to emerge from the activity of their competent components.
Collapse
Affiliation(s)
| | - Michael Levin
- Department of Biology, Tufts University, Medford, MA, 02155, USA; Allen Discovery Center at Tufts University, Medford, MA, 02155, USA; Wyss Institute for Biologically Inspired Engineering, Harvard University, Boston, MA, 02115, USA.
| |
Collapse
|
10
|
Sharon O, Ben Simon E, Shah VD, Desel T, Walker MP. The new science of sleep: From cells to large-scale societies. PLoS Biol 2024; 22:e3002684. [PMID: 38976664 PMCID: PMC11230563 DOI: 10.1371/journal.pbio.3002684] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 07/10/2024] Open
Abstract
In the past 20 years, more remarkable revelations about sleep and its varied functions have arguably been made than in the previous 200. Building on this swell of recent findings, this essay provides a broad sampling of selected research highlights across genetic, molecular, cellular, and physiological systems within the body, networks within the brain, and large-scale social dynamics. Based on this raft of exciting new discoveries, we have come to realize that sleep, in this moment of its evolution, is very much polyfunctional (rather than monofunctional), yet polyfunctional for reasons we had never previously considered. Moreover, these new polyfunctional insights powerfully reaffirm sleep as a critical biological, and thus health-sustaining, requisite. Indeed, perhaps the only thing more impressive than the unanticipated nature of these newly emerging sleep functions is their striking divergence, from operations of molecular mechanisms inside cells to entire group societal dynamics.
Collapse
Affiliation(s)
- Omer Sharon
- Department of Psychology, University of California, Berkeley, California, United States of America
- Helen Wills Neuroscience Institute, University of California, Berkeley, California, United States of America
| | - Eti Ben Simon
- Department of Psychology, University of California, Berkeley, California, United States of America
- Helen Wills Neuroscience Institute, University of California, Berkeley, California, United States of America
| | - Vyoma D. Shah
- Department of Psychology, University of California, Berkeley, California, United States of America
- Helen Wills Neuroscience Institute, University of California, Berkeley, California, United States of America
| | - Tenzin Desel
- Department of Psychology, University of California, Berkeley, California, United States of America
- Helen Wills Neuroscience Institute, University of California, Berkeley, California, United States of America
| | - Matthew P. Walker
- Department of Psychology, University of California, Berkeley, California, United States of America
- Helen Wills Neuroscience Institute, University of California, Berkeley, California, United States of America
| |
Collapse
|
11
|
Wang R, Chen ZS. Large-scale foundation models and generative AI for BigData neuroscience. Neurosci Res 2024:S0168-0102(24)00075-0. [PMID: 38897235 DOI: 10.1016/j.neures.2024.06.003] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/21/2023] [Revised: 04/15/2024] [Accepted: 05/15/2024] [Indexed: 06/21/2024]
Abstract
Recent advances in machine learning have led to revolutionary breakthroughs in computer games, image and natural language understanding, and scientific discovery. Foundation models and large-scale language models (LLMs) have recently achieved human-like intelligence thanks to BigData. With the help of self-supervised learning (SSL) and transfer learning, these models may potentially reshape the landscapes of neuroscience research and make a significant impact on the future. Here we present a mini-review on recent advances in foundation models and generative AI models as well as their applications in neuroscience, including natural language and speech, semantic memory, brain-machine interfaces (BMIs), and data augmentation. We argue that this paradigm-shift framework will open new avenues for many neuroscience research directions and discuss the accompanying challenges and opportunities.
Collapse
Affiliation(s)
- Ran Wang
- Department of Psychiatry, New York University Grossman School of Medicine, New York, NY 10016, USA
| | - Zhe Sage Chen
- Department of Psychiatry, New York University Grossman School of Medicine, New York, NY 10016, USA; Department of Neuroscience and Physiology, Neuroscience Institute, New York University Grossman School of Medicine, New York, NY 10016, USA; Department of Biomedical Engineering, New York University Tandon School of Engineering, Brooklyn, NY 11201, USA.
| |
Collapse
|
12
|
Pinho AL, Richard H, Ponce AF, Eickenberg M, Amadon A, Dohmatob E, Denghien I, Torre JJ, Shankar S, Aggarwal H, Thual A, Chapalain T, Ginisty C, Becuwe-Desmidt S, Roger S, Lecomte Y, Berland V, Laurier L, Joly-Testault V, Médiouni-Cloarec G, Doublé C, Martins B, Varoquaux G, Dehaene S, Hertz-Pannier L, Thirion B. Individual Brain Charting dataset extension, third release for movie watching and retinotopy data. Sci Data 2024; 11:590. [PMID: 38839770 PMCID: PMC11153490 DOI: 10.1038/s41597-024-03390-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/21/2023] [Accepted: 05/20/2024] [Indexed: 06/07/2024] Open
Abstract
The Individual Brain Charting (IBC) is a multi-task functional Magnetic Resonance Imaging dataset acquired at high spatial-resolution and dedicated to the cognitive mapping of the human brain. It consists in the deep phenotyping of twelve individuals, covering a broad range of psychological domains suitable for functional-atlasing applications. Here, we present the inclusion of task data from both naturalistic stimuli and trial-based designs, to uncover structures of brain activation. We rely on the Fast Shared Response Model (FastSRM) to provide a data-driven solution for modelling naturalistic stimuli, typically containing many features. We show that data from left-out runs can be reconstructed using FastSRM, enabling the extraction of networks from the visual, auditory and language systems. We also present the topographic organization of the visual system through retinotopy. In total, six new tasks were added to IBC, wherein four trial-based retinotopic tasks contributed with a mapping of the visual field to the cortex. IBC is open access: source plus derivatives imaging data and meta-data are available in public repositories.
Collapse
Affiliation(s)
- Ana Luísa Pinho
- Université Paris-Saclay, Inria, CEA, Palaiseau, 91120, France.
- Department of Computer Science, Western University, London, Ontario, Canada.
- Western Centre for Brain and Mind, Western University, London, Ontario, Canada.
| | - Hugo Richard
- Université Paris-Saclay, Inria, CEA, Palaiseau, 91120, France
- Criteo AI Labs, Paris, France
- FAIRPLAY - IA coopérative: équité, vie privée, incitations, Paris, France
| | | | - Michael Eickenberg
- Université Paris-Saclay, Inria, CEA, Palaiseau, 91120, France
- Flatiron Institute, New York, USA
| | - Alexis Amadon
- Université Paris-Saclay, CEA, CNRS, BAOBAB, NeuroSpin, 91191, Gif-sur-Yvette, France
| | - Elvis Dohmatob
- Université Paris-Saclay, Inria, CEA, Palaiseau, 91120, France
- Meta FAIR, Paris, France
| | - Isabelle Denghien
- Cognitive Neuroimaging Unit, INSERM, CEA, Université Paris-Saclay, NeuroSpin center, 91191, Gif-sur-Yvette, France
| | | | - Swetha Shankar
- Université Paris-Saclay, Inria, CEA, Palaiseau, 91120, France
| | | | - Alexis Thual
- Université Paris-Saclay, Inria, CEA, Palaiseau, 91120, France
- Cognitive Neuroimaging Unit, INSERM, CEA, Université Paris-Saclay, NeuroSpin center, 91191, Gif-sur-Yvette, France
- Collège de France, Paris, France
| | | | | | | | | | - Yann Lecomte
- CEA Saclay/DRF/IFJ/NeuroSpin/UNIACT, Paris, France
| | | | | | | | | | | | | | - Gaël Varoquaux
- Université Paris-Saclay, Inria, CEA, Palaiseau, 91120, France
| | - Stanislas Dehaene
- Cognitive Neuroimaging Unit, INSERM, CEA, Université Paris-Saclay, NeuroSpin center, 91191, Gif-sur-Yvette, France
- Collège de France, Paris, France
| | - Lucie Hertz-Pannier
- CEA Saclay/DRF/IFJ/NeuroSpin/UNIACT, Paris, France
- UMR 1141, NeuroDiderot, Université de Paris, Paris, France
| | | |
Collapse
|
13
|
Markow ZE, Tripathy K, Svoboda AM, Schroeder ML, Rafferty SM, Richter EJ, Eggebrecht AT, Anastasio MA, Chevillet MA, Mugler EM, Naufel SN, Yin A, Trobaugh JW, Culver JP. Identifying Naturalistic Movies from Human Brain Activity with High-Density Diffuse Optical Tomography. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2024:2023.11.27.566650. [PMID: 38076976 PMCID: PMC10705261 DOI: 10.1101/2023.11.27.566650] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 01/31/2024]
Abstract
Modern neuroimaging modalities, particularly functional MRI (fMRI), can decode detailed human experiences. Thousands of viewed images can be identified or classified, and sentences can be reconstructed. Decoding paradigms often leverage encoding models that reduce the stimulus space into a smaller yet generalizable feature set. However, the neuroimaging devices used for detailed decoding are non-portable, like fMRI, or invasive, like electrocorticography, excluding application in naturalistic use. Wearable, non-invasive, but lower-resolution devices such as electroencephalography and functional near-infrared spectroscopy (fNIRS) have been limited to decoding between stimuli used during training. Herein we develop and evaluate model-based decoding with high-density diffuse optical tomography (HD-DOT), a higher-resolution expansion of fNIRS with demonstrated promise as a surrogate for fMRI. Using a motion energy model of visual content, we decoded the identities of novel movie clips outside the training set with accuracy far above chance for single-trial decoding. Decoding was robust to modulations of testing time window, different training and test imaging sessions, hemodynamic contrast, and optode array density. Our results suggest that HD-DOT can translate detailed decoding into naturalistic use.
Collapse
|
14
|
Dado T, Papale P, Lozano A, Le L, Wang F, van Gerven M, Roelfsema P, Güçlütürk Y, Güçlü U. Brain2GAN: Feature-disentangled neural encoding and decoding of visual perception in the primate brain. PLoS Comput Biol 2024; 20:e1012058. [PMID: 38709818 PMCID: PMC11098503 DOI: 10.1371/journal.pcbi.1012058] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/10/2023] [Revised: 05/16/2024] [Accepted: 04/08/2024] [Indexed: 05/08/2024] Open
Abstract
A challenging goal of neural coding is to characterize the neural representations underlying visual perception. To this end, multi-unit activity (MUA) of macaque visual cortex was recorded in a passive fixation task upon presentation of faces and natural images. We analyzed the relationship between MUA and latent representations of state-of-the-art deep generative models, including the conventional and feature-disentangled representations of generative adversarial networks (GANs) (i.e., z- and w-latents of StyleGAN, respectively) and language-contrastive representations of latent diffusion networks (i.e., CLIP-latents of Stable Diffusion). A mass univariate neural encoding analysis of the latent representations showed that feature-disentangled w representations outperform both z and CLIP representations in explaining neural responses. Further, w-latent features were found to be positioned at the higher end of the complexity gradient which indicates that they capture visual information relevant to high-level neural activity. Subsequently, a multivariate neural decoding analysis of the feature-disentangled representations resulted in state-of-the-art spatiotemporal reconstructions of visual perception. Taken together, our results not only highlight the important role of feature-disentanglement in shaping high-level neural representations underlying visual perception but also serve as an important benchmark for the future of neural coding.
Collapse
Affiliation(s)
- Thirza Dado
- Donders Institute for Brain, Cognition and Behaviour, Radboud University, Nijmegen, Netherlands
| | - Paolo Papale
- Department of Vision and Cognition, Netherlands Institute for Neuroscience, Amsterdam, Netherlands
| | - Antonio Lozano
- Department of Vision and Cognition, Netherlands Institute for Neuroscience, Amsterdam, Netherlands
| | - Lynn Le
- Donders Institute for Brain, Cognition and Behaviour, Radboud University, Nijmegen, Netherlands
| | - Feng Wang
- Department of Vision and Cognition, Netherlands Institute for Neuroscience, Amsterdam, Netherlands
| | - Marcel van Gerven
- Donders Institute for Brain, Cognition and Behaviour, Radboud University, Nijmegen, Netherlands
| | - Pieter Roelfsema
- Department of Vision and Cognition, Netherlands Institute for Neuroscience, Amsterdam, Netherlands
- Laboratory of Visual Brain Therapy, Sorbonne University, Paris, France
- Department of Integrative Neurophysiology, VU Amsterdam, Amsterdam, Netherlands
- Department of Psychiatry, Amsterdam UMC, Amsterdam, Netherlands
| | - Yağmur Güçlütürk
- Donders Institute for Brain, Cognition and Behaviour, Radboud University, Nijmegen, Netherlands
| | - Umut Güçlü
- Donders Institute for Brain, Cognition and Behaviour, Radboud University, Nijmegen, Netherlands
| |
Collapse
|
15
|
Lu Z, Julian JB, Aguirre GK, Epstein RA. Neural compass in the human brain during naturalistic virtual navigation. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2024:2024.04.18.590112. [PMID: 38712211 PMCID: PMC11071287 DOI: 10.1101/2024.04.18.590112] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/08/2024]
Abstract
Humans and animals maintain a consistent representation of their facing direction during spatial navigation. In rodents, head direction cells are believed to support this "neural compass", but identifying a similar mechanism in humans during dynamic naturalistic navigation has been challenging. To address this issue, we acquired fMRI data while participants freely navigated through a virtual reality city. Encoding model analyses revealed voxel clusters in retrosplenial complex and superior parietal lobule that exhibited reliable tuning as a function of facing direction. Crucially, these directional tunings were consistent across perceptually different versions of the city, spatially separated locations within the city, and motivationally distinct phases of the behavioral task. Analysis of the model weights indicated that these regions may represent facing direction relative to the principal axis of the environment. These findings reveal specific mechanisms in the human brain that allow us to maintain a sense of direction during naturalistic, dynamic navigation.
Collapse
|
16
|
Antonello R, Huth A. Predictive Coding or Just Feature Discovery? An Alternative Account of Why Language Models Fit Brain Data. NEUROBIOLOGY OF LANGUAGE (CAMBRIDGE, MASS.) 2024; 5:64-79. [PMID: 38645616 PMCID: PMC11025645 DOI: 10.1162/nol_a_00087] [Citation(s) in RCA: 4] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 02/28/2022] [Accepted: 10/26/2022] [Indexed: 04/23/2024]
Abstract
Many recent studies have shown that representations drawn from neural network language models are extremely effective at predicting brain responses to natural language. But why do these models work so well? One proposed explanation is that language models and brains are similar because they have the same objective: to predict upcoming words before they are perceived. This explanation is attractive because it lends support to the popular theory of predictive coding. We provide several analyses that cast doubt on this claim. First, we show that the ability to predict future words does not uniquely (or even best) explain why some representations are a better match to the brain than others. Second, we show that within a language model, representations that are best at predicting future words are strictly worse brain models than other representations. Finally, we argue in favor of an alternative explanation for the success of language models in neuroscience: These models are effective at predicting brain responses because they generally capture a wide variety of linguistic phenomena.
Collapse
Affiliation(s)
- Richard Antonello
- Department of Computer Science, University of Texas at Austin, Austin, TX, USA
| | - Alexander Huth
- Department of Computer Science, University of Texas at Austin, Austin, TX, USA
| |
Collapse
|
17
|
Li R, Li J, Wang C, Liu H, Liu T, Wang X, Zou T, Huang W, Yan H, Chen H. Multi-Semantic Decoding of Visual Perception with Graph Neural Networks. Int J Neural Syst 2024; 34:2450016. [PMID: 38372016 DOI: 10.1142/s0129065724500163] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/20/2024]
Abstract
Constructing computational decoding models to account for the cortical representation of semantic information plays a crucial role in understanding visual perception. The human visual system processes interactive relationships among different objects when perceiving the semantic contents of natural visions. However, the existing semantic decoding models commonly regard categories as completely separate and independent visually and semantically and rarely consider the relationships from prior information. In this work, a novel semantic graph learning model was proposed to decode multiple semantic categories of perceived natural images from brain activity. The proposed model was validated on the functional magnetic resonance imaging data collected from five normal subjects while viewing 2750 natural images comprising 52 semantic categories. The results showed that the Graph Neural Network-based decoding model achieved higher accuracies than other deep neural network models. Moreover, the co-occurrence probability among semantic categories showed a significant correlation with the decoding accuracy. Additionally, the results suggested that semantic content organized in a hierarchical way with higher visual areas was more closely related to the internal visual experience. Together, this study provides a superior computational framework for multi-semantic decoding that supports the visual integration mechanism of semantic processing.
Collapse
Affiliation(s)
- Rong Li
- The Center of Psychosomatic Medicine, Sichuan Provincial Center for Mental Health, Sichuan Provincial People's Hospital, University of Electronic Science and Technology of China, Chengdu 611731, P. R. China
- School of Life Science and Technology, University of Electronic Science and Technology of China, Chengdu 611731, P. R. China
- MOE Key Lab for Neuroinformation, High-Field Magnetic Resonance Brain Imaging, Key Laboratory of Sichuan Province, University of Electronic Science and Technology of China, Chengdu 611731, P. R. China
| | - Jiyi Li
- School of Life Science and Technology, University of Electronic Science and Technology of China, Chengdu 611731, P. R. China
- MOE Key Lab for Neuroinformation, High-Field Magnetic Resonance Brain Imaging, Key Laboratory of Sichuan Province, University of Electronic Science and Technology of China, Chengdu 611731, P. R. China
| | - Chong Wang
- School of Life Science and Technology, University of Electronic Science and Technology of China, Chengdu 611731, P. R. China
- MOE Key Lab for Neuroinformation, High-Field Magnetic Resonance Brain Imaging, Key Laboratory of Sichuan Province, University of Electronic Science and Technology of China, Chengdu 611731, P. R. China
| | - Haoxiang Liu
- School of Life Science and Technology, University of Electronic Science and Technology of China, Chengdu 611731, P. R. China
- MOE Key Lab for Neuroinformation, High-Field Magnetic Resonance Brain Imaging, Key Laboratory of Sichuan Province, University of Electronic Science and Technology of China, Chengdu 611731, P. R. China
| | - Tao Liu
- School of Life Science and Technology, University of Electronic Science and Technology of China, Chengdu 611731, P. R. China
- MOE Key Lab for Neuroinformation, High-Field Magnetic Resonance Brain Imaging, Key Laboratory of Sichuan Province, University of Electronic Science and Technology of China, Chengdu 611731, P. R. China
| | - Xuyang Wang
- School of Life Science and Technology, University of Electronic Science and Technology of China, Chengdu 611731, P. R. China
- MOE Key Lab for Neuroinformation, High-Field Magnetic Resonance Brain Imaging, Key Laboratory of Sichuan Province, University of Electronic Science and Technology of China, Chengdu 611731, P. R. China
| | - Ting Zou
- School of Life Science and Technology, University of Electronic Science and Technology of China, Chengdu 611731, P. R. China
- MOE Key Lab for Neuroinformation, High-Field Magnetic Resonance Brain Imaging, Key Laboratory of Sichuan Province, University of Electronic Science and Technology of China, Chengdu 611731, P. R. China
| | - Wei Huang
- School of Life Science and Technology, University of Electronic Science and Technology of China, Chengdu 611731, P. R. China
- MOE Key Lab for Neuroinformation, High-Field Magnetic Resonance Brain Imaging, Key Laboratory of Sichuan Province, University of Electronic Science and Technology of China, Chengdu 611731, P. R. China
| | - Hongmei Yan
- School of Life Science and Technology, University of Electronic Science and Technology of China, Chengdu 611731, P. R. China
- MOE Key Lab for Neuroinformation, High-Field Magnetic Resonance Brain Imaging, Key Laboratory of Sichuan Province, University of Electronic Science and Technology of China, Chengdu 611731, P. R. China
| | - Huafu Chen
- The Center of Psychosomatic Medicine, Sichuan Provincial Center for Mental Health, Sichuan Provincial People's Hospital, University of Electronic Science and Technology of China, Chengdu 611731, P. R. China
- School of Life Science and Technology, University of Electronic Science and Technology of China, Chengdu 611731, P. R. China
- MOE Key Lab for Neuroinformation, High-Field Magnetic Resonance Brain Imaging, Key Laboratory of Sichuan Province, University of Electronic Science and Technology of China, Chengdu 611731, P. R. China
| |
Collapse
|
18
|
Chen C, Dupré la Tour T, Gallant JL, Klein D, Deniz F. The cortical representation of language timescales is shared between reading and listening. Commun Biol 2024; 7:284. [PMID: 38454134 PMCID: PMC11245628 DOI: 10.1038/s42003-024-05909-z] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/25/2023] [Accepted: 02/12/2024] [Indexed: 03/09/2024] Open
Abstract
Language comprehension involves integrating low-level sensory inputs into a hierarchy of increasingly high-level features. Prior work studied brain representations of different levels of the language hierarchy, but has not determined whether these brain representations are shared between written and spoken language. To address this issue, we analyze fMRI BOLD data that were recorded while participants read and listened to the same narratives in each modality. Levels of the language hierarchy are operationalized as timescales, where each timescale refers to a set of spectral components of a language stimulus. Voxelwise encoding models are used to determine where different timescales are represented across the cerebral cortex, for each modality separately. These models reveal that between the two modalities timescale representations are organized similarly across the cortical surface. Our results suggest that, after low-level sensory processing, language integration proceeds similarly regardless of stimulus modality.
Collapse
Affiliation(s)
- Catherine Chen
- Department of Electrical Engineering and Computer Sciences, University of California, Berkeley, CA, USA.
| | - Tom Dupré la Tour
- Helen Wills Neuroscience Institute, University of California, Berkeley, CA, USA
| | - Jack L Gallant
- Helen Wills Neuroscience Institute, University of California, Berkeley, CA, USA
| | - Daniel Klein
- Department of Electrical Engineering and Computer Sciences, University of California, Berkeley, CA, USA
| | - Fatma Deniz
- Helen Wills Neuroscience Institute, University of California, Berkeley, CA, USA.
- Institute of Software Engineering and Theoretical Computer Science, Technische Universität Berlin, Berlin, Germany.
- Bernstein Center for Computational Neuroscience, Berlin, Germany.
| |
Collapse
|
19
|
Ravindra V, Fang CH, Grama A. May I see what you see? Predicting visual features from neuronal activity. iScience 2024; 27:108819. [PMID: 38303691 PMCID: PMC10831884 DOI: 10.1016/j.isci.2024.108819] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/24/2023] [Revised: 11/19/2023] [Accepted: 01/02/2024] [Indexed: 02/03/2024] Open
Abstract
Understanding brain response to audiovisual stimuli is a key challenge in understanding neuronal processes. In this paper, we describe our effort aimed at reconstructing video frames from observed functional MRI images. We also demonstrate that our model can predict visual objects. Our method constructs an autoencoder model for a set of training video segments to code video streams into their corresponding latent representations. Next, we learn a mapping from the observed fMRI response to the corresponding latent video frame representation. Finally, we pass the latent vectors computed using the fMRI response through the decoder to reconstruct the predicted image. We show that the representations of video frames and those constructed from corresponding fMRI images are highly clustered, the latent representations can be used to predict objects in video frames using just the fMRI frames, and fMRI responses can be used to reconstruct the inputs to predict the presence of faces.
Collapse
|
20
|
Koide-Majima N, Nishimoto S, Majima K. Mental image reconstruction from human brain activity: Neural decoding of mental imagery via deep neural network-based Bayesian estimation. Neural Netw 2024; 170:349-363. [PMID: 38016230 DOI: 10.1016/j.neunet.2023.11.024] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/10/2023] [Revised: 09/22/2023] [Accepted: 11/08/2023] [Indexed: 11/30/2023]
Abstract
Visual images observed by humans can be reconstructed from their brain activity. However, the visualization (externalization) of mental imagery is challenging. Only a few studies have reported successful visualization of mental imagery, and their visualizable images have been limited to specific domains such as human faces or alphabetical letters. Therefore, visualizing mental imagery for arbitrary natural images stands as a significant milestone. In this study, we achieved this by enhancing a previous method. Specifically, we demonstrated that the visual image reconstruction method proposed in the seminal study by Shen et al. (2019) heavily relied on low-level visual information decoded from the brain and could not efficiently utilize the semantic information that would be recruited during mental imagery. To address this limitation, we extended the previous method to a Bayesian estimation framework and introduced the assistance of semantic information into it. Our proposed framework successfully reconstructed both seen images (i.e., those observed by the human eye) and imagined images from brain activity. Quantitative evaluation showed that our framework could identify seen and imagined images highly accurately compared to the chance accuracy (seen: 90.7%, imagery: 75.6%, chance accuracy: 50.0%). In contrast, the previous method could only identify seen images (seen: 64.3%, imagery: 50.4%). These results suggest that our framework would provide a unique tool for directly investigating the subjective contents of the brain such as illusions, hallucinations, and dreams.
Collapse
Affiliation(s)
- Naoko Koide-Majima
- Center for Information and Neural Networks (CiNet), National Institute of Information and Communications Technology, Osaka 565-0871, Japan; Graduate School of Frontier Biosciences, Osaka University, Osaka 565-0871, Japan
| | - Shinji Nishimoto
- Center for Information and Neural Networks (CiNet), National Institute of Information and Communications Technology, Osaka 565-0871, Japan; Graduate School of Frontier Biosciences, Osaka University, Osaka 565-0871, Japan; Graduate School of Medicine, Osaka University, Osaka 565-0871, Japan
| | - Kei Majima
- Institute for Quantum Life Science, National Institutes for Quantum Science and Technology, Chiba 263-8555, Japan; JST PRESTO, Saitama 332-0012, Japan.
| |
Collapse
|
21
|
Maruya A, Zaidi Q. Perceptual transitions between object rigidity and non-rigidity: Competition and cooperation among motion energy, feature tracking, and shape-based priors. J Vis 2024; 24:3. [PMID: 38306112 PMCID: PMC10848565 DOI: 10.1167/jov.24.2.3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/01/2023] [Accepted: 12/20/2023] [Indexed: 02/03/2024] Open
Abstract
Why do moving objects appear rigid when projected retinal images are deformed non-rigidly? We used rotating rigid objects that can appear rigid or non-rigid to test whether shape features contribute to rigidity perception. When two circular rings were rigidly linked at an angle and jointly rotated at moderate speeds, observers reported that the rings wobbled and were not linked rigidly, but rigid rotation was reported at slow speeds. When gaps, paint, or vertices were added, the rings appeared rigidly rotating even at moderate speeds. At high speeds, all configurations appeared non-rigid. Salient features thus contribute to rigidity at slow and moderate speeds but not at high speeds. Simulated responses of arrays of motion-energy cells showed that motion flow vectors are predominantly orthogonal to the contours of the rings, not parallel to the rotation direction. A convolutional neural network trained to distinguish flow patterns for wobbling versus rotation gave a high probability of wobbling for the motion-energy flows. However, the convolutional neural network gave high probabilities of rotation for motion flows generated by tracking features with arrays of MT pattern-motion cells and corner detectors. In addition, circular rings can appear to spin and roll despite the absence of any sensory evidence, and this illusion is prevented by vertices, gaps, and painted segments, showing the effects of rotational symmetry and shape. Combining convolutional neural network outputs that give greater weight to motion energy at fast speeds and to feature tracking at slow speeds, with the shape-based priors for wobbling and rolling, explained rigid and non-rigid percepts across shapes and speeds (R2 = 0.95). The results demonstrate how cooperation and competition between different neuronal classes lead to specific states of visual perception and to transitions between the states.
Collapse
Affiliation(s)
- Akihito Maruya
- Graduate Center for Vision Research, State University of New York, New York, NY, USA
| | - Qasim Zaidi
- Graduate Center for Vision Research, State University of New York, New York, NY, USA
| |
Collapse
|
22
|
Kim I, Kupers ER, Lerma-Usabiaga G, Grill-Spector K. Characterizing Spatiotemporal Population Receptive Fields in Human Visual Cortex with fMRI. J Neurosci 2024; 44:e0803232023. [PMID: 37963768 PMCID: PMC10866195 DOI: 10.1523/jneurosci.0803-23.2023] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/04/2023] [Revised: 10/20/2023] [Accepted: 10/24/2023] [Indexed: 11/16/2023] Open
Abstract
The use of fMRI and computational modeling has advanced understanding of spatial characteristics of population receptive fields (pRFs) in human visual cortex. However, we know relatively little about the spatiotemporal characteristics of pRFs because neurons' temporal properties are one to two orders of magnitude faster than fMRI BOLD responses. Here, we developed an image-computable framework to estimate spatiotemporal pRFs from fMRI data. First, we developed a simulation software that predicts fMRI responses to a time-varying visual input given a spatiotemporal pRF model and solves the model parameters. The simulator revealed that ground-truth spatiotemporal parameters can be accurately recovered at the millisecond resolution from synthesized fMRI responses. Then, using fMRI and a novel stimulus paradigm, we mapped spatiotemporal pRFs in individual voxels across human visual cortex in 10 participants (both females and males). We find that a compressive spatiotemporal (CST) pRF model better explains fMRI responses than a conventional spatial pRF model across visual areas spanning the dorsal, lateral, and ventral streams. Further, we find three organizational principles of spatiotemporal pRFs: (1) from early to later areas within a visual stream, spatial and temporal windows of pRFs progressively increase in size and show greater compressive nonlinearities, (2) later visual areas show diverging spatial and temporal windows across streams, and (3) within early visual areas (V1-V3), both spatial and temporal windows systematically increase with eccentricity. Together, this computational framework and empirical results open exciting new possibilities for modeling and measuring fine-grained spatiotemporal dynamics of neural responses using fMRI.
Collapse
Affiliation(s)
- Insub Kim
- Department of Psychology, Stanford University, Stanford, CA, 94305
| | - Eline R Kupers
- Department of Psychology, Stanford University, Stanford, CA, 94305
| | - Garikoitz Lerma-Usabiaga
- BCBL. Basque Center on Cognition, Brain and Language, 20009 San Sebastian, Spain
- IKERBASQUE. Basque Foundation for Science, 48009 Bilbao, Spain
| | - Kalanit Grill-Spector
- Department of Psychology, Stanford University, Stanford, CA, 94305
- Wu Tsai Neurosciences Institute, Stanford University, Stanford, CA, 94305
| |
Collapse
|
23
|
Thirion B, Aggarwal H, Ponce AF, Pinho AL, Thual A. Should one go for individual- or group-level brain parcellations? A deep-phenotyping benchmark. Brain Struct Funct 2024; 229:161-181. [PMID: 38012283 DOI: 10.1007/s00429-023-02723-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/08/2023] [Accepted: 10/11/2023] [Indexed: 11/29/2023]
Abstract
The analysis and understanding of brain characteristics often require considering region-level information rather than voxel-sampled data. Subject-specific parcellations have been put forward in recent years, as they can adapt to individual brain organization and thus offer more accurate individual summaries than standard atlases. However, the price to pay for adaptability is the lack of group-level consistency of the data representation. Here, we investigate whether the good representations brought by individualized models are merely an effect of circular analysis, in which individual brain features are better represented by subject-specific summaries, or whether this carries over to new individuals, i.e., whether one can actually adapt an existing parcellation to new individuals and still obtain good summaries in these individuals. For this, we adapt a dictionary-learning method to produce brain parcellations. We use it on a deep-phenotyping dataset to assess quantitatively the patterns of activity obtained under naturalistic and controlled-task-based settings. We show that the benefits of individual parcellations are substantial, but that they vary a lot across brain systems.
Collapse
Affiliation(s)
| | | | | | - Ana Luísa Pinho
- Department of Computer Science, Western University, London, ON, Canada
- Western Institute for Neuroscience, Western University, London, ON, Canada
| | - Alexis Thual
- Inria, CEA, Université Paris-Saclay, 91120, Palaiseau, France
- Inserm, Collège de France, Paris, France
| |
Collapse
|
24
|
Chen C, Dupré la Tour T, Gallant JL, Klein D, Deniz F. The Cortical Representation of Language Timescales is Shared between Reading and Listening. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2023:2023.01.06.522601. [PMID: 37577530 PMCID: PMC10418083 DOI: 10.1101/2023.01.06.522601] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 08/15/2023]
Abstract
Language comprehension involves integrating low-level sensory inputs into a hierarchy of increasingly high-level features. Prior work studied brain representations of different levels of the language hierarchy, but has not determined whether these brain representations are shared between written and spoken language. To address this issue, we analyzed fMRI BOLD data recorded while participants read and listened to the same narratives in each modality. Levels of the language hierarchy were operationalized as timescales, where each timescale refers to a set of spectral components of a language stimulus. Voxelwise encoding models were used to determine where different timescales are represented across the cerebral cortex, for each modality separately. These models reveal that between the two modalities timescale representations are organized similarly across the cortical surface. Our results suggest that, after low-level sensory processing, language integration proceeds similarly regardless of stimulus modality.
Collapse
Affiliation(s)
- Catherine Chen
- Department of Electrical Engineering and Computer Sciences, University of California, Berkeley, USA
| | - Tom Dupré la Tour
- Helen Wills Neuroscience Institute, University of California, Berkeley, USA
| | - Jack L. Gallant
- Helen Wills Neuroscience Institute, University of California, Berkeley, USA
| | - Dan Klein
- Department of Electrical Engineering and Computer Sciences, University of California, Berkeley, USA
| | - Fatma Deniz
- Helen Wills Neuroscience Institute, University of California, Berkeley, USA
- Institute of Software Engineering and Theoretical Computer Science, Technische Universität Berlin, Berlin, Germany
- Bernstein Center for Computational Neuroscience, Berlin, Germany
| |
Collapse
|
25
|
Tang J, Du M, Vo VA, Lal V, Huth AG. Brain encoding models based on multimodal transformers can transfer across language and vision. ADVANCES IN NEURAL INFORMATION PROCESSING SYSTEMS 2023; 36:29654-29666. [PMID: 39015152 PMCID: PMC11250991] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Subscribe] [Scholar Register] [Indexed: 07/18/2024]
Abstract
Encoding models have been used to assess how the human brain represents concepts in language and vision. While language and vision rely on similar concept representations, current encoding models are typically trained and tested on brain responses to each modality in isolation. Recent advances in multimodal pretraining have produced transformers that can extract aligned representations of concepts in language and vision. In this work, we used representations from multimodal transformers to train encoding models that can transfer across fMRI responses to stories and movies. We found that encoding models trained on brain responses to one modality can successfully predict brain responses to the other modality, particularly in cortical regions that represent conceptual meaning. Further analysis of these encoding models revealed shared semantic dimensions that underlie concept representations in language and vision. Comparing encoding models trained using representations from multimodal and unimodal transformers, we found that multimodal transformers learn more aligned representations of concepts in language and vision. Our results demonstrate how multimodal transformers can provide insights into the brain's capacity for multimodal processing.
Collapse
|
26
|
Maruya A, Zaidi Q. Perceptual Transitions between Object Rigidity & Non-rigidity: Competition and cooperation between motion-energy, feature-tracking and shape-based priors. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2023:2023.04.07.536067. [PMID: 37503257 PMCID: PMC10369874 DOI: 10.1101/2023.04.07.536067] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 07/29/2023]
Abstract
Why do moving objects appear rigid when projected retinal images are deformed non-rigidly? We used rotating rigid objects that can appear rigid or non-rigid to test whether shape features contribute to rigidity perception. When two circular rings were rigidly linked at an angle and jointly rotated at moderate speeds, observers reported that the rings wobbled and were not linked rigidly but rigid rotation was reported at slow speeds. When gaps, paint or vertices were added, the rings appeared rigidly rotating even at moderate speeds. At high speeds, all configurations appeared non-rigid. Salient features thus contribute to rigidity at slow and moderate speeds, but not at high speeds. Simulated responses of arrays of motion-energy cells showed that motion flow vectors are predominantly orthogonal to the contours of the rings, not parallel to the rotation direction. A convolutional neural network trained to distinguish flow patterns for wobbling versus rotation, gave a high probability of wobbling for the motion-energy flows. However, the CNN gave high probabilities of rotation for motion flows generated by tracking features with arrays of MT pattern-motion cells and corner detectors. In addition, circular rings can appear to spin and roll despite the absence of any sensory evidence, and this illusion is prevented by vertices, gaps, and painted segments, showing the effects of rotational symmetry and shape. Combining CNN outputs that give greater weight to motion energy at fast speeds and to feature tracking at slow, with the shape-based priors for wobbling and rolling, explained rigid and nonrigid percepts across shapes and speeds (R2=0.95). The results demonstrate how cooperation and competition between different neuronal classes leads to specific states of visual perception and to transitions between the states.
Collapse
Affiliation(s)
- Akihito Maruya
- Graduate Center for Vision Research, State University of New York, 33 West 42nd St, New York, NY 10036
| | - Qasim Zaidi
- Graduate Center for Vision Research, State University of New York, 33 West 42nd St, New York, NY 10036
| |
Collapse
|
27
|
Morgenroth E, Vilaclara L, Muszynski M, Gaviria J, Vuilleumier P, Van De Ville D. Probing neurodynamics of experienced emotions-a Hitchhiker's guide to film fMRI. Soc Cogn Affect Neurosci 2023; 18:nsad063. [PMID: 37930850 PMCID: PMC10656947 DOI: 10.1093/scan/nsad063] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/16/2023] [Revised: 08/04/2023] [Accepted: 11/01/2023] [Indexed: 11/08/2023] Open
Abstract
Film functional magnetic resonance imaging (fMRI) has gained tremendous popularity in many areas of neuroscience. However, affective neuroscience remains somewhat behind in embracing this approach, even though films lend themselves to study how brain function gives rise to complex, dynamic and multivariate emotions. Here, we discuss the unique capabilities of film fMRI for emotion research, while providing a general guide of conducting such research. We first give a brief overview of emotion theories as these inform important design choices. Next, we discuss films as experimental paradigms for emotion elicitation and address the process of annotating them. We then situate film fMRI in the context of other fMRI approaches, and present an overview of results from extant studies so far with regard to advantages of film fMRI. We also give an overview of state-of-the-art analysis techniques including methods that probe neurodynamics. Finally, we convey limitations of using film fMRI to study emotion. In sum, this review offers a practitioners' guide to the emerging field of film fMRI and underscores how it can advance affective neuroscience.
Collapse
Affiliation(s)
- Elenor Morgenroth
- Neuro-X Institute, École Polytechnique Fédérale de Lausanne, Geneva 1202, Switzerland
- Department of Radiology and Medical Informatics, University of Geneva, Geneva 1202, Switzerland
- Swiss Center for Affective Sciences, University of Geneva, Geneva 1202, Switzerland
| | - Laura Vilaclara
- Neuro-X Institute, École Polytechnique Fédérale de Lausanne, Geneva 1202, Switzerland
- Department of Radiology and Medical Informatics, University of Geneva, Geneva 1202, Switzerland
| | - Michal Muszynski
- Department of Basic Neurosciences, University of Geneva, Geneva 1202, Switzerland
| | - Julian Gaviria
- Swiss Center for Affective Sciences, University of Geneva, Geneva 1202, Switzerland
- Department of Basic Neurosciences, University of Geneva, Geneva 1202, Switzerland
- Department of Psychiatry, University of Geneva, Geneva 1202, Switzerland
| | - Patrik Vuilleumier
- Swiss Center for Affective Sciences, University of Geneva, Geneva 1202, Switzerland
- Department of Basic Neurosciences, University of Geneva, Geneva 1202, Switzerland
- CIBM Center for Biomedical Imaging, Geneva 1202, Switzerland
| | - Dimitri Van De Ville
- Neuro-X Institute, École Polytechnique Fédérale de Lausanne, Geneva 1202, Switzerland
- Department of Radiology and Medical Informatics, University of Geneva, Geneva 1202, Switzerland
- CIBM Center for Biomedical Imaging, Geneva 1202, Switzerland
| |
Collapse
|
28
|
Levin M. Bioelectric networks: the cognitive glue enabling evolutionary scaling from physiology to mind. Anim Cogn 2023; 26:1865-1891. [PMID: 37204591 PMCID: PMC10770221 DOI: 10.1007/s10071-023-01780-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/28/2022] [Revised: 04/12/2023] [Accepted: 04/24/2023] [Indexed: 05/20/2023]
Abstract
Each of us made the remarkable journey from mere matter to mind: starting life as a quiescent oocyte ("just chemistry and physics"), and slowly, gradually, becoming an adult human with complex metacognitive processes, hopes, and dreams. In addition, even though we feel ourselves to be a unified, single Self, distinct from the emergent dynamics of termite mounds and other swarms, the reality is that all intelligence is collective intelligence: each of us consists of a huge number of cells working together to generate a coherent cognitive being with goals, preferences, and memories that belong to the whole and not to its parts. Basal cognition is the quest to understand how Mind scales-how large numbers of competent subunits can work together to become intelligences that expand the scale of their possible goals. Crucially, the remarkable trick of turning homeostatic, cell-level physiological competencies into large-scale behavioral intelligences is not limited to the electrical dynamics of the brain. Evolution was using bioelectric signaling long before neurons and muscles appeared, to solve the problem of creating and repairing complex bodies. In this Perspective, I review the deep symmetry between the intelligence of developmental morphogenesis and that of classical behavior. I describe the highly conserved mechanisms that enable the collective intelligence of cells to implement regulative embryogenesis, regeneration, and cancer suppression. I sketch the story of an evolutionary pivot that repurposed the algorithms and cellular machinery that enable navigation of morphospace into the behavioral navigation of the 3D world which we so readily recognize as intelligence. Understanding the bioelectric dynamics that underlie construction of complex bodies and brains provides an essential path to understanding the natural evolution, and bioengineered design, of diverse intelligences within and beyond the phylogenetic history of Earth.
Collapse
Affiliation(s)
- Michael Levin
- Allen Discovery Center at Tufts University, 200 Boston Ave., Suite 4600, Medford, MA, 02155, USA.
- Wyss Institute for Biologically Inspired Engineering at Harvard University, Boston, MA, USA.
| |
Collapse
|
29
|
Purandare C, Mehta M. Mega-scale movie-fields in the mouse visuo-hippocampal network. eLife 2023; 12:RP85069. [PMID: 37910428 PMCID: PMC10619982 DOI: 10.7554/elife.85069] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/03/2023] Open
Abstract
Natural visual experience involves a continuous series of related images while the subject is immobile. How does the cortico-hippocampal circuit process a visual episode? The hippocampus is crucial for episodic memory, but most rodent single unit studies require spatial exploration or active engagement. Hence, we investigated neural responses to a silent movie (Allen Brain Observatory) in head-fixed mice without any task or locomotion demands, or rewards. Surprisingly, a third (33%, 3379/10263) of hippocampal -dentate gyrus, CA3, CA1 and subiculum- neurons showed movie-selectivity, with elevated firing in specific movie sub-segments, termed movie-fields, similar to the vast majority of thalamo-cortical (LGN, V1, AM-PM) neurons (97%, 6554/6785). Movie-tuning remained intact in immobile or spontaneously running mice. Visual neurons had >5 movie-fields per cell, but only ~2 in hippocampus. The movie-field durations in all brain regions spanned an unprecedented 1000-fold range: from 0.02s to 20s, termed mega-scale coding. Yet, the total duration of all the movie-fields of a cell was comparable across neurons and brain regions. The hippocampal responses thus showed greater continuous-sequence encoding than visual areas, as evidenced by fewer and broader movie-fields than in visual areas. Consistently, repeated presentation of the movie images in a fixed, but scrambled sequence virtually abolished hippocampal but not visual-cortical selectivity. The preference for continuous, compared to scrambled sequence was eight-fold greater in hippocampal than visual areas, further supporting episodic-sequence encoding. Movies could thus provide a unified way to probe neural mechanisms of episodic information processing and memory, even in immobile subjects, across brain regions, and species.
Collapse
Affiliation(s)
- Chinmay Purandare
- Department of Bioengineering, University of California, Los AngelesLos AngelesUnited States
- W.M. Keck Center for Neurophysics, Department of Physics and Astronomy, University of California, Los AngelesLos AngelesUnited States
- Department of Neurology, University of California, Los AngelesLos AngelesUnited States
| | - Mayank Mehta
- W.M. Keck Center for Neurophysics, Department of Physics and Astronomy, University of California, Los AngelesLos AngelesUnited States
- Department of Neurology, University of California, Los AngelesLos AngelesUnited States
- Department of Electrical and Computer Engineering, University of California, Los AngelesLos AngelesUnited States
| |
Collapse
|
30
|
Du C, Fu K, Wen B, He H. Topographic representation of visually evoked emotional experiences in the human cerebral cortex. iScience 2023; 26:107571. [PMID: 37664621 PMCID: PMC10470388 DOI: 10.1016/j.isci.2023.107571] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/10/2023] [Revised: 07/03/2023] [Accepted: 08/07/2023] [Indexed: 09/05/2023] Open
Abstract
Affective neuroscience seeks to uncover the neural underpinnings of emotions that humans experience. However, it remains unclear whether an affective space underlies the discrete emotion categories in the human brain, and how it relates to the hypothesized affective dimensions. To address this question, we developed a voxel-wise encoding model to investigate the cortical organization of human emotions. Results revealed that the distributed emotion representations are constructed through a fundamental affective space. We further compared each dimension of this space to 14 hypothesized affective dimensions, and found that many affective dimensions are captured by the fundamental affective space. Our results suggest that emotional experiences are represented by broadly spatial overlapping cortical patterns and form smooth gradients across large areas of the cortex. This finding reveals the specific structure of the affective space and its relationship to hypothesized affective dimensions, while highlighting the distributed nature of emotional representations in the cortex.
Collapse
Affiliation(s)
- Changde Du
- Laboratory of Brain Atlas and Brain-Inspired Intelligence, State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Science, Beijing 100190, China
| | - Kaicheng Fu
- Laboratory of Brain Atlas and Brain-Inspired Intelligence, State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Science, Beijing 100190, China
- School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, China
| | - Bincheng Wen
- Center for Excellence in Brain Science and Intelligence Technology, Key Laboratory of Primate Neurobiology, Institute of Neuroscience, Chinese Academy of Sciences, Shanghai 200031, China
- School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, China
| | - Huiguang He
- Laboratory of Brain Atlas and Brain-Inspired Intelligence, State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Science, Beijing 100190, China
- School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, China
| |
Collapse
|
31
|
Ren Z, Li J, Xue X, Li X, Yang F, Jiao Z, Gao X. Reconstructing controllable faces from brain activity with hierarchical multiview representations. Neural Netw 2023; 166:487-500. [PMID: 37574622 DOI: 10.1016/j.neunet.2023.07.016] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/04/2022] [Revised: 05/21/2023] [Accepted: 07/12/2023] [Indexed: 08/15/2023]
Abstract
Reconstructing visual experience from brain responses measured by functional magnetic resonance imaging (fMRI) is a challenging yet important research topic in brain decoding, especially it has proved more difficult to decode visually similar stimuli, such as faces. Although face attributes are known as the key to face recognition, most existing methods generally ignore how to decode facial attributes more precisely in perceived face reconstruction, which often leads to indistinguishable reconstructed faces. To solve this problem, we propose a novel neural decoding framework called VSPnet (voxel2style2pixel) by establishing hierarchical encoding and decoding networks with disentangled latent representations as media, so that to recover visual stimuli more elaborately. And we design a hierarchical visual encoder (named HVE) to pre-extract features containing both high-level semantic knowledge and low-level visual details from stimuli. The proposed VSPnet consists of two networks: Multi-branch cognitive encoder and style-based image generator. The encoder network is constructed by multiple linear regression branches to map brain signals to the latent space provided by the pre-extracted visual features and obtain representations containing hierarchical information consistent to the corresponding stimuli. We make the generator network inspired by StyleGAN to untangle the complexity of fMRI representations and generate images. And the HVE network is composed of a standard feature pyramid over a ResNet backbone. Extensive experimental results on the latest public datasets have demonstrated the reconstruction accuracy of our proposed method outperforms the state-of-the-art approaches and the identifiability of different reconstructed faces has been greatly improved. In particular, we achieve feature editing for several facial attributes in fMRI domain based on the multiview (i.e., visual stimuli and evoked fMRI) latent representations.
Collapse
Affiliation(s)
- Ziqi Ren
- School of Electronic Engineering, Xidian University, Xi'an 710071, China
| | - Jie Li
- School of Electronic Engineering, Xidian University, Xi'an 710071, China
| | - Xuetong Xue
- School of Electronic Engineering, Xidian University, Xi'an 710071, China
| | - Xin Li
- Group 42 (G42), Abu Dhabi, United Arab Emirates
| | - Fan Yang
- Group 42 (G42), Abu Dhabi, United Arab Emirates
| | - Zhicheng Jiao
- The Warren Alpert Medical School, Brown University, RI, USA; Department of Diagnostic Imaging, Rhode Island Hospital, RI, USA
| | - Xinbo Gao
- School of Electronic Engineering, Xidian University, Xi'an 710071, China.
| |
Collapse
|
32
|
Zhao Y, Chen Y, Cheng K, Huang W. Artificial intelligence based multimodal language decoding from brain activity: A review. Brain Res Bull 2023; 201:110713. [PMID: 37487829 DOI: 10.1016/j.brainresbull.2023.110713] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/27/2023] [Revised: 06/26/2023] [Accepted: 07/20/2023] [Indexed: 07/26/2023]
Abstract
Decoding brain activity is conducive to the breakthrough of brain-computer interface (BCI) technology. The development of artificial intelligence (AI) continually promotes the progress of brain language decoding technology. Existent research has mainly focused on a single modality and paid insufficient attention to AI methods. Therefore, our objective is to provide an overview of relevant decoding research from the perspective of different modalities and methodologies. The modalities involve text, speech, image, and video, whereas the core method is using AI-built decoders to translate brain signals induced by multimodal stimuli into text or vocal language. The semantic information of brain activity can be successfully decoded into a language at various levels, ranging from words through sentences to discourses. However, the decoding effect is affected by various factors, such as the decoding model, vector representation model, and brain regions. Challenges and future directions are also discussed. The advances in brain language decoding and BCI technology will potentially assist patients with clinical aphasia in regaining the ability to communicate.
Collapse
Affiliation(s)
- Yuhao Zhao
- College of Language Intelligence, Sichuan International Studies University, Chongqing 400031, PR China
| | - Yu Chen
- Technical College for the Deaf, Tianjin University of Technology, Tianjin 300384, PR China
| | - Kaiwen Cheng
- College of Language Intelligence, Sichuan International Studies University, Chongqing 400031, PR China.
| | - Wei Huang
- Sichuan Provincial Key Laboratory for Human Disease Gene Study, Sichuan Provincial People's Hospital, University of Electronic Science and Technology of China, Chengdu 611731, PR China.
| |
Collapse
|
33
|
Wang C, Yan H, Huang W, Sheng W, Wang Y, Fan YS, Liu T, Zou T, Li R, Chen H. Neural encoding with unsupervised spiking convolutional neural network. Commun Biol 2023; 6:880. [PMID: 37640808 PMCID: PMC10462614 DOI: 10.1038/s42003-023-05257-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/06/2023] [Accepted: 08/18/2023] [Indexed: 08/31/2023] Open
Abstract
Accurately predicting the brain responses to various stimuli poses a significant challenge in neuroscience. Despite recent breakthroughs in neural encoding using convolutional neural networks (CNNs) in fMRI studies, there remain critical gaps between the computational rules of traditional artificial neurons and real biological neurons. To address this issue, a spiking CNN (SCNN)-based framework is presented in this study to achieve neural encoding in a more biologically plausible manner. The framework utilizes unsupervised SCNN to extract visual features of image stimuli and employs a receptive field-based regression algorithm to predict fMRI responses from the SCNN features. Experimental results on handwritten characters, handwritten digits and natural images demonstrate that the proposed approach can achieve remarkably good encoding performance and can be utilized for "brain reading" tasks such as image reconstruction and identification. This work suggests that SNN can serve as a promising tool for neural encoding.
Collapse
Affiliation(s)
- Chong Wang
- The Center of Psychosomatic Medicine, Sichuan Provincial Center for Mental Health, Sichuan Provincial People's Hospital, University of Electronic Science and Technology of China, Chengdu, 611731, China
- School of Life Science and Technology, University of Electronic Science and Technology of China, Chengdu, 610054, China
- MOE Key Lab for Neuroinformation; High-Field Magnetic Resonance Brain Imaging Key Laboratory of Sichuan Province, University of Electronic Science and Technology of China, Chengdu, 610054, China
| | - Hongmei Yan
- School of Life Science and Technology, University of Electronic Science and Technology of China, Chengdu, 610054, China.
- MOE Key Lab for Neuroinformation; High-Field Magnetic Resonance Brain Imaging Key Laboratory of Sichuan Province, University of Electronic Science and Technology of China, Chengdu, 610054, China.
| | - Wei Huang
- School of Life Science and Technology, University of Electronic Science and Technology of China, Chengdu, 610054, China
- MOE Key Lab for Neuroinformation; High-Field Magnetic Resonance Brain Imaging Key Laboratory of Sichuan Province, University of Electronic Science and Technology of China, Chengdu, 610054, China
| | - Wei Sheng
- School of Life Science and Technology, University of Electronic Science and Technology of China, Chengdu, 610054, China
- MOE Key Lab for Neuroinformation; High-Field Magnetic Resonance Brain Imaging Key Laboratory of Sichuan Province, University of Electronic Science and Technology of China, Chengdu, 610054, China
| | - Yuting Wang
- School of Life Science and Technology, University of Electronic Science and Technology of China, Chengdu, 610054, China
- MOE Key Lab for Neuroinformation; High-Field Magnetic Resonance Brain Imaging Key Laboratory of Sichuan Province, University of Electronic Science and Technology of China, Chengdu, 610054, China
| | - Yun-Shuang Fan
- School of Life Science and Technology, University of Electronic Science and Technology of China, Chengdu, 610054, China
- MOE Key Lab for Neuroinformation; High-Field Magnetic Resonance Brain Imaging Key Laboratory of Sichuan Province, University of Electronic Science and Technology of China, Chengdu, 610054, China
| | - Tao Liu
- School of Life Science and Technology, University of Electronic Science and Technology of China, Chengdu, 610054, China
| | - Ting Zou
- School of Life Science and Technology, University of Electronic Science and Technology of China, Chengdu, 610054, China
| | - Rong Li
- School of Life Science and Technology, University of Electronic Science and Technology of China, Chengdu, 610054, China.
- MOE Key Lab for Neuroinformation; High-Field Magnetic Resonance Brain Imaging Key Laboratory of Sichuan Province, University of Electronic Science and Technology of China, Chengdu, 610054, China.
| | - Huafu Chen
- The Center of Psychosomatic Medicine, Sichuan Provincial Center for Mental Health, Sichuan Provincial People's Hospital, University of Electronic Science and Technology of China, Chengdu, 611731, China.
- School of Life Science and Technology, University of Electronic Science and Technology of China, Chengdu, 610054, China.
- MOE Key Lab for Neuroinformation; High-Field Magnetic Resonance Brain Imaging Key Laboratory of Sichuan Province, University of Electronic Science and Technology of China, Chengdu, 610054, China.
| |
Collapse
|
34
|
LeBel A, Wagner L, Jain S, Adhikari-Desai A, Gupta B, Morgenthal A, Tang J, Xu L, Huth AG. A natural language fMRI dataset for voxelwise encoding models. Sci Data 2023; 10:555. [PMID: 37612332 PMCID: PMC10447563 DOI: 10.1038/s41597-023-02437-z] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/26/2022] [Accepted: 08/02/2023] [Indexed: 08/25/2023] Open
Abstract
Speech comprehension is a complex process that draws on humans' abilities to extract lexical information, parse syntax, and form semantic understanding. These sub-processes have traditionally been studied using separate neuroimaging experiments that attempt to isolate specific effects of interest. More recently it has become possible to study all stages of language comprehension in a single neuroimaging experiment using narrative natural language stimuli. The resulting data are richly varied at every level, enabling analyses that can probe everything from spectral representations to high-level representations of semantic meaning. We provide a dataset containing BOLD fMRI responses recorded while 8 participants each listened to 27 complete, natural, narrative stories (~6 hours). This dataset includes pre-processed and raw MRIs, as well as hand-constructed 3D cortical surfaces for each participant. To address the challenges of analyzing naturalistic data, this dataset is accompanied by a python library containing basic code for creating voxelwise encoding models. Altogether, this dataset provides a large and novel resource for understanding speech and language processing in the human brain.
Collapse
Affiliation(s)
- Amanda LeBel
- Helen Wills Neuroscience Institute, University of California, Berkeley, Berkeley, CA, 94704, USA
| | - Lauren Wagner
- Department of Psychiatry and Biobehavioral Sciences, University of California, Los Angeles, CA, 90095, USA
| | - Shailee Jain
- Department of Computer Science, The University of Texas at Austin, Austin, TX, 78712, USA
| | - Aneesh Adhikari-Desai
- Department of Computer Science, The University of Texas at Austin, Austin, TX, 78712, USA
- Department of Neuroscience, The University of Texas at Austin, Austin, TX, 78712, USA
| | - Bhavin Gupta
- Department of Computer Science, The University of Texas at Austin, Austin, TX, 78712, USA
| | - Allyson Morgenthal
- Department of Neuroscience, The University of Texas at Austin, Austin, TX, 78712, USA
| | - Jerry Tang
- Department of Computer Science, The University of Texas at Austin, Austin, TX, 78712, USA
| | - Lixiang Xu
- Department of Physics, The University of Texas at Austin, Austin, TX, 78712, USA
| | - Alexander G Huth
- Department of Computer Science, The University of Texas at Austin, Austin, TX, 78712, USA.
- Department of Neuroscience, The University of Texas at Austin, Austin, TX, 78712, USA.
| |
Collapse
|
35
|
Mizrahi T, Axelrod V. Naturalistic auditory stimuli with fNIRS prefrontal cortex imaging: A potential paradigm for disorder of consciousness diagnostics (a study with healthy participants). Neuropsychologia 2023; 187:108604. [PMID: 37271305 DOI: 10.1016/j.neuropsychologia.2023.108604] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/30/2023] [Revised: 06/01/2023] [Accepted: 06/02/2023] [Indexed: 06/06/2023]
Abstract
Disorder of consciousness (DOC) is a devastating condition due to brain damage. A patient in this condition is non-responsive, but nevertheless might be conscious at least at some level. Determining the conscious level of DOC patients is important for both medical and ethical reasons, but reliably achieving this has been a major challenge. Naturalistic stimuli in combination with neuroimaging have been proposed as a promising approach for DOC patient diagnosis. Capitalizing on and extending this proposal, the goal of the present study conducted with healthy participants was to develop a new paradigm with naturalistic auditory stimuli and functional near-infrared spectroscopy (fNIRS) - an approach that can be used at the bedside. Twenty-four healthy participants passively listened to 9 min of auditory story, scrambled auditory story, classical music, and scrambled classical music segments while their prefrontal cortex activity was recorded using fNIRS. We found much higher intersubject correlation (ISC) during story compared to scrambled story conditions both at the group level and in the majority of individual subjects, suggesting that fNIRS imaging of the prefrontal cortex might be a sensitive method to capture neural changes associated with narrative comprehension. In contrast, the ISC during the classical music segment did not differ reliably from scrambled classical music and was also much lower than the story condition. Our main result is that naturalistic auditory stories with fNIRS might be used in a clinical setup to identify high-level processing and potential consciousness in DOC patients.
Collapse
Affiliation(s)
- Tamar Mizrahi
- The Gonda Multidisciplinary Brain Research Center, Bar Ilan University, Ramat Gan, Israel; Head Injuries Rehabilitation Department, Sheba Medical Center, Ramat Gan, Israel
| | - Vadim Axelrod
- The Gonda Multidisciplinary Brain Research Center, Bar Ilan University, Ramat Gan, Israel.
| |
Collapse
|
36
|
Markow ZE, Trobaugh JW, Richter EJ, Tripathy K, Rafferty SM, Svoboda AM, Schroeder ML, Burns-Yocum TM, Bergonzi KM, Chevillet MA, Mugler EM, Eggebrecht AT, Culver JP. Ultra-high density imaging arrays for diffuse optical tomography of human brain improve resolution, signal-to-noise, and information decoding. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2023:2023.07.21.549920. [PMID: 37547013 PMCID: PMC10401969 DOI: 10.1101/2023.07.21.549920] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 08/08/2023]
Abstract
Functional magnetic resonance imaging (fMRI) has dramatically advanced non-invasive human brain mapping and decoding. Functional near-infrared spectroscopy (fNIRS) and high-density diffuse optical tomography (HD-DOT) non-invasively measure blood oxygen fluctuations related to brain activity, like fMRI, at the brain surface, using more-lightweight equipment that circumvents ergonomic and logistical limitations of fMRI. HD-DOT grids have smaller inter-optode spacing (∼13 mm) than sparse fNIRS (∼30 mm) and therefore provide higher image quality, with spatial resolution ∼1/2 that of fMRI. Herein, simulations indicated reducing inter-optode spacing to 6.5 mm would further improve image quality and noise-resolution tradeoff, with diminishing returns below 6.5 mm. We then constructed an ultra-high-density DOT system (6.5-mm spacing) with 140 dB dynamic range that imaged stimulus-evoked activations with 30-50% higher spatial resolution and repeatable multi-focal activity with excellent agreement with participant-matched fMRI. Further, this system decoded visual stimulus position with 19-35% lower error than previous HD-DOT, throughout occipital cortex.
Collapse
|
37
|
Meschke EX, Castello MVDO, la Tour TD, Gallant JL. Model connectivity: leveraging the power of encoding models to overcome the limitations of functional connectivity. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2023:2023.07.17.549356. [PMID: 37503232 PMCID: PMC10370105 DOI: 10.1101/2023.07.17.549356] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 07/29/2023]
Abstract
Functional connectivity (FC) is the most popular method for recovering functional networks of brain areas with fMRI. However, because FC is defined as temporal correlations in brain activity, FC networks are confounded by noise and lack a precise functional role. To overcome these limitations, we developed model connectivity (MC). MC is defined as similarities in encoding model weights, which quantify reliable functional activity in terms of interpretable stimulus- or task-related features. To compare FC and MC, both methods were applied to a naturalistic story listening dataset. FC recovered spatially broad networks that are confounded by noise, and that lack a clear role during natural language comprehension. By contrast, MC recovered spatially localized networks that are robust to noise, and that represent distinct categories of semantic concepts. Thus, MC is a powerful data-driven approach for recovering and interpreting the functional networks that support complex cognitive processes.
Collapse
|
38
|
Marrazzo G, De Martino F, Lage-Castellanos A, Vaessen MJ, de Gelder B. Voxelwise encoding models of body stimuli reveal a representational gradient from low-level visual features to postural features in occipitotemporal cortex. Neuroimage 2023:120240. [PMID: 37348622 DOI: 10.1016/j.neuroimage.2023.120240] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/14/2023] [Revised: 06/16/2023] [Accepted: 06/19/2023] [Indexed: 06/24/2023] Open
Abstract
Research on body representation in the brain has focused on category-specific representation, using fMRI to investigate the response pattern to body stimuli in occipitotemporal cortex without so far addressing the issue of the specific computations involved in body selective regions, only defined by higher order category selectivity. This study used ultra-high field fMRI and banded ridge regression to investigate the coding of body images, by comparing the performance of three encoding models in predicting brain activity in occipitotemporal cortex and specifically the extrastriate body area (EBA). Our results suggest that bodies are encoded in occipitotemporal cortex and in the EBA according to a combination of low-level visual features and postural features.
Collapse
Affiliation(s)
- Giuseppe Marrazzo
- Department of Cognitive Neuroscience, Faculty of Psychology and Neuroscience, Maastricht University, Limburg 6200 MD, Maastricht, The Netherlands
| | - Federico De Martino
- Department of Cognitive Neuroscience, Faculty of Psychology and Neuroscience, Maastricht University, Limburg 6200 MD, Maastricht, The Netherlands; Center for Magnetic Resonance Research, Department of Radiology, University of Minnesota, Minneapolis, MN 55455, United States and Department of NeuroInformatics
| | - Agustin Lage-Castellanos
- Department of Cognitive Neuroscience, Faculty of Psychology and Neuroscience, Maastricht University, Limburg 6200 MD, Maastricht, The Netherlands; Cuban Center for Neuroscience, Street 190 e/25 and 27 Cubanacán Playa Havana, CP 11600, Cuba
| | - Maarten J Vaessen
- Department of Cognitive Neuroscience, Faculty of Psychology and Neuroscience, Maastricht University, Limburg 6200 MD, Maastricht, The Netherlands
| | - Beatrice de Gelder
- Department of Cognitive Neuroscience, Faculty of Psychology and Neuroscience, Maastricht University, Limburg 6200 MD, Maastricht, The Netherlands.
| |
Collapse
|
39
|
Bedel HA, Sivgin I, Dalmaz O, Dar SUH, Çukur T. BolT: Fused window transformers for fMRI time series analysis. Med Image Anal 2023; 88:102841. [PMID: 37224718 DOI: 10.1016/j.media.2023.102841] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/19/2022] [Revised: 02/07/2023] [Accepted: 05/10/2023] [Indexed: 05/26/2023]
Abstract
Deep-learning models have enabled performance leaps in analysis of high-dimensional functional MRI (fMRI) data. Yet, many previous methods are suboptimally sensitive for contextual representations across diverse time scales. Here, we present BolT, a blood-oxygen-level-dependent transformer model, for analyzing multi-variate fMRI time series. BolT leverages a cascade of transformer encoders equipped with a novel fused window attention mechanism. Encoding is performed on temporally-overlapped windows within the time series to capture local representations. To integrate information temporally, cross-window attention is computed between base tokens in each window and fringe tokens from neighboring windows. To gradually transition from local to global representations, the extent of window overlap and thereby number of fringe tokens are progressively increased across the cascade. Finally, a novel cross-window regularization is employed to align high-level classification features across the time series. Comprehensive experiments on large-scale public datasets demonstrate the superior performance of BolT against state-of-the-art methods. Furthermore, explanatory analyses to identify landmark time points and regions that contribute most significantly to model decisions corroborate prominent neuroscientific findings in the literature.
Collapse
Affiliation(s)
- Hasan A Bedel
- Department of Electrical and Electronics Engineering, Bilkent University, Ankara 06800, Turkey; National Magnetic Resonance Research Center (UMRAM), Bilkent University, Ankara 06800, Turkey
| | - Irmak Sivgin
- Department of Electrical and Electronics Engineering, Bilkent University, Ankara 06800, Turkey; National Magnetic Resonance Research Center (UMRAM), Bilkent University, Ankara 06800, Turkey
| | - Onat Dalmaz
- Department of Electrical and Electronics Engineering, Bilkent University, Ankara 06800, Turkey; National Magnetic Resonance Research Center (UMRAM), Bilkent University, Ankara 06800, Turkey
| | - Salman U H Dar
- Department of Electrical and Electronics Engineering, Bilkent University, Ankara 06800, Turkey; National Magnetic Resonance Research Center (UMRAM), Bilkent University, Ankara 06800, Turkey
| | - Tolga Çukur
- Department of Electrical and Electronics Engineering, Bilkent University, Ankara 06800, Turkey; National Magnetic Resonance Research Center (UMRAM), Bilkent University, Ankara 06800, Turkey; Neuroscience Program, Bilkent University, Ankara 06800, Turkey.
| |
Collapse
|
40
|
Kim I, Kupers ER, Lerma-Usabiaga G, Grill-Spector K. Characterizing spatiotemporal population receptive fields in human visual cortex with fMRI. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2023:2023.05.02.539164. [PMID: 37205541 PMCID: PMC10187260 DOI: 10.1101/2023.05.02.539164] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/21/2023]
Abstract
The use of fMRI and computational modeling has advanced understanding of spatial characteristics of population receptive fields (pRFs) in human visual cortex. However, we know relatively little about the spatiotemporal characteristics of pRFs because neurons' temporal properties are one to two orders of magnitude faster than fMRI BOLD responses. Here, we developed an image-computable framework to estimate spatiotemporal pRFs from fMRI data. First, we developed a simulation software that predicts fMRI responses to a time varying visual input given a spatiotemporal pRF model and solves the model parameters. The simulator revealed that ground-truth spatiotemporal parameters can be accurately recovered at the millisecond resolution from synthesized fMRI responses. Then, using fMRI and a novel stimulus paradigm, we mapped spatiotemporal pRFs in individual voxels across human visual cortex in 10 participants. We find that a compressive spatiotemporal (CST) pRF model better explains fMRI responses than a conventional spatial pRF model across visual areas spanning the dorsal, lateral, and ventral streams. Further, we find three organizational principles of spatiotemporal pRFs: (i) from early to later areas within a visual stream, spatial and temporal integration windows of pRFs progressively increase in size and show greater compressive nonlinearities, (ii) later visual areas show diverging spatial and temporal integration windows across streams, and (iii) within early visual areas (V1-V3), both spatial and temporal integration windows systematically increase with eccentricity. Together, this computational framework and empirical results open exciting new possibilities for modeling and measuring fine-grained spatiotemporal dynamics of neural responses in the human brain using fMRI.
Collapse
Affiliation(s)
- Insub Kim
- Department of Psychology, Stanford University, Stanford, CA, USA
| | - Eline R. Kupers
- Department of Psychology, Stanford University, Stanford, CA, USA
| | - Garikoitz Lerma-Usabiaga
- BCBL. Basque Center on Cognition, Brain and Language, San Sebastian, Spain
- IKERBASQUE. Basque foundation for science, Bilbao, Spain
| | - Kalanit Grill-Spector
- Department of Psychology, Stanford University, Stanford, CA, USA
- Wu Tsai Neurosciences Institute, Stanford University, Stanford, CA, USA
| |
Collapse
|
41
|
Tang J, LeBel A, Jain S, Huth AG. Semantic reconstruction of continuous language from non-invasive brain recordings. Nat Neurosci 2023; 26:858-866. [PMID: 37127759 PMCID: PMC11304553 DOI: 10.1038/s41593-023-01304-9] [Citation(s) in RCA: 44] [Impact Index Per Article: 44.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/01/2022] [Accepted: 03/15/2023] [Indexed: 05/03/2023]
Abstract
A brain-computer interface that decodes continuous language from non-invasive recordings would have many scientific and practical applications. Currently, however, non-invasive language decoders can only identify stimuli from among a small set of words or phrases. Here we introduce a non-invasive decoder that reconstructs continuous language from cortical semantic representations recorded using functional magnetic resonance imaging (fMRI). Given novel brain recordings, this decoder generates intelligible word sequences that recover the meaning of perceived speech, imagined speech and even silent videos, demonstrating that a single decoder can be applied to a range of tasks. We tested the decoder across cortex and found that continuous language can be separately decoded from multiple regions. As brain-computer interfaces should respect mental privacy, we tested whether successful decoding requires subject cooperation and found that subject cooperation is required both to train and to apply the decoder. Our findings demonstrate the viability of non-invasive language brain-computer interfaces.
Collapse
Affiliation(s)
- Jerry Tang
- Department of Computer Science, The University of Texas at Austin, Austin, TX, USA
| | - Amanda LeBel
- Department of Neuroscience, The University of Texas at Austin, Austin, TX, USA
| | - Shailee Jain
- Department of Computer Science, The University of Texas at Austin, Austin, TX, USA
| | - Alexander G Huth
- Department of Computer Science, The University of Texas at Austin, Austin, TX, USA.
- Department of Neuroscience, The University of Texas at Austin, Austin, TX, USA.
| |
Collapse
|
42
|
Deniz F, Tseng C, Wehbe L, Dupré la Tour T, Gallant JL. Semantic Representations during Language Comprehension Are Affected by Context. J Neurosci 2023; 43:3144-3158. [PMID: 36973013 PMCID: PMC10146529 DOI: 10.1523/jneurosci.2459-21.2023] [Citation(s) in RCA: 6] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/13/2021] [Revised: 02/17/2023] [Accepted: 02/26/2023] [Indexed: 03/29/2023] Open
Abstract
The meaning of words in natural language depends crucially on context. However, most neuroimaging studies of word meaning use isolated words and isolated sentences with little context. Because the brain may process natural language differently from how it processes simplified stimuli, there is a pressing need to determine whether prior results on word meaning generalize to natural language. fMRI was used to record human brain activity while four subjects (two female) read words in four conditions that vary in context: narratives, isolated sentences, blocks of semantically similar words, and isolated words. We then compared the signal-to-noise ratio (SNR) of evoked brain responses, and we used a voxelwise encoding modeling approach to compare the representation of semantic information across the four conditions. We find four consistent effects of varying context. First, stimuli with more context evoke brain responses with higher SNR across bilateral visual, temporal, parietal, and prefrontal cortices compared with stimuli with little context. Second, increasing context increases the representation of semantic information across bilateral temporal, parietal, and prefrontal cortices at the group level. In individual subjects, only natural language stimuli consistently evoke widespread representation of semantic information. Third, context affects voxel semantic tuning. Finally, models estimated using stimuli with little context do not generalize well to natural language. These results show that context has large effects on the quality of neuroimaging data and on the representation of meaning in the brain. Thus, neuroimaging studies that use stimuli with little context may not generalize well to the natural regime.SIGNIFICANCE STATEMENT Context is an important part of understanding the meaning of natural language, but most neuroimaging studies of meaning use isolated words and isolated sentences with little context. Here, we examined whether the results of neuroimaging studies that use out-of-context stimuli generalize to natural language. We find that increasing context improves the quality of neuro-imaging data and changes where and how semantic information is represented in the brain. These results suggest that findings from studies using out-of-context stimuli may not generalize to natural language used in daily life.
Collapse
Affiliation(s)
- Fatma Deniz
- Helen Wills Neuroscience Institute, University of California, Berkeley, California 94720
- Institute of Software Engineering and Theoretical Computer Science, Technische Universität Berlin, Berlin 10623, Germany
| | - Christine Tseng
- Helen Wills Neuroscience Institute, University of California, Berkeley, California 94720
| | - Leila Wehbe
- Machine Learning Department, Carnegie Mellon University, Pittsburgh, Pennsylvania 15213
| | - Tom Dupré la Tour
- Helen Wills Neuroscience Institute, University of California, Berkeley, California 94720
| | - Jack L Gallant
- Helen Wills Neuroscience Institute, University of California, Berkeley, California 94720
- Department of Psychology, University of California, Berkeley, California 94720
| |
Collapse
|
43
|
Lin R, Naselaris T, Kay K, Wehbe L. Stacked regressions and structured variance partitioning for interpretable brain maps. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2023:2023.04.23.537988. [PMID: 37163111 PMCID: PMC10168225 DOI: 10.1101/2023.04.23.537988] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/11/2023]
Abstract
Relating brain activity associated with a complex stimulus to different properties of that stimulus is a powerful approach for constructing functional brain maps. However, when stimuli are naturalistic, their properties are often correlated (e.g., visual and semantic features of natural images, or different layers of a convolutional neural network that are used as features of images). Correlated properties can act as confounders for each other and complicate the interpretability of brain maps, and can impact the robustness of statistical estimators. Here, we present an approach for brain mapping based on two proposed methods: stacking different encoding models and structured variance partitioning. Our stacking algorithm combines encoding models that each use as input a feature space that describes a different stimulus attribute. The algorithm learns to predict the activity of a voxel as a linear combination of the outputs of different encoding models. We show that the resulting combined model can predict held-out brain activity better or at least as well as the individual encoding models. Further, the weights of the linear combination are readily interpretable; they show the importance of each feature space for predicting a voxel. We then build on our stacking models to introduce structured variance partitioning, a new type of variance partitioning that takes into account the known relationships between features. Our approach constrains the size of the hypothesis space and allows us to ask targeted questions about the similarity between feature spaces and brain regions even in the presence of correlations between the feature spaces. We validate our approach in simulation, showcase its brain mapping potential on fMRI data, and release a Python package. Our methods can be useful for researchers interested in aligning brain activity with different layers of a neural network, or with other types of correlated feature spaces.
Collapse
Affiliation(s)
- Ruogu Lin
- Computational Biology Department, Carnegie Mellon University
| | - Thomas Naselaris
- Department of Neuroscience, University of Minnesota
- Center for Magnetic Resonance Research (CMRR), Department of Radiology, University of Minnesota
| | - Kendrick Kay
- Center for Magnetic Resonance Research (CMRR), Department of Radiology, University of Minnesota
| | - Leila Wehbe
- Neuroscience Institute, Carnegie Mellon University
- Machine Learning Department, Carnegie Mellon University
| |
Collapse
|
44
|
Nakai T, Nishimoto S. Artificial neural network modelling of the neural population code underlying mathematical operations. Neuroimage 2023; 270:119980. [PMID: 36848969 DOI: 10.1016/j.neuroimage.2023.119980] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/05/2022] [Revised: 02/10/2023] [Accepted: 02/23/2023] [Indexed: 02/28/2023] Open
Abstract
Mathematical operations have long been regarded as a sparse, symbolic process in neuroimaging studies. In contrast, advances in artificial neural networks (ANN) have enabled extracting distributed representations of mathematical operations. Recent neuroimaging studies have compared distributed representations of the visual, auditory and language domains in ANNs and biological neural networks (BNNs). However, such a relationship has not yet been examined in mathematics. Here we hypothesise that ANN-based distributed representations can explain brain activity patterns of symbolic mathematical operations. We used the fMRI data of a series of mathematical problems with nine different combinations of operators to construct voxel-wise encoding/decoding models using both sparse operator and latent ANN features. Representational similarity analysis demonstrated shared representations between ANN and BNN, an effect particularly evident in the intraparietal sulcus. Feature-brain similarity (FBS) analysis served to reconstruct a sparse representation of mathematical operations based on distributed ANN features in each cortical voxel. Such reconstruction was more efficient when using features from deeper ANN layers. Moreover, latent ANN features allowed the decoding of novel operators not used during model training from brain activity. The current study provides novel insights into the neural code underlying mathematical thought.
Collapse
Affiliation(s)
- Tomoya Nakai
- Center for Information and Neural Networks, National Institute of Information and Communications Technology, Suita, Japan; Lyon Neuroscience Research Center (CRNL), INSERM U1028 - CNRS UMR5292, University of Lyon, Bron, France.
| | - Shinji Nishimoto
- Center for Information and Neural Networks, National Institute of Information and Communications Technology, Suita, Japan; Graduate School of Frontier Biosciences, Osaka University, Suita, Japan; Graduate School of Medicine, Osaka University, Suita, Japan
| |
Collapse
|
45
|
Matsumoto Y, Nishida S, Hayashi R, Son S, Murakami A, Yoshikawa N, Ito H, Oishi N, Masuda N, Murai T, Friston K, Nishimoto S, Takahashi H. Disorganization of Semantic Brain Networks in Schizophrenia Revealed by fMRI. Schizophr Bull 2023; 49:498-506. [PMID: 36542452 PMCID: PMC10016409 DOI: 10.1093/schbul/sbac157] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 12/24/2022]
Abstract
OBJECTIVES Schizophrenia is a mental illness that presents with thought disorders including delusions and disorganized speech. Thought disorders have been regarded as a consequence of the loosening of associations between semantic concepts since the term "schizophrenia" was first coined by Bleuler. However, a mechanistic account of this cardinal disturbance in terms of functional dysconnection has been lacking. To evaluate how aberrant semantic connections are expressed through brain activity, we characterized large-scale network structures of concept representations using functional magnetic resonance imaging (fMRI). STUDY DESIGN We quantified various concept representations in patients' brains from fMRI activity evoked by movie scenes using encoding modeling. We then constructed semantic brain networks by evaluating the similarity of these semantic representations and conducted graph theory-based network analyses. STUDY RESULTS Neurotypical networks had small-world properties similar to those of natural languages, suggesting small-worldness as a universal property in semantic knowledge networks. Conversely, small-worldness was significantly reduced in networks of schizophrenia patients and was correlated with psychological measures of delusions. Patients' semantic networks were partitioned into more distinct categories and had more random within-category structures than those of controls. CONCLUSIONS The differences in conceptual representations manifest altered semantic clustering and associative intrusions that underlie thought disorders. This is the first study to provide pathophysiological evidence for the loosening of associations as reflected in randomization of semantic networks in schizophrenia. Our method provides a promising approach for understanding the neural basis of altered or creative inner experiences of individuals with mental illness or exceptional abilities, respectively.
Collapse
Affiliation(s)
- Yukiko Matsumoto
- Department of Psychiatry and Behavioral Sciences, Graduate School of Medical and Dental Sciences, Tokyo Medical and Dental University, Tokyo, Japan.,Department of Psychiatry, Graduate School of Medicine, Kyoto University, Kyoto, Japan
| | - Satoshi Nishida
- Center for Information and Neural Networks (CiNet), National Institute of Information and Communications Technology (NICT), Suita, Osaka, Japan.,Graduate School of Frontier Biosciences, Osaka University, Suita, Osaka, Japan
| | - Ryusuke Hayashi
- Human Informatics and Interaction Research Institute, National Institute of Advanced Industrial Science and Technology (AIST), Tsukuba, Ibaraki, Japan
| | - Shuraku Son
- Department of Psychiatry, Graduate School of Medicine, Kyoto University, Kyoto, Japan
| | - Akio Murakami
- Department of Psychiatry, Graduate School of Medicine, Kyoto University, Kyoto, Japan
| | | | - Hiroyoshi Ito
- Human Informatics and Interaction Research Institute, National Institute of Advanced Industrial Science and Technology (AIST), Tsukuba, Ibaraki, Japan.,Faculty of Library, Information and Media Science, University of Tsukuba, Tsukuba, Ibaraki, Japan
| | - Naoya Oishi
- Medical Innovation Center, Graduate School of Medicine, Kyoto University, Kyoto, Japan
| | - Naoki Masuda
- Department of Mathematics, University at Buffalo, State University of New York, Buffalo, USA
| | - Toshiya Murai
- Department of Psychiatry, Graduate School of Medicine, Kyoto University, Kyoto, Japan
| | - Karl Friston
- The Wellcome Centre for Human Neuroimaging, UCL Queen Square Institute of Neurology, London, UK
| | - Shinji Nishimoto
- Center for Information and Neural Networks (CiNet), National Institute of Information and Communications Technology (NICT), Suita, Osaka, Japan.,Graduate School of Frontier Biosciences, Osaka University, Suita, Osaka, Japan.,Graduate School of Medicine, Osaka University, Suita, Osaka, Japan
| | - Hidehiko Takahashi
- Department of Psychiatry and Behavioral Sciences, Graduate School of Medical and Dental Sciences, Tokyo Medical and Dental University, Tokyo, Japan.,Department of Psychiatry, Graduate School of Medicine, Kyoto University, Kyoto, Japan.,Center for Brain Integration Research, Tokyo Medical and Dental University, Tokyo, Japan
| |
Collapse
|
46
|
Setti F, Handjaras G, Bottari D, Leo A, Diano M, Bruno V, Tinti C, Cecchetti L, Garbarini F, Pietrini P, Ricciardi E. A modality-independent proto-organization of human multisensory areas. Nat Hum Behav 2023; 7:397-410. [PMID: 36646839 PMCID: PMC10038796 DOI: 10.1038/s41562-022-01507-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/31/2022] [Accepted: 12/05/2022] [Indexed: 01/18/2023]
Abstract
The processing of multisensory information is based upon the capacity of brain regions, such as the superior temporal cortex, to combine information across modalities. However, it is still unclear whether the representation of coherent auditory and visual events requires any prior audiovisual experience to develop and function. Here we measured brain synchronization during the presentation of an audiovisual, audio-only or video-only version of the same narrative in distinct groups of sensory-deprived (congenitally blind and deaf) and typically developed individuals. Intersubject correlation analysis revealed that the superior temporal cortex was synchronized across auditory and visual conditions, even in sensory-deprived individuals who lack any audiovisual experience. This synchronization was primarily mediated by low-level perceptual features, and relied on a similar modality-independent topographical organization of slow temporal dynamics. The human superior temporal cortex is naturally endowed with a functional scaffolding to yield a common representation across multisensory events.
Collapse
Affiliation(s)
- Francesca Setti
- MoMiLab, IMT School for Advanced Studies Lucca, Lucca, Italy
| | | | - Davide Bottari
- MoMiLab, IMT School for Advanced Studies Lucca, Lucca, Italy
| | - Andrea Leo
- Department of Translational Research and Advanced Technologies in Medicine and Surgery, University of Pisa, Pisa, Italy
| | - Matteo Diano
- Department of Psychology, University of Turin, Turin, Italy
| | - Valentina Bruno
- Manibus Lab, Department of Psychology, University of Turin, Turin, Italy
| | - Carla Tinti
- Department of Psychology, University of Turin, Turin, Italy
| | - Luca Cecchetti
- MoMiLab, IMT School for Advanced Studies Lucca, Lucca, Italy
| | | | - Pietro Pietrini
- MoMiLab, IMT School for Advanced Studies Lucca, Lucca, Italy
| | | |
Collapse
|
47
|
Nakai T, Nishimoto S. Quantitative modelling demonstrates format-invariant representations of mathematical problems in the brain. Eur J Neurosci 2023; 57:1003-1017. [PMID: 36710081 DOI: 10.1111/ejn.15925] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/22/2022] [Revised: 01/19/2023] [Accepted: 01/23/2023] [Indexed: 01/31/2023]
Abstract
Mathematical problems can be described in either symbolic form or natural language. Previous studies have reported that activation overlaps exist for these two types of mathematical problems, but it is unclear whether they are based on similar brain representations. Furthermore, quantitative modelling of mathematical problem solving has yet to be attempted. In the present study, subjects underwent 3 h of functional magnetic resonance experiments involving math word and math expression problems, and a read word condition without any calculations was used as a control. To evaluate the brain representations of mathematical problems quantitatively, we constructed voxel-wise encoding models. Both intra- and cross-format encoding modelling significantly predicted brain activity predominantly in the left intraparietal sulcus (IPS), even after subtraction of the control condition. Representational similarity analysis and principal component analysis revealed that mathematical problems with different formats had similar cortical organization in the IPS. These findings support the idea that mathematical problems are represented in the brain in a format-invariant manner.
Collapse
Affiliation(s)
- Tomoya Nakai
- Lyon Neuroscience Research Center (CRNL), INSERM U1028-CNRS UMR5292, University of Lyon, Bron, France.,Center for Information and Neural Networks, National Institute of Information and Communications Technology, Suita, Japan
| | - Shinji Nishimoto
- Center for Information and Neural Networks, National Institute of Information and Communications Technology, Suita, Japan.,Graduate School of Frontier Biosciences, Osaka University, Suita, Japan.,Graduate School of Medicine, Osaka University, Suita, Japan
| |
Collapse
|
48
|
Aqil M, Roseman L. More than meets the eye: The role of sensory dimensions in psychedelic brain dynamics, experience, and therapeutics. Neuropharmacology 2023; 223:109300. [PMID: 36334767 DOI: 10.1016/j.neuropharm.2022.109300] [Citation(s) in RCA: 11] [Impact Index Per Article: 11.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/04/2022] [Revised: 10/08/2022] [Accepted: 10/20/2022] [Indexed: 11/11/2022]
Abstract
Psychedelics are undergoing a major resurgence of scientific and clinical interest. While multiple theories and frameworks have been proposed, there is yet no universal agreement on the mechanisms underlying the complex effects of psychedelics on subjective experience and brain dynamics, nor their therapeutic benefits. Despite being prominent in psychedelic phenomenology and distinct from those elicited by other classes of hallucinogens, the effects of psychedelics on low-level sensory - particularly visual - dimensions of experience, and corresponding brain dynamics, have often been disregarded by contemporary research as 'epiphenomenal byproducts'. Here, we review available evidence from neuroimaging, pharmacology, questionnaires, and clinical studies; we propose extensions to existing models, provide testable hypotheses for the potential therapeutic roles of psychedelic-induced visual hallucinations, and simulations of visual phenomena relying on low-level cortical dynamics. In sum, we show that psychedelic-induced alterations in low-level sensory dimensions 1) are unlikely to be entirely causally reconducible to high-level alterations, but rather co-occur with them in a dialogical interplay, and 2) are likely to play a causally relevant role in determining high-level alterations and therapeutic outcomes. We conclude that reevaluating the currently underappreciated role of sensory dimensions in psychedelic states will be highly valuable for neuroscience and clinical practice, and that integrating low-level and domain-specific aspects of psychedelic effects into existing nonspecific models is a necessary step to further understand how these substances effect both acute and long-term change in the human brain.
Collapse
Affiliation(s)
- Marco Aqil
- Spinoza Centre for Neuroimaging, the Netherlands; Computational Cognitive Neuroscience and Neuroimaging, Institute for Neuroscience, the Netherlands; Experimental and Applied Psychology, Vrije University Amsterdam, the Netherlands.
| | - Leor Roseman
- Centre for Psychedelic Research, Imperial College London, London, United Kingdom
| |
Collapse
|
49
|
Li W, Zheng S, Liao Y, Hong R, He C, Chen W, Deng C, Li X. The brain-inspired decoder for natural visual image reconstruction. Front Neurosci 2023; 17:1130606. [PMID: 37205046 PMCID: PMC10185745 DOI: 10.3389/fnins.2023.1130606] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/23/2022] [Accepted: 03/28/2023] [Indexed: 05/21/2023] Open
Abstract
The visual system provides a valuable model for studying the working mechanisms of sensory processing and high-level consciousness. A significant challenge in this field is the reconstruction of images from decoded neural activity, which could not only test the accuracy of our understanding of the visual system but also provide a practical tool for solving real-world problems. Although recent advances in deep learning have improved the decoding of neural spike trains, little attention has been paid to the underlying mechanisms of the visual system. To address this issue, we propose a deep learning neural network architecture that incorporates the biological properties of the visual system, such as receptive fields, to reconstruct visual images from spike trains. Our model outperforms current models and has been evaluated on different datasets from both retinal ganglion cells (RGCs) and the primary visual cortex (V1) neural spikes. Our model demonstrated the great potential of brain-inspired algorithms to solve a challenge that our brain solves.
Collapse
Affiliation(s)
- Wenyi Li
- Brain Cognition and Brain Disease Institute (BCBDI), Shenzhen-Hong Kong Institute of Brain Science-Shenzhen Fundamental Research Institutions, CAS Key Laboratory of Brain Connectome and Manipulation, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China
- University of Chinese Academy of Sciences, Beijing, China
| | - Shengjie Zheng
- Brain Cognition and Brain Disease Institute (BCBDI), Shenzhen-Hong Kong Institute of Brain Science-Shenzhen Fundamental Research Institutions, CAS Key Laboratory of Brain Connectome and Manipulation, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China
- University of Chinese Academy of Sciences, Beijing, China
| | - Yufan Liao
- Clinical Medicine Institute, Chengdu University of Traditional Chinese Medicine, Chengdu, China
| | - Rongqi Hong
- Brain Cognition and Brain Disease Institute (BCBDI), Shenzhen-Hong Kong Institute of Brain Science-Shenzhen Fundamental Research Institutions, CAS Key Laboratory of Brain Connectome and Manipulation, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China
| | - Chenggang He
- Brain Cognition and Brain Disease Institute (BCBDI), Shenzhen-Hong Kong Institute of Brain Science-Shenzhen Fundamental Research Institutions, CAS Key Laboratory of Brain Connectome and Manipulation, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China
- Illinois Institute of Technology, Chicago, IL, United States
| | - Weiliang Chen
- Brain Cognition and Brain Disease Institute (BCBDI), Shenzhen-Hong Kong Institute of Brain Science-Shenzhen Fundamental Research Institutions, CAS Key Laboratory of Brain Connectome and Manipulation, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China
- University of Chinese Academy of Sciences, Beijing, China
| | - Chunshan Deng
- Brain Cognition and Brain Disease Institute (BCBDI), Shenzhen-Hong Kong Institute of Brain Science-Shenzhen Fundamental Research Institutions, CAS Key Laboratory of Brain Connectome and Manipulation, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China
| | - Xiaojian Li
- Brain Cognition and Brain Disease Institute (BCBDI), Shenzhen-Hong Kong Institute of Brain Science-Shenzhen Fundamental Research Institutions, CAS Key Laboratory of Brain Connectome and Manipulation, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China
- *Correspondence: Xiaojian Li
| |
Collapse
|
50
|
Antonello RJ, Vaidya AR, Huth AG. Scaling laws for language encoding models in fMRI. ADVANCES IN NEURAL INFORMATION PROCESSING SYSTEMS 2023; 36:21895-21907. [PMID: 39035676 PMCID: PMC11258918] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Subscribe] [Scholar Register] [Indexed: 07/23/2024]
Abstract
Representations from transformer-based unidirectional language models are known to be effective at predicting brain responses to natural language. However, most studies comparing language models to brains have used GPT-2 or similarly sized language models. Here we tested whether larger open-source models such as those from the OPT and LLaMA families are better at predicting brain responses recorded using fMRI. Mirroring scaling results from other contexts, we found that brain prediction performance scales logarithmically with model size from 125M to 30B parameter models, with ~15% increased encoding performance as measured by correlation with a held-out test set across 3 subjects. Similar logarithmic behavior was observed when scaling the size of the fMRI training set. We also characterized scaling for acoustic encoding models that use HuBERT, WavLM, and Whisper, and we found comparable improvements with model size. A noise ceiling analysis of these large, high-performance encoding models showed that performance is nearing the theoretical maximum for brain areas such as the precuneus and higher auditory cortex. These results suggest that increasing scale in both models and data will yield incredibly effective models of language processing in the brain, enabling better scientific understanding as well as applications such as decoding.
Collapse
Affiliation(s)
| | - Aditya R Vaidya
- Department of Computer Science, The University of Texas at Austin
| | - Alexander G Huth
- Departments of Computer Science and Neuroscience, The University of Texas at Austin
| |
Collapse
|