1
|
Nigam T, Schwiedrzik CM. Predictions enable top-down pattern separation in the macaque face-processing hierarchy. Nat Commun 2024; 15:7196. [PMID: 39169024 PMCID: PMC11339276 DOI: 10.1038/s41467-024-51543-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/28/2023] [Accepted: 08/07/2024] [Indexed: 08/23/2024] Open
Abstract
Distinguishing faces requires well distinguishable neural activity patterns. Contextual information may separate neural representations, leading to enhanced identity recognition. Here, we use functional magnetic resonance imaging to investigate how predictions derived from contextual information affect the separability of neural activity patterns in the macaque face-processing system, a 3-level processing hierarchy in ventral visual cortex. We find that in the presence of predictions, early stages of this hierarchy exhibit well separable and high-dimensional neural geometries resembling those at the top of the hierarchy. This is accompanied by a systematic shift of tuning properties from higher to lower areas, endowing lower areas with higher-order, invariant representations instead of their feedforward tuning properties. Thus, top-down signals dynamically transform neural representations of faces into separable and high-dimensional neural geometries. Our results provide evidence how predictive context transforms flexible representational spaces to optimally use the computational resources provided by cortical processing hierarchies for better and faster distinction of facial identities.
Collapse
Affiliation(s)
- Tarana Nigam
- Neural Circuits and Cognition Lab, European Neuroscience Institute Göttingen - A Joint Initiative of the University Medical Center Göttingen and the Max Planck Institute for Multidisciplinary Sciences, Grisebachstraße 5, 37077, Göttingen, Germany
- Perception and Plasticity Group, German Primate Center - Leibniz Institute for Primate Research, Kellnerweg 4, 37077, Göttingen, Germany
- Leibniz ScienceCampus 'Primate Cognition', Göttingen, Germany
- International Max Planck Research School 'Neurosciences', Georg August University Göttingen, Grisebachstraße 5, 37077, Göttingen, Germany
| | - Caspar M Schwiedrzik
- Neural Circuits and Cognition Lab, European Neuroscience Institute Göttingen - A Joint Initiative of the University Medical Center Göttingen and the Max Planck Institute for Multidisciplinary Sciences, Grisebachstraße 5, 37077, Göttingen, Germany.
- Perception and Plasticity Group, German Primate Center - Leibniz Institute for Primate Research, Kellnerweg 4, 37077, Göttingen, Germany.
- Leibniz ScienceCampus 'Primate Cognition', Göttingen, Germany.
| |
Collapse
|
2
|
Courellis HS, Minxha J, Cardenas AR, Kimmel DL, Reed CM, Valiante TA, Salzman CD, Mamelak AN, Fusi S, Rutishauser U. Abstract representations emerge in human hippocampal neurons during inference. Nature 2024; 632:841-849. [PMID: 39143207 PMCID: PMC11338822 DOI: 10.1038/s41586-024-07799-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/30/2023] [Accepted: 07/09/2024] [Indexed: 08/16/2024]
Abstract
Humans have the remarkable cognitive capacity to rapidly adapt to changing environments. Central to this capacity is the ability to form high-level, abstract representations that take advantage of regularities in the world to support generalization1. However, little is known about how these representations are encoded in populations of neurons, how they emerge through learning and how they relate to behaviour2,3. Here we characterized the representational geometry of populations of neurons (single units) recorded in the hippocampus, amygdala, medial frontal cortex and ventral temporal cortex of neurosurgical patients performing an inferential reasoning task. We found that only the neural representations formed in the hippocampus simultaneously encode several task variables in an abstract, or disentangled, format. This representational geometry is uniquely observed after patients learn to perform inference, and consists of disentangled directly observable and discovered latent task variables. Learning to perform inference by trial and error or through verbal instructions led to the formation of hippocampal representations with similar geometric properties. The observed relation between representational format and inference behaviour suggests that abstract and disentangled representational geometries are important for complex cognition.
Collapse
Affiliation(s)
- Hristos S Courellis
- Department of Neurosurgery, Cedars-Sinai Medical Center, Los Angeles, CA, USA.
- Division of Biology and Biological Engineering, California Institute of Technology, Pasadena, CA, USA.
| | - Juri Minxha
- Department of Neurosurgery, Cedars-Sinai Medical Center, Los Angeles, CA, USA
- Division of Biology and Biological Engineering, California Institute of Technology, Pasadena, CA, USA
- Mortimer B. Zuckerman Mind Brain Behavior Institute, Columbia University, New York, NY, USA
| | - Araceli R Cardenas
- Krembil Research Institute and Division of Neurosurgery, University Health Network (UHN), University of Toronto, Toronto, Ontario, Canada
| | - Daniel L Kimmel
- Mortimer B. Zuckerman Mind Brain Behavior Institute, Columbia University, New York, NY, USA
- Department of Psychiatry, Columbia University, New York, NY, USA
| | - Chrystal M Reed
- Department of Neurology, Cedars-Sinai Medical Center, Los Angeles, CA, USA
| | - Taufik A Valiante
- Krembil Research Institute and Division of Neurosurgery, University Health Network (UHN), University of Toronto, Toronto, Ontario, Canada
| | - C Daniel Salzman
- Mortimer B. Zuckerman Mind Brain Behavior Institute, Columbia University, New York, NY, USA
- Department of Psychiatry, Columbia University, New York, NY, USA
- New York State Psychiatric Institute, New York, NY, USA
- Department of Neuroscience, Columbia University, New York, NY, USA
- Kavli Institute for Brain Sciences, Columbia University, New York, NY, USA
| | - Adam N Mamelak
- Department of Neurosurgery, Cedars-Sinai Medical Center, Los Angeles, CA, USA
| | - Stefano Fusi
- Mortimer B. Zuckerman Mind Brain Behavior Institute, Columbia University, New York, NY, USA
- Department of Neuroscience, Columbia University, New York, NY, USA
- Kavli Institute for Brain Sciences, Columbia University, New York, NY, USA
| | - Ueli Rutishauser
- Department of Neurosurgery, Cedars-Sinai Medical Center, Los Angeles, CA, USA.
- Division of Biology and Biological Engineering, California Institute of Technology, Pasadena, CA, USA.
- Department of Neurology, Cedars-Sinai Medical Center, Los Angeles, CA, USA.
- Center for Neural Science and Medicine, Department of Biomedical Sciences, Cedars-Sinai Medical Center, Los Angeles, CA, USA.
| |
Collapse
|
3
|
Jurewicz K, Sleezer BJ, Mehta PS, Hayden BY, Ebitz RB. Irrational choices via a curvilinear representational geometry for value. Nat Commun 2024; 15:6424. [PMID: 39080250 PMCID: PMC11289086 DOI: 10.1038/s41467-024-49568-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/29/2023] [Accepted: 06/06/2024] [Indexed: 08/02/2024] Open
Abstract
We make decisions by comparing values, but it is not yet clear how value is represented in the brain. Many models assume, if only implicitly, that the representational geometry of value is linear. However, in part due to a historical focus on noisy single neurons, rather than neuronal populations, this hypothesis has not been rigorously tested. Here, we examine the representational geometry of value in the ventromedial prefrontal cortex (vmPFC), a part of the brain linked to economic decision-making, in two male rhesus macaques. We find that values are encoded along a curved manifold in vmPFC. This curvilinear geometry predicts a specific pattern of irrational decision-making: that decision-makers will make worse choices when an irrelevant, decoy option is worse in value, compared to when it is better. We observe this type of irrational choices in behavior. Together, these results not only suggest that the representational geometry of value is nonlinear, but that this nonlinearity could impose bounds on rational decision-making.
Collapse
Affiliation(s)
- Katarzyna Jurewicz
- Department of Neurosciences, Faculté de médecine, and Centre interdisciplinaire de recherche sur le cerveau et l'apprentissage, Université de Montréal, Montréal, QC, Canada
- Department of Physiology, Faculty of Medicine and Health Sciences, McGill University, Montréal, QC, Canada
| | - Brianna J Sleezer
- Department of Neuroscience, Center for Magnetic Resonance Research, and Center for Neuroengineering, University of Minnesota, Minneapolis, MN, USA
| | - Priyanka S Mehta
- Department of Neuroscience, Center for Magnetic Resonance Research, and Center for Neuroengineering, University of Minnesota, Minneapolis, MN, USA
- Psychology Program, Department of Human Behavior, Justice, and Diversity, University of Wisconsin, Superior, Superior, WI, USA
| | - Benjamin Y Hayden
- Department of Neurosurgery, Baylor College of Medicine, Houston, TX, USA
| | - R Becket Ebitz
- Department of Neurosciences, Faculté de médecine, and Centre interdisciplinaire de recherche sur le cerveau et l'apprentissage, Université de Montréal, Montréal, QC, Canada.
| |
Collapse
|
4
|
Fleming SM, Shea N. Quality space computations for consciousness. Trends Cogn Sci 2024:S1364-6613(24)00165-7. [PMID: 39025769 DOI: 10.1016/j.tics.2024.06.007] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/31/2024] [Revised: 06/18/2024] [Accepted: 06/19/2024] [Indexed: 07/20/2024]
Abstract
The quality space hypothesis about conscious experience proposes that conscious sensory states are experienced in relation to other possible sensory states. For instance, the colour red is experienced as being more like orange, and less like green or blue. Recent empirical findings suggest that subjective similarity space can be explained in terms of similarities in neural activation patterns. Here, we consider how localist, workspace, and higher-order theories of consciousness can accommodate claims about the qualitative character of experience and functionally support a quality space. We review existing empirical evidence for each of these positions, and highlight novel experimental tools, such as altering local activation spaces via brain stimulation or behavioural training, that can distinguish these accounts.
Collapse
Affiliation(s)
- Stephen M Fleming
- Wellcome Centre for Human Neuroimaging, University College London, London, UK; Max Planck UCL Centre for Computational Psychiatry and Ageing Research, University College London, London, UK; Department of Experimental Psychology, University College London, London, UK; Canadian Institute for Advanced Research (CIFAR), Brain, Mind, and Consciousness Program, Toronto, ON, Canada.
| | - Nicholas Shea
- Institute of Philosophy, School of Advanced Study, University of London, London, UK; Faculty of Philosophy, University of Oxford, Oxford, UK.
| |
Collapse
|
5
|
Tye KM, Miller EK, Taschbach FH, Benna MK, Rigotti M, Fusi S. Mixed selectivity: Cellular computations for complexity. Neuron 2024; 112:2289-2303. [PMID: 38729151 PMCID: PMC11257803 DOI: 10.1016/j.neuron.2024.04.017] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/11/2023] [Revised: 03/08/2024] [Accepted: 04/12/2024] [Indexed: 05/12/2024]
Abstract
The property of mixed selectivity has been discussed at a computational level and offers a strategy to maximize computational power by adding versatility to the functional role of each neuron. Here, we offer a biologically grounded implementational-level mechanistic explanation for mixed selectivity in neural circuits. We define pure, linear, and nonlinear mixed selectivity and discuss how these response properties can be obtained in simple neural circuits. Neurons that respond to multiple, statistically independent variables display mixed selectivity. If their activity can be expressed as a weighted sum, then they exhibit linear mixed selectivity; otherwise, they exhibit nonlinear mixed selectivity. Neural representations based on diverse nonlinear mixed selectivity are high dimensional; hence, they confer enormous flexibility to a simple downstream readout neural circuit. However, a simple neural circuit cannot possibly encode all possible mixtures of variables simultaneously, as this would require a combinatorially large number of mixed selectivity neurons. Gating mechanisms like oscillations and neuromodulation can solve this problem by dynamically selecting which variables are mixed and transmitted to the readout.
Collapse
Affiliation(s)
- Kay M Tye
- Salk Institute for Biological Studies, La Jolla, CA, USA; Howard Hughes Medical Institute, La Jolla, CA; Department of Neurobiology, School of Biological Sciences, University of California, San Diego, La Jolla, CA 92093, USA; Kavli Institute for Brain and Mind, San Diego, CA, USA.
| | - Earl K Miller
- The Picower Institute for Learning and Memory, Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA 02139, USA.
| | - Felix H Taschbach
- Salk Institute for Biological Studies, La Jolla, CA, USA; Biological Science Graduate Program, University of California, San Diego, La Jolla, CA 92093, USA; Department of Neurobiology, School of Biological Sciences, University of California, San Diego, La Jolla, CA 92093, USA.
| | - Marcus K Benna
- Department of Neurobiology, School of Biological Sciences, University of California, San Diego, La Jolla, CA 92093, USA.
| | | | - Stefano Fusi
- Center for Theoretical Neuroscience, Columbia University, New York, NY, USA; Zuckerman Mind Brain Behavior Institute, Columbia University, New York, NY, USA; Department of Neuroscience, Columbia University, New York, NY, USA; Kavli Institute for Brain Science, Columbia University, New York, NY, USA.
| |
Collapse
|
6
|
Lindsey JW, Issa EB. Factorized visual representations in the primate visual system and deep neural networks. eLife 2024; 13:RP91685. [PMID: 38968311 PMCID: PMC11226229 DOI: 10.7554/elife.91685] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 07/07/2024] Open
Abstract
Object classification has been proposed as a principal objective of the primate ventral visual stream and has been used as an optimization target for deep neural network models (DNNs) of the visual system. However, visual brain areas represent many different types of information, and optimizing for classification of object identity alone does not constrain how other information may be encoded in visual representations. Information about different scene parameters may be discarded altogether ('invariance'), represented in non-interfering subspaces of population activity ('factorization') or encoded in an entangled fashion. In this work, we provide evidence that factorization is a normative principle of biological visual representations. In the monkey ventral visual hierarchy, we found that factorization of object pose and background information from object identity increased in higher-level regions and strongly contributed to improving object identity decoding performance. We then conducted a large-scale analysis of factorization of individual scene parameters - lighting, background, camera viewpoint, and object pose - in a diverse library of DNN models of the visual system. Models which best matched neural, fMRI, and behavioral data from both monkeys and humans across 12 datasets tended to be those which factorized scene parameters most strongly. Notably, invariance to these parameters was not as consistently associated with matches to neural and behavioral data, suggesting that maintaining non-class information in factorized activity subspaces is often preferred to dropping it altogether. Thus, we propose that factorization of visual scene information is a widely used strategy in brains and DNN models thereof.
Collapse
Affiliation(s)
- Jack W Lindsey
- Zuckerman Mind Brain Behavior Institute, Columbia UniversityNew YorkUnited States
- Department of Neuroscience, Columbia UniversityNew YorkUnited States
| | - Elias B Issa
- Zuckerman Mind Brain Behavior Institute, Columbia UniversityNew YorkUnited States
- Department of Neuroscience, Columbia UniversityNew YorkUnited States
| |
Collapse
|
7
|
Ostojic S, Fusi S. Computational role of structure in neural activity and connectivity. Trends Cogn Sci 2024; 28:677-690. [PMID: 38553340 DOI: 10.1016/j.tics.2024.03.003] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/24/2023] [Revised: 02/29/2024] [Accepted: 03/07/2024] [Indexed: 07/05/2024]
Abstract
One major challenge of neuroscience is identifying structure in seemingly disorganized neural activity. Different types of structure have different computational implications that can help neuroscientists understand the functional role of a particular brain area. Here, we outline a unified approach to characterize structure by inspecting the representational geometry and the modularity properties of the recorded activity and show that a similar approach can also reveal structure in connectivity. We start by setting up a general framework for determining geometry and modularity in activity and connectivity and relating these properties with computations performed by the network. We then use this framework to review the types of structure found in recent studies of model networks performing three classes of computations.
Collapse
Affiliation(s)
- Srdjan Ostojic
- Laboratoire de Neurosciences Cognitives et Computationnelles, INSERM U960, Ecole Normale Superieure - PSL Research University, 75005 Paris, France.
| | - Stefano Fusi
- Center for Theoretical Neuroscience, Columbia University, New York, NY, USA; Zuckerman Mind Brain Behavior Institute, Columbia University, New York, NY, USA; Department of Neuroscience, Columbia University, New York, NY, USA; Kavli Institute for Brain Science, Columbia University, New York, NY, USA
| |
Collapse
|
8
|
Zhang J, Zhou H, Wang S. Distinct visual processing networks for foveal and peripheral visual fields. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2024:2024.06.24.600415. [PMID: 38979165 PMCID: PMC11230199 DOI: 10.1101/2024.06.24.600415] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 07/10/2024]
Abstract
Foveal and peripheral vision are two distinct modes of visual processing essential for navigating the world. However, it remains unclear if they engage different neural mechanisms and circuits within the visual attentional system. Here, we trained macaques to perform a free-gaze visual search task using natural face and object stimuli and recorded a large number of 14588 visually responsive neurons from a broadly distributed network of brain regions involved in visual attentional processing. Foveal and peripheral units had substantially different proportions across brain regions and exhibited systematic differences in encoding visual information and visual attention. The spike-LFP coherence of foveal units was more extensively modulated by both attention and visual selectivity, thus indicating differential engagement of the attention and visual coding network compared to peripheral units. Furthermore, we delineated the interaction and coordination between foveal and peripheral processing for spatial attention and saccade selection. Finally, the search became more efficient with increasing target-induced desynchronization, and foveal and peripheral units exhibited different correlations between neural responses and search behavior. Together, the systematic differences between foveal and peripheral processing provide valuable insights into how the brain processes and integrates visual information from different regions of the visual field. Significance Statement This study investigates the systematic differences between foveal and peripheral vision, two crucial components of visual processing essential for navigating our surroundings. By simultaneously recording from a large number of neurons in the visual attentional neural network, we revealed substantial variations in the proportion and functional characteristics of foveal and peripheral units across different brain regions. We uncovered differential modulation of functional connectivity by attention and visual selectivity, elucidated the intricate interplay between foveal and peripheral processing in spatial attention and saccade selection, and linked neural responses to search behavior. Overall, our study contributes to a deeper understanding of how the brain processes and integrates visual information for active visual behaviors.
Collapse
|
9
|
Zhang J, Cao R, Zhu X, Zhou H, Wang S. Distinct attentional profile and functional connectivity of neurons with visual feature coding in the primate brain. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2024:2024.06.24.600401. [PMID: 38979388 PMCID: PMC11230157 DOI: 10.1101/2024.06.24.600401] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 07/10/2024]
Abstract
Visual attention and object recognition are two critical cognitive functions that significantly influence our perception of the world. While these neural processes converge on the temporal cortex, the exact nature of their interactions remains largely unclear. Here, we systematically investigated the interplay between visual attention and object feature coding by training macaques to perform a free-gaze visual search task using natural face and object stimuli. With a large number of units recorded from multiple brain areas, we discovered that units exhibiting visual feature coding displayed a distinct attentional response profile and functional connectivity compared to units not exhibiting feature coding. Attention directed towards search targets enhanced the pattern separation of stimuli across brain areas, and this enhancement was more pronounced for units encoding visual features. Our findings suggest two stages of neural processing, with the early stage primarily focused on processing visual features and the late stage dedicated to processing attention. Importantly, feature coding in the early stage could predict the attentional effect in the late stage. Together, our results suggest an intricate interplay between visual feature and attention coding in the primate brain, which can be attributed to the differential functional connectivity and neural networks engaged in these processes.
Collapse
|
10
|
Chen YY, Areti A, Yoshor D, Foster BL. Perception and Memory Reinstatement Engage Overlapping Face-Selective Regions within Human Ventral Temporal Cortex. J Neurosci 2024; 44:e2180232024. [PMID: 38627090 PMCID: PMC11140664 DOI: 10.1523/jneurosci.2180-23.2024] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/22/2023] [Revised: 04/03/2024] [Accepted: 04/05/2024] [Indexed: 04/19/2024] Open
Abstract
Humans have the remarkable ability to vividly retrieve sensory details of past events. According to the theory of sensory reinstatement, during remembering, brain regions specialized for processing specific sensory stimuli are reactivated to support content-specific retrieval. Recently, several studies have emphasized transformations in the spatial organization of these reinstated activity patterns. Specifically, studies of scene stimuli suggest a clear anterior shift in the location of retrieval activations compared with the activity observed during perception. However, it is not clear that such transformations occur universally, with inconsistent evidence for other important stimulus categories, particularly faces. One challenge in addressing this question is the careful delineation of face-selective cortices, which are interdigitated with other selective regions, in configurations that spatially differ across individuals. Therefore, we conducted a multisession neuroimaging study to first carefully map individual participants' (nine males and seven females) face-selective regions within ventral temporal cortex (VTC), followed by a second session to examine the activity patterns within these regions during face memory encoding and retrieval. While face-selective regions were expectedly engaged during face perception at encoding, memory retrieval engagement exhibited a more selective and constricted reinstatement pattern within these regions, but did not show any consistent direction of spatial transformation (e.g., anteriorization). We also report on unique human intracranial recordings from VTC under the same experimental conditions. These findings highlight the importance of considering the complex configuration of category-selective cortex in elucidating principles shaping the neural transformations that occur from perception to memory.
Collapse
Affiliation(s)
- Yvonne Y Chen
- Department of Neurosurgery, Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania 19104
| | | | - Daniel Yoshor
- Department of Neurosurgery, Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania 19104
| | - Brett L Foster
- Department of Neurosurgery, Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania 19104
| |
Collapse
|
11
|
Di Bello F, Falcone R, Genovesio A. Simultaneous oscillatory encoding of "hot" and "cold" information during social interactions in the monkey medial prefrontal cortex. iScience 2024; 27:109559. [PMID: 38646179 PMCID: PMC11033171 DOI: 10.1016/j.isci.2024.109559] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/04/2023] [Revised: 11/27/2023] [Accepted: 03/22/2024] [Indexed: 04/23/2024] Open
Abstract
Social interactions in primates require social cognition abilities such as anticipating the partner's future choices as well as pure cognitive skills involving processing task-relevant information. The medial prefrontal cortex (mPFC) has been implicated in these cognitive processes. Here, we investigated the neural oscillations underlying the complex social behaviors involving the interplay of social roles (Actor vs. Observer) and interaction types (whether working with a "Good" or "Bad" partner). We found opposite power modulations of the beta and gamma bands by social roles, indicating dedicated processing for task-related information. Concurrently, the interaction type was conveyed by lower frequencies, which are commonly associated with neural circuits linked to performance and reward monitoring. Thus, the mPFC exhibits parallel coding of both "cold" processes (purely cognitive) and "hot" processes (reward and social-related). This allocation of neural resources gives the mPFC a key neural node, flexibly integrating multiple sources of information during social interactions.
Collapse
Affiliation(s)
- Fabio Di Bello
- Department of Physiology and Pharmacology, Sapienza University of Rome, Rome, Italy
| | - Rossella Falcone
- Department of Physiology and Pharmacology, Sapienza University of Rome, Rome, Italy
- Leo M. Davidoff Department of Neurological Surgery, Albert Einstein College of Medicine Montefiore Medical Center Bronx, Bronx, NY, USA
| | - Aldo Genovesio
- Department of Physiology and Pharmacology, Sapienza University of Rome, Rome, Italy
| |
Collapse
|
12
|
Kobylkov D, Vallortigara G. Face detection mechanisms: Nature vs. nurture. Front Neurosci 2024; 18:1404174. [PMID: 38812973 PMCID: PMC11133589 DOI: 10.3389/fnins.2024.1404174] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/20/2024] [Accepted: 05/02/2024] [Indexed: 05/31/2024] Open
Abstract
For many animals, faces are a vitally important visual stimulus. Hence, it is not surprising that face perception has become a very popular research topic in neuroscience, with ca. 2000 papers published every year. As a result, significant progress has been made in understanding the intricate mechanisms underlying this phenomenon. However, the ontogeny of face perception, in particular the role of innate predispositions, remains largely unexplored at the neural level. Several influential studies in monkeys have suggested that seeing faces is necessary for the development of the face-selective brain domains. At the same time, behavioural experiments with newborn human babies and newly-hatched domestic chicks demonstrate that a spontaneous preference towards faces emerges early in life without pre-existing experience. Moreover, we were recently able to record face-selective neural responses in the brain of young, face-naïve chicks, thus demonstrating the existence of an innate face detection mechanism. In this review, we discuss these seemingly contradictory results and propose potential experimental approaches to resolve some of the open questions.
Collapse
|
13
|
Hill DF, Hickman RW, Al-Mohammad A, Stasiak A, Schultz W. Dopamine neurons encode trial-by-trial subjective reward value in an auction-like task. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2024:2023.01.20.524896. [PMID: 36711724 PMCID: PMC9882283 DOI: 10.1101/2023.01.20.524896] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Indexed: 01/22/2023]
Abstract
The dopamine reward prediction error signal is known to be subjective but has so far only been assessed in aggregate choices. However, personal choices fluctuate across trials and thus reflect the instantaneous subjective reward value. In the well-established Becker-DeGroot-Marschak (BDM) auction-like mechanism, participants are encouraged to place bids that accurately reveal their instantaneous subjective reward value; inaccurate bidding results in suboptimal reward ('incentive compatibility'). In our experiment, male rhesus monkeys became experienced over several years to place accurate BDM bids for juice rewards without specific external constraints. Their bids for physically identical rewards varied trial by trial and increased overall for larger rewards. In these highly experienced animals, responses of midbrain dopamine neurons followed the trial-by-trial variations of bids despite constant, explicitly predicted reward amounts. Inversely, dopamine responses were similar with similar bids for different physical reward amounts. Support Vector Regression demonstrated accurate prediction of the animals' bids by as few as twenty dopamine neurons. Thus, the phasic dopamine reward signal reflects instantaneous subjective reward value.
Collapse
Affiliation(s)
- Daniel F Hill
- Department of Physiology, Development and Neuroscience , University of Cambridge, Cambridge CB2 3DY, United Kingdom
| | - Robert W Hickman
- Department of Physiology, Development and Neuroscience , University of Cambridge, Cambridge CB2 3DY, United Kingdom
| | - Alaa Al-Mohammad
- Department of Physiology, Development and Neuroscience , University of Cambridge, Cambridge CB2 3DY, United Kingdom
| | - Arkadiusz Stasiak
- Department of Physiology, Development and Neuroscience , University of Cambridge, Cambridge CB2 3DY, United Kingdom
| | - Wolfram Schultz
- Department of Physiology, Development and Neuroscience , University of Cambridge, Cambridge CB2 3DY, United Kingdom
| |
Collapse
|
14
|
Dado T, Papale P, Lozano A, Le L, Wang F, van Gerven M, Roelfsema P, Güçlütürk Y, Güçlü U. Brain2GAN: Feature-disentangled neural encoding and decoding of visual perception in the primate brain. PLoS Comput Biol 2024; 20:e1012058. [PMID: 38709818 PMCID: PMC11098503 DOI: 10.1371/journal.pcbi.1012058] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/10/2023] [Revised: 05/16/2024] [Accepted: 04/08/2024] [Indexed: 05/08/2024] Open
Abstract
A challenging goal of neural coding is to characterize the neural representations underlying visual perception. To this end, multi-unit activity (MUA) of macaque visual cortex was recorded in a passive fixation task upon presentation of faces and natural images. We analyzed the relationship between MUA and latent representations of state-of-the-art deep generative models, including the conventional and feature-disentangled representations of generative adversarial networks (GANs) (i.e., z- and w-latents of StyleGAN, respectively) and language-contrastive representations of latent diffusion networks (i.e., CLIP-latents of Stable Diffusion). A mass univariate neural encoding analysis of the latent representations showed that feature-disentangled w representations outperform both z and CLIP representations in explaining neural responses. Further, w-latent features were found to be positioned at the higher end of the complexity gradient which indicates that they capture visual information relevant to high-level neural activity. Subsequently, a multivariate neural decoding analysis of the feature-disentangled representations resulted in state-of-the-art spatiotemporal reconstructions of visual perception. Taken together, our results not only highlight the important role of feature-disentanglement in shaping high-level neural representations underlying visual perception but also serve as an important benchmark for the future of neural coding.
Collapse
Affiliation(s)
- Thirza Dado
- Donders Institute for Brain, Cognition and Behaviour, Radboud University, Nijmegen, Netherlands
| | - Paolo Papale
- Department of Vision and Cognition, Netherlands Institute for Neuroscience, Amsterdam, Netherlands
| | - Antonio Lozano
- Department of Vision and Cognition, Netherlands Institute for Neuroscience, Amsterdam, Netherlands
| | - Lynn Le
- Donders Institute for Brain, Cognition and Behaviour, Radboud University, Nijmegen, Netherlands
| | - Feng Wang
- Department of Vision and Cognition, Netherlands Institute for Neuroscience, Amsterdam, Netherlands
| | - Marcel van Gerven
- Donders Institute for Brain, Cognition and Behaviour, Radboud University, Nijmegen, Netherlands
| | - Pieter Roelfsema
- Department of Vision and Cognition, Netherlands Institute for Neuroscience, Amsterdam, Netherlands
- Laboratory of Visual Brain Therapy, Sorbonne University, Paris, France
- Department of Integrative Neurophysiology, VU Amsterdam, Amsterdam, Netherlands
- Department of Psychiatry, Amsterdam UMC, Amsterdam, Netherlands
| | - Yağmur Güçlütürk
- Donders Institute for Brain, Cognition and Behaviour, Radboud University, Nijmegen, Netherlands
| | - Umut Güçlü
- Donders Institute for Brain, Cognition and Behaviour, Radboud University, Nijmegen, Netherlands
| |
Collapse
|
15
|
She L, Benna MK, Shi Y, Fusi S, Tsao DY. Temporal multiplexing of perception and memory codes in IT cortex. Nature 2024; 629:861-868. [PMID: 38750353 PMCID: PMC11111405 DOI: 10.1038/s41586-024-07349-5] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/19/2021] [Accepted: 03/25/2024] [Indexed: 05/24/2024]
Abstract
A central assumption of neuroscience is that long-term memories are represented by the same brain areas that encode sensory stimuli1. Neurons in inferotemporal (IT) cortex represent the sensory percept of visual objects using a distributed axis code2-4. Whether and how the same IT neural population represents the long-term memory of visual objects remains unclear. Here we examined how familiar faces are encoded in the IT anterior medial face patch (AM), perirhinal face patch (PR) and temporal pole face patch (TP). In AM and PR we observed that the encoding axis for familiar faces is rotated relative to that for unfamiliar faces at long latency; in TP this memory-related rotation was much weaker. Contrary to previous claims, the relative response magnitude to familiar versus unfamiliar faces was not a stable indicator of familiarity in any patch5-11. The mechanism underlying the memory-related axis change is likely intrinsic to IT cortex, because inactivation of PR did not affect axis change dynamics in AM. Overall, our results suggest that memories of familiar faces are represented in AM and perirhinal cortex by a distinct long-latency code, explaining how the same cell population can encode both the percept and memory of faces.
Collapse
Affiliation(s)
- Liang She
- Division of Biology and Biological Engineering, Caltech, Pasadena, CA, USA.
| | - Marcus K Benna
- Mortimer B. Zuckerman Mind Brain Behavior Institute, Columbia University, New York City, NY, USA
- Neurobiology Section, Division of Biological Sciences, University of California, San Diego, San Diego, CA, USA
| | - Yuelin Shi
- Division of Biology and Biological Engineering, Caltech, Pasadena, CA, USA
| | - Stefano Fusi
- Mortimer B. Zuckerman Mind Brain Behavior Institute, Columbia University, New York City, NY, USA
| | - Doris Y Tsao
- Division of Biology and Biological Engineering, Caltech, Pasadena, CA, USA.
- Howard Hughes Medical Institute, University of California, Berkeley, CA, USA.
- Department of Neuroscience, University of California, Berkeley, CA, USA.
| |
Collapse
|
16
|
Rolls ET. Two what, two where, visual cortical streams in humans. Neurosci Biobehav Rev 2024; 160:105650. [PMID: 38574782 DOI: 10.1016/j.neubiorev.2024.105650] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/18/2023] [Revised: 03/25/2024] [Accepted: 03/31/2024] [Indexed: 04/06/2024]
Abstract
ROLLS, E. T. Two What, Two Where, Visual Cortical Streams in Humans. NEUROSCI BIOBEHAV REV 2024. Recent cortical connectivity investigations lead to new concepts about 'What' and 'Where' visual cortical streams in humans, and how they connect to other cortical systems. A ventrolateral 'What' visual stream leads to the inferior temporal visual cortex for object and face identity, and provides 'What' information to the hippocampal episodic memory system, the anterior temporal lobe semantic system, and the orbitofrontal cortex emotion system. A superior temporal sulcus (STS) 'What' visual stream utilising connectivity from the temporal and parietal visual cortex responds to moving objects and faces, and face expression, and connects to the orbitofrontal cortex for emotion and social behaviour. A ventromedial 'Where' visual stream builds feature combinations for scenes, and provides 'Where' inputs via the parahippocampal scene area to the hippocampal episodic memory system that are also useful for landmark-based navigation. The dorsal 'Where' visual pathway to the parietal cortex provides for actions in space, but also provides coordinate transforms to provide inputs to the parahippocampal scene area for self-motion update of locations in scenes in the dark or when the view is obscured.
Collapse
Affiliation(s)
- Edmund T Rolls
- Oxford Centre for Computational Neuroscience, Oxford, UK; Department of Computer Science, University of Warwick, Coventry CV4 7AL, UK; Institute of Science and Technology for Brain Inspired Intelligence, Fudan University, Shanghai 200403, China.
| |
Collapse
|
17
|
Wang Y, Cao R, Wang S. Encoding of Visual Objects in the Human Medial Temporal Lobe. J Neurosci 2024; 44:e2135232024. [PMID: 38429107 PMCID: PMC11026346 DOI: 10.1523/jneurosci.2135-23.2024] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/14/2023] [Revised: 02/10/2024] [Accepted: 02/25/2024] [Indexed: 03/03/2024] Open
Abstract
The human medial temporal lobe (MTL) plays a crucial role in recognizing visual objects, a key cognitive function that relies on the formation of semantic representations. Nonetheless, it remains unknown how visual information of general objects is translated into semantic representations in the MTL. Furthermore, the debate about whether the human MTL is involved in perception has endured for a long time. To address these questions, we investigated three distinct models of neural object coding-semantic coding, axis-based feature coding, and region-based feature coding-in each subregion of the human MTL, using high-resolution fMRI in two male and six female participants. Our findings revealed the presence of semantic coding throughout the MTL, with a higher prevalence observed in the parahippocampal cortex (PHC) and perirhinal cortex (PRC), while axis coding and region coding were primarily observed in the earlier regions of the MTL. Moreover, we demonstrated that voxels exhibiting axis coding supported the transition to region coding and contained information relevant to semantic coding. Together, by providing a detailed characterization of neural object coding schemes and offering a comprehensive summary of visual coding information for each MTL subregion, our results not only emphasize a clear role of the MTL in perceptual processing but also shed light on the translation of perception-driven representations of visual features into memory-driven representations of semantics along the MTL processing pathway.
Collapse
Affiliation(s)
- Yue Wang
- Department of Radiology, Washington University in St. Louis, St. Louis, Missouri 63110
| | - Runnan Cao
- Department of Radiology, Washington University in St. Louis, St. Louis, Missouri 63110
| | - Shuo Wang
- Department of Radiology, Washington University in St. Louis, St. Louis, Missouri 63110
| |
Collapse
|
18
|
Boyle LM, Posani L, Irfan S, Siegelbaum SA, Fusi S. Tuned geometries of hippocampal representations meet the computational demands of social memory. Neuron 2024; 112:1358-1371.e9. [PMID: 38382521 PMCID: PMC11186585 DOI: 10.1016/j.neuron.2024.01.021] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/16/2023] [Revised: 11/03/2023] [Accepted: 01/19/2024] [Indexed: 02/23/2024]
Abstract
Social memory consists of two processes: the detection of familiar compared with novel conspecifics and the detailed recollection of past social episodes. We investigated the neural bases for these processes using calcium imaging of dorsal CA2 hippocampal pyramidal neurons, known to be important for social memory, during social/spatial encounters with novel conspecifics and familiar littermates. Whereas novel individuals were represented in a low-dimensional geometry that allows for generalization of social identity across different spatial locations and of location across different identities, littermates were represented in a higher-dimensional geometry that supports high-capacity memory storage. Moreover, familiarity was represented in an abstract format, independent of individual identity. The degree to which familiarity increased the dimensionality of CA2 representations for individual mice predicted their performance in a social novelty recognition memory test. Thus, by tuning the geometry of structured neural activity, CA2 is able to meet the demands of distinct social memory processes.
Collapse
Affiliation(s)
- Lara M Boyle
- Department of Neuroscience, Vagelos College of Physicians and Surgeons, Columbia University Irving Medical Center, New York, NY 10027, USA
| | - Lorenzo Posani
- Center for Theoretical Neuroscience, Columbia University, New York, NY 10027, USA; Zuckerman Mind Brain Behavior Institute, Columbia University, New York, NY 10027, USA
| | | | - Steven A Siegelbaum
- Department of Neuroscience, Vagelos College of Physicians and Surgeons, Columbia University Irving Medical Center, New York, NY 10027, USA; Zuckerman Mind Brain Behavior Institute, Columbia University, New York, NY 10027, USA; Department of Pharmacology, Vagelos College of Physicians and Surgeons, Columbia University Irving Medical Center, New York, NY 10032, USA; Kavli Institute for Brain Science, Columbia University, New York, NY 10027, USA.
| | - Stefano Fusi
- Department of Neuroscience, Vagelos College of Physicians and Surgeons, Columbia University Irving Medical Center, New York, NY 10027, USA; Center for Theoretical Neuroscience, Columbia University, New York, NY 10027, USA; Zuckerman Mind Brain Behavior Institute, Columbia University, New York, NY 10027, USA; Kavli Institute for Brain Science, Columbia University, New York, NY 10027, USA.
| |
Collapse
|
19
|
Jernigan CM, Freiwald WA, Sheehan MJ. Neural correlates of individual facial recognition in a social wasp. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2024:2024.04.11.589095. [PMID: 38659842 PMCID: PMC11042187 DOI: 10.1101/2024.04.11.589095] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 04/26/2024]
Abstract
Individual recognition is critical for social behavior across species. Whether recognition is mediated by circuits specialized for social information processing has been a matter of debate. Here we examine the neurobiological underpinning of individual visual facial recognition in Polistes fuscatus paper wasps. Front-facing images of conspecific wasps broadly increase activity across many brain regions relative to other stimuli. Notably, we identify a localized subpopulation of neurons in the protocerebrum which show specialized selectivity for front-facing wasp images, which we term wasp cells. These wasp cells encode information regarding the facial patterns, with ensemble activity correlating with facial identity. Wasp cells are strikingly analogous to face cells in primates, indicating that specialized circuits are likely an adaptive feature of neural architecture to support visual recognition.
Collapse
Affiliation(s)
- Christopher M. Jernigan
- Laboratory for Animal Social Evolution and Recognition, Department of Neurobiology and Behavior, Cornell University; Ithaca, NY, 14853, USA
| | - Winrich A. Freiwald
- Laboratory of Neural Systems, The Rockefeller University, New York, NY 10065, USA
| | - Michael J. Sheehan
- Laboratory for Animal Social Evolution and Recognition, Department of Neurobiology and Behavior, Cornell University; Ithaca, NY, 14853, USA
| |
Collapse
|
20
|
DeYoe EA, Huddleston W, Greenberg AS. Are neuronal mechanisms of attention universal across human sensory and motor brain maps? Psychon Bull Rev 2024:10.3758/s13423-024-02495-3. [PMID: 38587756 DOI: 10.3758/s13423-024-02495-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 03/10/2024] [Indexed: 04/09/2024]
Abstract
One's experience of shifting attention from the color to the smell to the act of picking a flower seems like a unitary process applied, at will, to one modality after another. Yet, the unique and separable experiences of sight versus smell versus movement might suggest that the neural mechanisms of attention have been separately optimized to employ each modality to its greatest advantage. Moreover, addressing the issue of universality can be particularly difficult due to a paucity of existing cross-modal comparisons and a dearth of neurophysiological methods that can be applied equally well across disparate modalities. Here we outline some of the conceptual and methodological issues related to this problem and present an instructive example of an experimental approach that can be applied widely throughout the human brain to permit detailed, quantitative comparison of attentional mechanisms across modalities. The ultimate goal is to spur efforts across disciplines to provide a large and varied database of empirical observations that will either support the notion of a universal neural substrate for attention or more clearly identify the degree to which attentional mechanisms are specialized for each modality.
Collapse
Affiliation(s)
- Edgar A DeYoe
- Department of Radiology, Medical College of Wisconsin, 8701 Watertown Plank Rd, Milwaukee, WI, 53226, USA.
- , Signal Mountain, USA.
| | - Wendy Huddleston
- School of Rehabilitation Sciences and Technology, College of Health Professions and Sciences, University of Wisconsin - Milwaukee, 3409 N. Downer Ave, Milwaukee, WI, 53211, USA
| | - Adam S Greenberg
- Department of Biomedical Engineering, Medical College of Wisconsin and Marquette University, Milwaukee, WI, 53226, USA
| |
Collapse
|
21
|
Kay K, Biderman N, Khajeh R, Beiran M, Cueva CJ, Shohamy D, Jensen G, Wei XX, Ferrera VP, Abbott LF. Emergent neural dynamics and geometry for generalization in a transitive inference task. PLoS Comput Biol 2024; 20:e1011954. [PMID: 38662797 PMCID: PMC11125559 DOI: 10.1371/journal.pcbi.1011954] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/26/2023] [Revised: 05/24/2024] [Accepted: 02/28/2024] [Indexed: 05/25/2024] Open
Abstract
Relational cognition-the ability to infer relationships that generalize to novel combinations of objects-is fundamental to human and animal intelligence. Despite this importance, it remains unclear how relational cognition is implemented in the brain due in part to a lack of hypotheses and predictions at the levels of collective neural activity and behavior. Here we discovered, analyzed, and experimentally tested neural networks (NNs) that perform transitive inference (TI), a classic relational task (if A > B and B > C, then A > C). We found NNs that (i) generalized perfectly, despite lacking overt transitive structure prior to training, (ii) generalized when the task required working memory (WM), a capacity thought to be essential to inference in the brain, (iii) emergently expressed behaviors long observed in living subjects, in addition to a novel order-dependent behavior, and (iv) expressed different task solutions yielding alternative behavioral and neural predictions. Further, in a large-scale experiment, we found that human subjects performing WM-based TI showed behavior inconsistent with a class of NNs that characteristically expressed an intuitive task solution. These findings provide neural insights into a classical relational ability, with wider implications for how the brain realizes relational cognition.
Collapse
Affiliation(s)
- Kenneth Kay
- Mortimer B. Zuckerman Mind Brain Behavior Institute, Columbia University, New York, New York, United States of America
- Center for Theoretical Neuroscience, Columbia University, New York, New York, United States of America
- Grossman Center for the Statistics of Mind, Columbia University, New York, New York, United States of America
| | - Natalie Biderman
- Mortimer B. Zuckerman Mind Brain Behavior Institute, Columbia University, New York, New York, United States of America
- Department of Psychology, Columbia University, New York, New York, United States of America
| | - Ramin Khajeh
- Mortimer B. Zuckerman Mind Brain Behavior Institute, Columbia University, New York, New York, United States of America
- Center for Theoretical Neuroscience, Columbia University, New York, New York, United States of America
| | - Manuel Beiran
- Mortimer B. Zuckerman Mind Brain Behavior Institute, Columbia University, New York, New York, United States of America
- Center for Theoretical Neuroscience, Columbia University, New York, New York, United States of America
| | - Christopher J. Cueva
- Department of Brain and Cognitive Sciences, MIT, Cambridge, Massachusetts, United States of America
| | - Daphna Shohamy
- Mortimer B. Zuckerman Mind Brain Behavior Institute, Columbia University, New York, New York, United States of America
- Department of Psychology, Columbia University, New York, New York, United States of America
- The Kavli Institute for Brain Science, Columbia University, New York, New York, United States of America
| | - Greg Jensen
- Mortimer B. Zuckerman Mind Brain Behavior Institute, Columbia University, New York, New York, United States of America
- Department of Neuroscience, Columbia University Medical Center, New York, New York, United States of America
- Department of Psychology at Reed College, Portland, Oregon, United States of America
| | - Xue-Xin Wei
- Departments of Neuroscience and Psychology, The University of Texas at Austin, Austin, Texas, United States of America
| | - Vincent P. Ferrera
- Mortimer B. Zuckerman Mind Brain Behavior Institute, Columbia University, New York, New York, United States of America
- Department of Neuroscience, Columbia University Medical Center, New York, New York, United States of America
- Department of Psychiatry, Columbia University Medical Center, New York, New York, United States of America
| | - LF Abbott
- Mortimer B. Zuckerman Mind Brain Behavior Institute, Columbia University, New York, New York, United States of America
- Center for Theoretical Neuroscience, Columbia University, New York, New York, United States of America
- The Kavli Institute for Brain Science, Columbia University, New York, New York, United States of America
- Department of Neuroscience, Columbia University Medical Center, New York, New York, United States of America
| |
Collapse
|
22
|
Hurley ME, Sonig A, Herrington J, Storch EA, Lázaro-Muñoz G, Blumenthal-Barby J, Kostick-Quenet K. Ethical considerations for integrating multimodal computer perception and neurotechnology. Front Hum Neurosci 2024; 18:1332451. [PMID: 38435745 PMCID: PMC10904467 DOI: 10.3389/fnhum.2024.1332451] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/03/2023] [Accepted: 01/30/2024] [Indexed: 03/05/2024] Open
Abstract
Background Artificial intelligence (AI)-based computer perception technologies (e.g., digital phenotyping and affective computing) promise to transform clinical approaches to personalized care in psychiatry and beyond by offering more objective measures of emotional states and behavior, enabling precision treatment, diagnosis, and symptom monitoring. At the same time, passive and continuous nature by which they often collect data from patients in non-clinical settings raises ethical issues related to privacy and self-determination. Little is known about how such concerns may be exacerbated by the integration of neural data, as parallel advances in computer perception, AI, and neurotechnology enable new insights into subjective states. Here, we present findings from a multi-site NCATS-funded study of ethical considerations for translating computer perception into clinical care and contextualize them within the neuroethics and neurorights literatures. Methods We conducted qualitative interviews with patients (n = 20), caregivers (n = 20), clinicians (n = 12), developers (n = 12), and clinician developers (n = 2) regarding their perspective toward using PC in clinical care. Transcripts were analyzed in MAXQDA using Thematic Content Analysis. Results Stakeholder groups voiced concerns related to (1) perceived invasiveness of passive and continuous data collection in private settings; (2) data protection and security and the potential for negative downstream/future impacts on patients of unintended disclosure; and (3) ethical issues related to patients' limited versus hyper awareness of passive and continuous data collection and monitoring. Clinicians and developers highlighted that these concerns may be exacerbated by the integration of neural data with other computer perception data. Discussion Our findings suggest that the integration of neurotechnologies with existing computer perception technologies raises novel concerns around dignity-related and other harms (e.g., stigma, discrimination) that stem from data security threats and the growing potential for reidentification of sensitive data. Further, our findings suggest that patients' awareness and preoccupation with feeling monitored via computer sensors ranges from hypo- to hyper-awareness, with either extreme accompanied by ethical concerns (consent vs. anxiety and preoccupation). These results highlight the need for systematic research into how best to implement these technologies into clinical care in ways that reduce disruption, maximize patient benefits, and mitigate long-term risks associated with the passive collection of sensitive emotional, behavioral and neural data.
Collapse
Affiliation(s)
- Meghan E. Hurley
- Center for Medical Ethics and Health Policy, Baylor College of Medicine, Houston, TX, United States
| | - Anika Sonig
- Center for Medical Ethics and Health Policy, Baylor College of Medicine, Houston, TX, United States
| | - John Herrington
- Department of Child and Adolescent Psychiatry and Behavioral Sciences, Children’s Hospital of Philadelphia, Philadelphia, PA, United States
| | - Eric A. Storch
- Department of Psychiatry and Behavioral Sciences, Baylor College of Medicine, Houston, TX, United States
| | - Gabriel Lázaro-Muñoz
- Center for Bioethics, Harvard Medical School, Boston, MA, United States
- Department of Psychiatry and Behavioral Sciences, Massachusetts General Hospital, Boston, MA, United States
| | | | - Kristin Kostick-Quenet
- Center for Medical Ethics and Health Policy, Baylor College of Medicine, Houston, TX, United States
| |
Collapse
|
23
|
Sharma KK, Diltz MA, Lincoln T, Albuquerque ER, Romanski LM. Neuronal Population Encoding of Identity in Primate Prefrontal Cortex. J Neurosci 2024; 44:e0703232023. [PMID: 37963766 PMCID: PMC10860606 DOI: 10.1523/jneurosci.0703-23.2023] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/19/2023] [Revised: 08/22/2023] [Accepted: 10/10/2023] [Indexed: 11/16/2023] Open
Abstract
The ventrolateral prefrontal cortex (VLPFC) shows robust activation during the perception of faces and voices. However, little is known about what categorical features of social stimuli drive neural activity in this region. Since perception of identity and expression are critical social functions, we examined whether neural responses to naturalistic stimuli were driven by these two categorical features in the prefrontal cortex. We recorded single neurons in the VLPFC, while two male rhesus macaques (Macaca mulatta) viewed short audiovisual videos of unfamiliar conspecifics making expressions of aggressive, affiliative, and neutral valence. Of the 285 neurons responsive to the audiovisual stimuli, 111 neurons had a main effect (two-way ANOVA) of identity, expression, or their interaction in their stimulus-related firing rates; however, decoding of expression and identity using single-unit firing rates rendered poor accuracy. Interestingly, when decoding from pseudo-populations of recorded neurons, the accuracy for both expression and identity increased with population size, suggesting that the population transmitted information relevant to both variables. Principal components analysis of mean population activity across time revealed that population responses to the same identity followed similar trajectories in the response space, facilitating segregation from other identities. Our results suggest that identity is a critical feature of social stimuli that dictates the structure of population activity in the VLPFC, during the perception of vocalizations and their corresponding facial expressions. These findings enhance our understanding of the role of the VLPFC in social behavior.
Collapse
Affiliation(s)
- K K Sharma
- Department of Neuroscience, School of Medicine and Dentistry, University of Rochester, Rochester, New York 14620
| | - M A Diltz
- Department of Neuroscience, School of Medicine and Dentistry, University of Rochester, Rochester, New York 14620
| | - T Lincoln
- Department of Neuroscience, School of Medicine and Dentistry, University of Rochester, Rochester, New York 14620
| | - E R Albuquerque
- Department of Neuroscience, School of Medicine and Dentistry, University of Rochester, Rochester, New York 14620
| | - L M Romanski
- Department of Neuroscience, School of Medicine and Dentistry, University of Rochester, Rochester, New York 14620
| |
Collapse
|
24
|
Soulos P, Isik L. Disentangled deep generative models reveal coding principles of the human face processing network. PLoS Comput Biol 2024; 20:e1011887. [PMID: 38408105 PMCID: PMC10919870 DOI: 10.1371/journal.pcbi.1011887] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/15/2023] [Revised: 03/07/2024] [Accepted: 02/02/2024] [Indexed: 02/28/2024] Open
Abstract
Despite decades of research, much is still unknown about the computations carried out in the human face processing network. Recently, deep networks have been proposed as a computational account of human visual processing, but while they provide a good match to neural data throughout visual cortex, they lack interpretability. We introduce a method for interpreting brain activity using a new class of deep generative models, disentangled representation learning models, which learn a low-dimensional latent space that "disentangles" different semantically meaningful dimensions of faces, such as rotation, lighting, or hairstyle, in an unsupervised manner by enforcing statistical independence between dimensions. We find that the majority of our model's learned latent dimensions are interpretable by human raters. Further, these latent dimensions serve as a good encoding model for human fMRI data. We next investigate the representation of different latent dimensions across face-selective voxels. We find that low- and high-level face features are represented in posterior and anterior face-selective regions, respectively, corroborating prior models of human face recognition. Interestingly, though, we find identity-relevant and irrelevant face features across the face processing network. Finally, we provide new insight into the few "entangled" (uninterpretable) dimensions in our model by showing that they match responses in the ventral stream and carry information about facial identity. Disentangled face encoding models provide an exciting alternative to standard "black box" deep learning approaches for modeling and interpreting human brain data.
Collapse
Affiliation(s)
- Paul Soulos
- Department of Cognitive Science, Johns Hopkins University, Baltimore, Maryland, United States of America
| | - Leyla Isik
- Department of Cognitive Science, Johns Hopkins University, Baltimore, Maryland, United States of America
| |
Collapse
|
25
|
Ibáñez-Berganza M, Lucibello C, Mariani L, Pezzulo G. Information-theoretical analysis of the neural code for decoupled face representation. PLoS One 2024; 19:e0295054. [PMID: 38277355 PMCID: PMC10817192 DOI: 10.1371/journal.pone.0295054] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/10/2023] [Accepted: 11/15/2023] [Indexed: 01/28/2024] Open
Abstract
Processing faces accurately and efficiently is a key capability of humans and other animals that engage in sophisticated social tasks. Recent studies reported a decoupled coding for faces in the primate inferotemporal cortex, with two separate neural populations coding for the geometric position of (texture-free) facial landmarks and for the image texture at fixed landmark positions, respectively. Here, we formally assess the efficiency of this decoupled coding by appealing to the information-theoretic notion of description length, which quantifies the amount of information that is saved when encoding novel facial images, with a given precision. We show that despite decoupled coding describes the facial images in terms of two sets of principal components (of landmark shape and image texture), it is more efficient (i.e., yields more information compression) than the encoding in terms of the image principal components only, which corresponds to the widely used eigenface method. The advantage of decoupled coding over eigenface coding increases with image resolution and is especially prominent when coding variants of training set images that only differ in facial expressions. Moreover, we demonstrate that decoupled coding entails better performance in three different tasks: the representation of facial images, the (daydream) sampling of novel facial images, and the recognition of facial identities and gender. In summary, our study provides a first principle perspective on the efficiency and accuracy of the decoupled coding of facial stimuli reported in the primate inferotemporal cortex.
Collapse
Affiliation(s)
- Miguel Ibáñez-Berganza
- IMT School for Advanced Studies, Lucca, Italy
- Istituto Italiano di Tecnologia, Napoli, Italy
| | - Carlo Lucibello
- Institute for Data Science and Analytics, Bocconi University, Milano, Italy
| | - Luca Mariani
- Department of Physics “E. R. Caianiello”, University of Salerno, Fisciano, Italy
| | - Giovanni Pezzulo
- Institute of Cognitive Sciences and Technologies, National Research Council, Roma, Italy
| |
Collapse
|
26
|
Gao C, Liu D, Xu C, Xie W, Zhang X, Bai J, Lin Z, Zhang C, Hu Y, Guo T, Chen H. Toward grouped-reservoir computing: organic neuromorphic vertical transistor with distributed reservoir states for efficient recognition and prediction. Nat Commun 2024; 15:740. [PMID: 38272878 PMCID: PMC10810880 DOI: 10.1038/s41467-024-44942-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/17/2023] [Accepted: 01/10/2024] [Indexed: 01/27/2024] Open
Abstract
Reservoir computing has attracted considerable attention due to its low training cost. However, existing neuromorphic hardware, focusing mainly on shallow-reservoir computing, faces challenges in providing adequate spatial and temporal scales characteristic for effective computing. Here, we report an ultra-short channel organic neuromorphic vertical transistor with distributed reservoir states. The carrier dynamics used to map signals are enriched by coupled multivariate physics mechanisms, while the vertical architecture employed greatly increases the feedback intensity of the device. Consequently, the device as a reservoir, effectively mapping sequential signals into distributed reservoir state space with 1152 reservoir states, and the range ratio of temporal and spatial characteristics can simultaneously reach 2640 and 650, respectively. The grouped-reservoir computing based on the device can simultaneously adapt to different spatiotemporal task, achieving recognition accuracy over 94% and prediction correlation over 95%. This work proposes a new strategy for developing high-performance reservoir computing networks.
Collapse
Affiliation(s)
- Changsong Gao
- Institute of Optoelectronic Display, National & Local United Engineering Lab of Flat Panel Display Technology, Fuzhou University, 350002, Fuzhou, China
- Fujian Science & Technology Innovation Laboratory for Optoelectronic Information of China, 350100, Fuzhou, China
| | - Di Liu
- Institute of Optoelectronic Display, National & Local United Engineering Lab of Flat Panel Display Technology, Fuzhou University, 350002, Fuzhou, China
- Fujian Science & Technology Innovation Laboratory for Optoelectronic Information of China, 350100, Fuzhou, China
| | - Chenhui Xu
- Institute of Optoelectronic Display, National & Local United Engineering Lab of Flat Panel Display Technology, Fuzhou University, 350002, Fuzhou, China
- Fujian Science & Technology Innovation Laboratory for Optoelectronic Information of China, 350100, Fuzhou, China
| | - Weidong Xie
- Institute of Optoelectronic Display, National & Local United Engineering Lab of Flat Panel Display Technology, Fuzhou University, 350002, Fuzhou, China
- Fujian Science & Technology Innovation Laboratory for Optoelectronic Information of China, 350100, Fuzhou, China
| | - Xianghong Zhang
- Institute of Optoelectronic Display, National & Local United Engineering Lab of Flat Panel Display Technology, Fuzhou University, 350002, Fuzhou, China
- Fujian Science & Technology Innovation Laboratory for Optoelectronic Information of China, 350100, Fuzhou, China
| | - Junhua Bai
- Joint School of National University of Singapore and Tianjin University, International Campus of Tianjin University, Binhai New City, 350207, Fuzhou, China
| | - Zhixian Lin
- Institute of Optoelectronic Display, National & Local United Engineering Lab of Flat Panel Display Technology, Fuzhou University, 350002, Fuzhou, China
- School of Advanced Manufacturing, Fuzhou University, 362200, Quanzhou, China
| | - Cheng Zhang
- Department of Physics, Fuzhou University, 350108, Fuzhou, China
| | - Yuanyuan Hu
- Changsha Semiconductor Technology and Application Innovation Research Institute, College of Semiconductors (College of Integrated Circuits), Hunan University, 410082, Changsha, China
| | - Tailiang Guo
- Institute of Optoelectronic Display, National & Local United Engineering Lab of Flat Panel Display Technology, Fuzhou University, 350002, Fuzhou, China
- Fujian Science & Technology Innovation Laboratory for Optoelectronic Information of China, 350100, Fuzhou, China
| | - Huipeng Chen
- Institute of Optoelectronic Display, National & Local United Engineering Lab of Flat Panel Display Technology, Fuzhou University, 350002, Fuzhou, China.
- Fujian Science & Technology Innovation Laboratory for Optoelectronic Information of China, 350100, Fuzhou, China.
| |
Collapse
|
27
|
Mahmoodi A, Harbison C, Bongioanni A, Emberton A, Roumazeilles L, Sallet J, Khalighinejad N, Rushworth MFS. A frontopolar-temporal circuit determines the impact of social information in macaque decision making. Neuron 2024; 112:84-92.e6. [PMID: 37863039 PMCID: PMC10914637 DOI: 10.1016/j.neuron.2023.09.035] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/01/2023] [Revised: 07/06/2023] [Accepted: 09/26/2023] [Indexed: 10/22/2023]
Abstract
When choosing, primates are guided not only by personal experience of objects but also by social information such as others' attitudes toward the objects. Crucially, both sources of information-personal and socially derived-vary in reliability. To choose optimally, one must sometimes override choice guidance by personal experience and follow social cues instead, and sometimes one must do the opposite. The dorsomedial frontopolar cortex (dmFPC) tracks reliability of social information and determines whether it will be attended to guide behavior. To do this, dmFPC activity enters specific patterns of interaction with a region in the mid-superior temporal sulcus (mSTS). Reversible disruption of dmFPC activity with transcranial ultrasound stimulation (TUS) led macaques to fail to be guided by social information when it was reliable but to be more likely to use it when it was unreliable. By contrast, mSTS disruption uniformly downregulated the impact of social information on behavior.
Collapse
Affiliation(s)
- Ali Mahmoodi
- Wellcome Centre for Integrative Neuroimaging, Department of Experimental Psychology, University of Oxford, Oxford, UK.
| | - Caroline Harbison
- Wellcome Centre for Integrative Neuroimaging, Department of Experimental Psychology, University of Oxford, Oxford, UK
| | - Alessandro Bongioanni
- Wellcome Centre for Integrative Neuroimaging, Department of Experimental Psychology, University of Oxford, Oxford, UK; Cognitive Neuroimaging Unit, CEA, INSERM, Université Paris-Saclay, NeuroSpin Center, 91191 Gif/Yvette, France
| | - Andrew Emberton
- Department of Biomedical Services, University of Oxford, Oxford, UK
| | - Lea Roumazeilles
- Wellcome Centre for Integrative Neuroimaging, Department of Experimental Psychology, University of Oxford, Oxford, UK
| | - Jerome Sallet
- Wellcome Centre for Integrative Neuroimaging, Department of Experimental Psychology, University of Oxford, Oxford, UK; Univ Lyon, Université Lyon 1, Inserm, Stem Cell and Brain Research Institute, U1208 Bron, France
| | - Nima Khalighinejad
- Wellcome Centre for Integrative Neuroimaging, Department of Experimental Psychology, University of Oxford, Oxford, UK
| | - Matthew F S Rushworth
- Wellcome Centre for Integrative Neuroimaging, Department of Experimental Psychology, University of Oxford, Oxford, UK
| |
Collapse
|
28
|
Yu H, Lin C, Sun S, Cao R, Kar K, Wang S. Multimodal investigations of emotional face processing and social trait judgment of faces. Ann N Y Acad Sci 2024; 1531:29-48. [PMID: 37965931 PMCID: PMC10858652 DOI: 10.1111/nyas.15084] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/16/2023]
Abstract
Faces are among the most important visual stimuli that humans perceive in everyday life. While extensive literature has examined emotional processing and social evaluations of faces, most studies have examined either topic using unimodal approaches. In this review, we promote the use of multimodal cognitive neuroscience approaches to study these processes, using two lines of research as examples: ambiguity in facial expressions of emotion and social trait judgment of faces. In the first set of studies, we identified an event-related potential that signals emotion ambiguity using electroencephalography and we found convergent neural responses to emotion ambiguity using functional neuroimaging and single-neuron recordings. In the second set of studies, we discuss how different neuroimaging and personality-dimensional approaches together provide new insights into social trait judgments of faces. In both sets of studies, we provide an in-depth comparison between neurotypicals and people with autism spectrum disorder. We offer a computational account for the behavioral and neural markers of the different facial processing between the two groups. Finally, we suggest new practices for studying the emotional processing and social evaluations of faces. All data discussed in the case studies of this review are publicly available.
Collapse
Affiliation(s)
- Hongbo Yu
- Department of Psychological & Brain Sciences, University of California Santa Barbara, Santa Barbara, California, USA
| | - Chujun Lin
- Department of Psychology, University of California San Diego, San Diego, California, USA
| | - Sai Sun
- Frontier Research Institute for Interdisciplinary Sciences, Tohoku University, Sendai, Japan
- Research Institute of Electrical Communication, Tohoku University, Sendai, Japan
| | - Runnan Cao
- Department of Radiology, Washington University in St. Louis, St. Louis, Missouri, USA
| | - Kohitij Kar
- Department of Biology, Centre for Vision Research, York University, Toronto, Ontario, Canada
| | - Shuo Wang
- Department of Radiology, Washington University in St. Louis, St. Louis, Missouri, USA
| |
Collapse
|
29
|
O'Sullivan FM, Ryan TJ. If Engrams Are the Answer, What Is the Question? ADVANCES IN NEUROBIOLOGY 2024; 38:273-302. [PMID: 39008021 DOI: 10.1007/978-3-031-62983-9_15] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 07/16/2024]
Abstract
Engram labelling and manipulation methodologies are now a staple of contemporary neuroscientific practice, giving the impression that the physical basis of engrams has been discovered. Despite enormous progress, engrams have not been clearly identified, and it is unclear what they should look like. There is an epistemic bias in engram neuroscience toward characterizing biological changes while neglecting the development of theory. However, the tools of engram biology are exciting precisely because they are not just an incremental step forward in understanding the mechanisms of plasticity and learning but because they can be leveraged to inform theory on one of the fundamental mysteries in neuroscience-how and in what format the brain stores information. We do not propose such a theory here, as we first require an appreciation for what is lacking. We outline a selection of issues in four sections from theoretical biology and philosophy that engram biology and systems neuroscience generally should engage with in order to construct useful future theoretical frameworks. Specifically, what is it that engrams are supposed to explain? How do the different building blocks of the brain-wide engram come together? What exactly are these component parts? And what information do they carry, if they carry anything at all? Asking these questions is not purely the privilege of philosophy but a key to informing scientific hypotheses that make the most of the experimental tools at our disposal. The risk for not engaging with these issues is high. Without a theory of what engrams are, what they do, and the wider computational processes they fit into, we may never know when they have been found.
Collapse
Affiliation(s)
- Fionn M O'Sullivan
- School of Biochemistry and Immunology, Trinity College Dublin, Dublin, Ireland
- Trinity College Institute of Neuroscience, Trinity College Dublin, Dublin, Ireland
| | - Tomás J Ryan
- School of Biochemistry and Immunology, Trinity College Dublin, Dublin, Ireland.
- Trinity College Institute of Neuroscience, Trinity College Dublin, Dublin, Ireland.
- Florey Institute of Neuroscience and Mental Health, Melbourne Brain Centre, University of Melbourne, Melbourne, VIC, Australia.
- Child & Brain Development Program, Canadian Institute for Advanced Research (CIFAR), Toronto, ON, Canada.
| |
Collapse
|
30
|
Yan Y, Zhan J, Garrod O, Cui X, Ince RAA, Schyns PG. Strength of predicted information content in the brain biases decision behavior. Curr Biol 2023; 33:5505-5514.e6. [PMID: 38065096 DOI: 10.1016/j.cub.2023.10.042] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/22/2023] [Revised: 10/11/2023] [Accepted: 10/23/2023] [Indexed: 12/21/2023]
Abstract
Prediction-for-perception theories suggest that the brain predicts incoming stimuli to facilitate their categorization.1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17 However, it remains unknown what the information contents of these predictions are, which hinders mechanistic explanations. This is because typical approaches cast predictions as an underconstrained contrast between two categories18,19,20,21,22,23,24-e.g., faces versus cars, which could lead to predictions of features specific to faces or cars, or features from both categories. Here, to pinpoint the information contents of predictions and thus their mechanistic processing in the brain, we identified the features that enable two different categorical perceptions of the same stimuli. We then trained multivariate classifiers to discern, from dynamic MEG brain responses, the features tied to each perception. With an auditory cueing design, we reveal where, when, and how the brain reactivates visual category features (versus the typical category contrast) before the stimulus is shown. We demonstrate that the predictions of category features have a more direct influence (bias) on subsequent decision behavior in participants than the typical category contrast. Specifically, these predictions are more precisely localized in the brain (lateralized), are more specifically driven by the auditory cues, and their reactivation strength before a stimulus presentation exerts a greater bias on how the individual participant later categorizes this stimulus. By characterizing the specific information contents that the brain predicts and then processes, our findings provide new insights into the brain's mechanisms of prediction for perception.
Collapse
Affiliation(s)
- Yuening Yan
- School of Psychology and Neuroscience, University of Glasgow, 62 Hillhead Street, Glasgow G12 8QB, UK
| | - Jiayu Zhan
- School of Psychological and Cognitive Sciences, Peking University, 5 Yiheyuan Road, Beijing 100871, China
| | - Oliver Garrod
- School of Psychology and Neuroscience, University of Glasgow, 62 Hillhead Street, Glasgow G12 8QB, UK
| | - Xuan Cui
- School of Psychology and Neuroscience, University of Glasgow, 62 Hillhead Street, Glasgow G12 8QB, UK
| | - Robin A A Ince
- School of Psychology and Neuroscience, University of Glasgow, 62 Hillhead Street, Glasgow G12 8QB, UK
| | - Philippe G Schyns
- School of Psychology and Neuroscience, University of Glasgow, 62 Hillhead Street, Glasgow G12 8QB, UK.
| |
Collapse
|
31
|
Shi Y, Bi D, Hesse JK, Lanfranchi FF, Chen S, Tsao DY. Rapid, concerted switching of the neural code in inferotemporal cortex. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2023:2023.12.06.570341. [PMID: 38106108 PMCID: PMC10723419 DOI: 10.1101/2023.12.06.570341] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 12/19/2023]
Abstract
A fundamental paradigm in neuroscience is the concept of neural coding through tuning functions 1 . According to this idea, neurons encode stimuli through fixed mappings of stimulus features to firing rates. Here, we report that the tuning of visual neurons can rapidly and coherently change across a population to attend to a whole and its parts. We set out to investigate a longstanding debate concerning whether inferotemporal (IT) cortex uses a specialized code for representing specific types of objects or whether it uses a general code that applies to any object. We found that face cells in macaque IT cortex initially adopted a general code optimized for face detection. But following a rapid, concerted population event lasting < 20 ms, the neural code transformed into a face-specific one with two striking properties: (i) response gradients to principal detection-related dimensions reversed direction, and (ii) new tuning developed to multiple higher feature space dimensions supporting fine face discrimination. These dynamics were face specific and did not occur in response to objects. Overall, these results show that, for faces, face cells shift from detection to discrimination by switching from an object-general code to a face-specific code. More broadly, our results suggest a novel mechanism for neural representation: concerted, stimulus-dependent switching of the neural code used by a cortical area.
Collapse
|
32
|
Suzuki M, Pennartz CMA, Aru J. How deep is the brain? The shallow brain hypothesis. Nat Rev Neurosci 2023; 24:778-791. [PMID: 37891398 DOI: 10.1038/s41583-023-00756-z] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 09/25/2023] [Indexed: 10/29/2023]
Abstract
Deep learning and predictive coding architectures commonly assume that inference in neural networks is hierarchical. However, largely neglected in deep learning and predictive coding architectures is the neurobiological evidence that all hierarchical cortical areas, higher or lower, project to and receive signals directly from subcortical areas. Given these neuroanatomical facts, today's dominance of cortico-centric, hierarchical architectures in deep learning and predictive coding networks is highly questionable; such architectures are likely to be missing essential computational principles the brain uses. In this Perspective, we present the shallow brain hypothesis: hierarchical cortical processing is integrated with a massively parallel process to which subcortical areas substantially contribute. This shallow architecture exploits the computational capacity of cortical microcircuits and thalamo-cortical loops that are not included in typical hierarchical deep learning and predictive coding networks. We argue that the shallow brain architecture provides several critical benefits over deep hierarchical structures and a more complete depiction of how mammalian brains achieve fast and flexible computational capabilities.
Collapse
Affiliation(s)
- Mototaka Suzuki
- Department of Cognitive and Systems Neuroscience, Swammerdam Institute for Life Sciences, University of Amsterdam, Amsterdam, The Netherlands.
| | - Cyriel M A Pennartz
- Department of Cognitive and Systems Neuroscience, Swammerdam Institute for Life Sciences, University of Amsterdam, Amsterdam, The Netherlands
| | - Jaan Aru
- Institute of Computer Science, University of Tartu, Tartu, Estonia.
| |
Collapse
|
33
|
Zavatone-Veth JA, Masset P, Tong WL, Zak JD, Murthy VN, Pehlevan C. Neural Circuits for Fast Poisson Compressed Sensing in the Olfactory Bulb. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2023:2023.06.21.545947. [PMID: 37961548 PMCID: PMC10634677 DOI: 10.1101/2023.06.21.545947] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/15/2023]
Abstract
Within a single sniff, the mammalian olfactory system can decode the identity and concentration of odorants wafted on turbulent plumes of air. Yet, it must do so given access only to the noisy, dimensionally-reduced representation of the odor world provided by olfactory receptor neurons. As a result, the olfactory system must solve a compressed sensing problem, relying on the fact that only a handful of the millions of possible odorants are present in a given scene. Inspired by this principle, past works have proposed normative compressed sensing models for olfactory decoding. However, these models have not captured the unique anatomy and physiology of the olfactory bulb, nor have they shown that sensing can be achieved within the 100-millisecond timescale of a single sniff. Here, we propose a rate-based Poisson compressed sensing circuit model for the olfactory bulb. This model maps onto the neuron classes of the olfactory bulb, and recapitulates salient features of their connectivity and physiology. For circuit sizes comparable to the human olfactory bulb, we show that this model can accurately detect tens of odors within the timescale of a single sniff. We also show that this model can perform Bayesian posterior sampling for accurate uncertainty estimation. Fast inference is possible only if the geometry of the neural code is chosen to match receptor properties, yielding a distributed neural code that is not axis-aligned to individual odor identities. Our results illustrate how normative modeling can help us map function onto specific neural circuits to generate new hypotheses.
Collapse
Affiliation(s)
- Jacob A Zavatone-Veth
- Center for Brain Science, Harvard University Cambridge, MA 02138
- Department of Physics, Harvard University Cambridge, MA 02138
| | - Paul Masset
- Center for Brain Science, Harvard University Cambridge, MA 02138
- Department of Molecular and Cellular Biology, Harvard University Cambridge, MA 02138
| | - William L Tong
- Center for Brain Science, Harvard University Cambridge, MA 02138
- John A. Paulson School of Engineering and Applied Sciences, Harvard University Cambridge, MA 02138
- Kempner Institute for the Study of Natural and Artificial Intelligence, Harvard University Cambridge, MA 02138
| | - Joseph D Zak
- Department of Biological Sciences, University of Illinois at Chicago Chicago, IL 60607
| | - Venkatesh N Murthy
- Center for Brain Science, Harvard University Cambridge, MA 02138
- Department of Molecular and Cellular Biology, Harvard University Cambridge, MA 02138
| | - Cengiz Pehlevan
- Center for Brain Science, Harvard University Cambridge, MA 02138
- John A. Paulson School of Engineering and Applied Sciences, Harvard University Cambridge, MA 02138
- Kempner Institute for the Study of Natural and Artificial Intelligence, Harvard University Cambridge, MA 02138
| |
Collapse
|
34
|
Tyree TJ, Metke M, Miller CT. Cross-modal representation of identity in the primate hippocampus. Science 2023; 382:417-423. [PMID: 37883535 PMCID: PMC11086670 DOI: 10.1126/science.adf0460] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/25/2022] [Accepted: 09/01/2023] [Indexed: 10/28/2023]
Abstract
Faces and voices are the dominant social signals used to recognize individuals among primates. Yet, it is not known how these signals are integrated into a cross-modal representation of individual identity in the primate brain. We discovered that, although single neurons in the marmoset hippocampus exhibited selective responses when presented with the face or voice of a specific individual, a parallel mechanism for representing the cross-modal identities for multiple individuals was evident within single neurons and at the population level. Manifold projections likewise showed the separability of individuals as well as clustering for others' families, which suggests that multiple learned social categories are encoded as related dimensions of identity in the hippocampus. Neural representations of identity in the hippocampus are thus both modality independent and reflect the primate social network.
Collapse
Affiliation(s)
- Timothy J Tyree
- Cortical Systems and Behavior Laboratory, University of California San Diego; 9500 Gilman Dr. La Jolla, CA 92039, USA
- Department of Physics, University of California San Diego; 9500 Gilman Dr. La Jolla, CA 92039, USA
| | - Michael Metke
- Cortical Systems and Behavior Laboratory, University of California San Diego; 9500 Gilman Dr. La Jolla, CA 92039, USA
- Neurosciences Graduate Program, University of California San Diego; 9500 Gilman Dr. La Jolla, CA 92039, USA
| | - Cory T Miller
- Cortical Systems and Behavior Laboratory, University of California San Diego; 9500 Gilman Dr. La Jolla, CA 92039, USA
- Neurosciences Graduate Program, University of California San Diego; 9500 Gilman Dr. La Jolla, CA 92039, USA
| |
Collapse
|
35
|
Jiahui G, Feilong M, Visconti di Oleggio Castello M, Nastase SA, Haxby JV, Gobbini MI. Modeling naturalistic face processing in humans with deep convolutional neural networks. Proc Natl Acad Sci U S A 2023; 120:e2304085120. [PMID: 37847731 PMCID: PMC10614847 DOI: 10.1073/pnas.2304085120] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/16/2023] [Accepted: 09/11/2023] [Indexed: 10/19/2023] Open
Abstract
Deep convolutional neural networks (DCNNs) trained for face identification can rival and even exceed human-level performance. The ways in which the internal face representations in DCNNs relate to human cognitive representations and brain activity are not well understood. Nearly all previous studies focused on static face image processing with rapid display times and ignored the processing of naturalistic, dynamic information. To address this gap, we developed the largest naturalistic dynamic face stimulus set in human neuroimaging research (700+ naturalistic video clips of unfamiliar faces). We used this naturalistic dataset to compare representational geometries estimated from DCNNs, behavioral responses, and brain responses. We found that DCNN representational geometries were consistent across architectures, cognitive representational geometries were consistent across raters in a behavioral arrangement task, and neural representational geometries in face areas were consistent across brains. Representational geometries in late, fully connected DCNN layers, which are optimized for individuation, were much more weakly correlated with cognitive and neural geometries than were geometries in late-intermediate layers. The late-intermediate face-DCNN layers successfully matched cognitive representational geometries, as measured with a behavioral arrangement task that primarily reflected categorical attributes, and correlated with neural representational geometries in known face-selective topographies. Our study suggests that current DCNNs successfully capture neural cognitive processes for categorical attributes of faces but less accurately capture individuation and dynamic features.
Collapse
Affiliation(s)
- Guo Jiahui
- Center for Cognitive Neuroscience, Dartmouth College, Hanover, NH03755
| | - Ma Feilong
- Center for Cognitive Neuroscience, Dartmouth College, Hanover, NH03755
| | | | - Samuel A. Nastase
- Princeton Neuroscience Institute, Princeton University, Princeton, NJ08544
| | - James V. Haxby
- Center for Cognitive Neuroscience, Dartmouth College, Hanover, NH03755
| | - M. Ida Gobbini
- Department of Medical and Surgical Sciences, University of Bologna, Bologna40138, Italy
- Istituti di Ricovero e Cura a Carattere Scientifico, Istituto delle Scienze Neurologiche di Bologna, Bologna40139, Italia
| |
Collapse
|
36
|
van Dyck LE, Gruber WR. Modeling Biological Face Recognition with Deep Convolutional Neural Networks. J Cogn Neurosci 2023; 35:1521-1537. [PMID: 37584587 DOI: 10.1162/jocn_a_02040] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 08/17/2023]
Abstract
Deep convolutional neural networks (DCNNs) have become the state-of-the-art computational models of biological object recognition. Their remarkable success has helped vision science break new ground, and recent efforts have started to transfer this achievement to research on biological face recognition. In this regard, face detection can be investigated by comparing face-selective biological neurons and brain areas to artificial neurons and model layers. Similarly, face identification can be examined by comparing in vivo and in silico multidimensional "face spaces." In this review, we summarize the first studies that use DCNNs to model biological face recognition. On the basis of a broad spectrum of behavioral and computational evidence, we conclude that DCNNs are useful models that closely resemble the general hierarchical organization of face recognition in the ventral visual pathway and the core face network. In two exemplary spotlights, we emphasize the unique scientific contributions of these models. First, studies on face detection in DCNNs indicate that elementary face selectivity emerges automatically through feedforward processing even in the absence of visual experience. Second, studies on face identification in DCNNs suggest that identity-specific experience and generative mechanisms facilitate this particular challenge. Taken together, as this novel modeling approach enables close control of predisposition (i.e., architecture) and experience (i.e., training data), it may be suited to inform long-standing debates on the substrates of biological face recognition.
Collapse
|
37
|
Ukita J, Ohki K. Adversarial attacks and defenses using feature-space stochasticity. Neural Netw 2023; 167:875-889. [PMID: 37722983 DOI: 10.1016/j.neunet.2023.08.022] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/21/2022] [Revised: 05/27/2023] [Accepted: 08/13/2023] [Indexed: 09/20/2023]
Abstract
Recent studies in deep neural networks have shown that injecting random noise in the input layer of the networks contributes towards ℓp-norm-bounded adversarial perturbations. However, to defend against unrestricted adversarial examples, most of which are not ℓp-norm-bounded in the input layer, such input-layer random noise may not be sufficient. In the first part of this study, we generated a novel class of unrestricted adversarial examples termed feature-space adversarial examples. These examples are far from the original data in the input space but adjacent to the original data in a hidden-layer feature space and far again in the output layer. In the second part of this study, we empirically showed that while injecting random noise in the input layer was unable to defend these feature-space adversarial examples, they were defended by injecting random noise in the hidden layer. These results highlight the novel benefit of stochasticity in higher layers, in that it is useful for defending against these feature-space adversarial examples, a class of unrestricted adversarial examples.
Collapse
Affiliation(s)
- Jumpei Ukita
- Department of Physiology, The University of Tokyo School of Medicine, 7-3-1, Hongo, Bunkyo-ku, 113-0033, Tokyo, Japan.
| | - Kenichi Ohki
- Department of Physiology, The University of Tokyo School of Medicine, 7-3-1, Hongo, Bunkyo-ku, 113-0033, Tokyo, Japan; International Research Center for Neurointelligence (WPI-IRCN), 7-3-1, Hongo, Bunkyo-ku, 113-0033, Tokyo, Japan; Institute for AI and Beyond, 7-3-1, Hongo, Bunkyo-ku, 113-0033, Tokyo, Japan.
| |
Collapse
|
38
|
Romanski LM, Sharma KK. Multisensory interactions of face and vocal information during perception and memory in ventrolateral prefrontal cortex. Philos Trans R Soc Lond B Biol Sci 2023; 378:20220343. [PMID: 37545305 PMCID: PMC10404928 DOI: 10.1098/rstb.2022.0343] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/16/2023] [Accepted: 03/21/2023] [Indexed: 08/08/2023] Open
Abstract
The ventral frontal lobe is a critical node in the circuit that underlies communication, a multisensory process where sensory features of faces and vocalizations come together. The neural basis of face and vocal integration is a topic of great importance since the integration of multiple sensory signals is essential for the decisions that govern our social interactions. Investigations have shown that the macaque ventrolateral prefrontal cortex (VLPFC), a proposed homologue of the human inferior frontal gyrus, is involved in the processing, integration and remembering of audiovisual signals. Single neurons in VLPFC encode and integrate species-specific faces and corresponding vocalizations. During working memory, VLPFC neurons maintain face and vocal information online and exhibit selective activity for face and vocal stimuli. Population analyses indicate that identity, a critical feature of social stimuli, is encoded by VLPFC neurons and dictates the structure of dynamic population activity in the VLPFC during the perception of vocalizations and their corresponding facial expressions. These studies suggest that VLPFC may play a primary role in integrating face and vocal stimuli with contextual information, in order to support decision making during social communication. This article is part of the theme issue 'Decision and control processes in multisensory perception'.
Collapse
Affiliation(s)
- Lizabeth M. Romanski
- Department of Neuroscience, University of Rochester School of Medicine, Rochester, NY 14642, USA
| | - Keshov K. Sharma
- Department of Neuroscience, University of Rochester School of Medicine, Rochester, NY 14642, USA
| |
Collapse
|
39
|
Vinken K, Prince JS, Konkle T, Livingstone MS. The neural code for "face cells" is not face-specific. SCIENCE ADVANCES 2023; 9:eadg1736. [PMID: 37647400 PMCID: PMC10468123 DOI: 10.1126/sciadv.adg1736] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 12/07/2022] [Accepted: 07/27/2023] [Indexed: 09/01/2023]
Abstract
Face cells are neurons that respond more to faces than to non-face objects. They are found in clusters in the inferotemporal cortex, thought to process faces specifically, and, hence, studied using faces almost exclusively. Analyzing neural responses in and around macaque face patches to hundreds of objects, we found graded response profiles for non-face objects that predicted the degree of face selectivity and provided information on face-cell tuning beyond that from actual faces. This relationship between non-face and face responses was not predicted by color and simple shape properties but by information encoded in deep neural networks trained on general objects rather than face classification. These findings contradict the long-standing assumption that face versus non-face selectivity emerges from face-specific features and challenge the practice of focusing on only the most effective stimulus. They provide evidence instead that category-selective neurons are best understood by their tuning directions in a domain-general object space.
Collapse
Affiliation(s)
- Kasper Vinken
- Department of Neurobiology, Harvard Medical School, Boston, MA 02115, USA
| | - Jacob S. Prince
- Department of Psychology, Harvard University, Cambridge, MA 02478, USA
| | - Talia Konkle
- Department of Psychology, Harvard University, Cambridge, MA 02478, USA
| | | |
Collapse
|
40
|
Hesse JK, Tsao DY. Functional modules for visual scene segmentation in macaque visual cortex. Proc Natl Acad Sci U S A 2023; 120:e2221122120. [PMID: 37523552 PMCID: PMC10410728 DOI: 10.1073/pnas.2221122120] [Citation(s) in RCA: 4] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/12/2022] [Accepted: 06/26/2023] [Indexed: 08/02/2023] Open
Abstract
Segmentation, the computation of object boundaries, is one of the most important steps in intermediate visual processing. Previous studies have reported cells across visual cortex that are modulated by segmentation features, but the functional role of these cells remains unclear. First, it is unclear whether these cells encode segmentation consistently since most studies used only a limited variety of stimulus types. Second, it is unclear whether these cells are organized into specialized modules or instead randomly scattered across the visual cortex: the former would lend credence to a functional role for putative segmentation cells. Here, we used fMRI-guided electrophysiology to systematically characterize the consistency and spatial organization of segmentation-encoding cells across the visual cortex. Using fMRI, we identified a set of patches in V2, V3, V3A, V4, and V4A that were more active for stimuli containing figures compared to ground, regardless of whether figures were defined by texture, motion, luminance, or disparity. We targeted these patches for single-unit recordings and found that cells inside segmentation patches were tuned to both figure-ground and borders more consistently across types of stimuli than cells in the visual cortex outside the patches. Remarkably, we found clusters of cells inside segmentation patches that showed the same border-ownership preference across all stimulus types. Finally, using a population decoding approach, we found that segmentation could be decoded with higher accuracy from segmentation patches than from either color-selective or control regions. Overall, our results suggest that segmentation signals are preferentially encoded in spatially discrete patches.
Collapse
Affiliation(s)
- Janis K. Hesse
- Department of Molecular and Cell Biology, University of California, Berkeley, CA94720
- Helen Wills Neuroscience Institute, University of California, Berkeley, CA94720
| | - Doris Y. Tsao
- Department of Molecular and Cell Biology, University of California, Berkeley, CA94720
- Helen Wills Neuroscience Institute, University of California, Berkeley, CA94720
| |
Collapse
|
41
|
Chen YY, Areti A, Yoshor D, Foster BL. Individual-specific memory reinstatement patterns within human face-selective cortex. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2023:2023.08.06.552130. [PMID: 37609262 PMCID: PMC10441346 DOI: 10.1101/2023.08.06.552130] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 08/24/2023]
Abstract
Humans have the remarkable ability to vividly retrieve sensory details of past events. According to the theory of sensory reinstatement, during remembering, brain regions involved in the sensory processing of prior events are reactivated to support this perception of the past. Recently, several studies have emphasized potential transformations in the spatial organization of reinstated activity patterns. In particular, studies of scene stimuli suggest a clear anterior shift in the location of retrieval activations compared with those during perception. However, it is not clear that such transformations occur universally, with evidence lacking for other important stimulus categories, particularly faces. Critical to addressing these questions, and to studies of reinstatement more broadly, is the growing importance of considering meaningful variations in the organization of sensory systems across individuals. Therefore, we conducted a multi-session neuroimaging study to first carefully map individual participants face-selective regions within ventral temporal cortex (VTC), followed by a second session to examine the correspondence of activity patterns during face memory encoding and retrieval. Our results showed distinct configurations of face-selective regions within the VTC across individuals. While a significant degree of overlap was observed between face perception and memory encoding, memory retrieval engagement exhibited a more selective and constricted reinstatement pattern within these regions. Importantly, these activity patterns were consistently tied to individual-specific neural substrates, but did not show any consistent direction of spatial transformation (e.g., anteriorization). To provide further insight to these findings, we also report on unique human intracranial recordings from VTC under the same experimental conditions. Our findings highlight the importance of considering individual variations in functional neuroanatomy in the context of assessing the nature of cortical reinstatement. Consideration of such factors will be important for establishing general principles shaping the neural transformations that occur from perception to memory.
Collapse
Affiliation(s)
- Yvonne Y Chen
- Department of Neurosurgery, Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania, 19104, USA
| | | | - Daniel Yoshor
- Department of Neurosurgery, Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania, 19104, USA
| | - Brett L Foster
- Department of Neurosurgery, Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania, 19104, USA
| |
Collapse
|
42
|
Fan Z, Liu Z, Yang J, Yang J, Sun F, Tang S, Wu G, Guo S, Ouyang X, Tao H. Hypoactive Visual Cortex, Prefrontal Cortex and Insula during Self-Face Recognition in Adults with First-Episode Major Depressive Disorder. Biomedicines 2023; 11:2200. [PMID: 37626697 PMCID: PMC10452386 DOI: 10.3390/biomedicines11082200] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/12/2023] [Revised: 07/30/2023] [Accepted: 08/01/2023] [Indexed: 08/27/2023] Open
Abstract
Self-face recognition is a vital aspect of self-referential processing, which is closely related to affective states. However, neuroimaging research on self-face recognition in adults with major depressive disorder is lacking. This study aims to investigate the alteration of brain activation during self-face recognition in adults with first-episode major depressive disorder (FEMDD) via functional magnetic resonance imaging (fMRI); FEMDD (n = 59) and healthy controls (HC, n = 36) who performed a self-face-recognition task during the fMRI scan. The differences in brain activation signal values between the two groups were analyzed, and Pearson correlation analysis was used to evaluate the relationship between the brain activation of significant group differences and the severity of depressive symptoms and negative self-evaluation; FEMDD showed significantly decreased brain activation in the bilateral occipital cortex, bilateral fusiform gyrus, right inferior frontal gyrus, and right insula during the task compared with HC. No significant correlation was detected between brain activation with significant group differences and the severity of depression and negative self-evaluation in FEMDD or HC. The results suggest the involvement of the malfunctioning visual cortex, prefrontal cortex, and insula in the pathophysiology of self-face recognition in FEMDD, which may provide a novel therapeutic target for adults with FEMDD.
Collapse
Affiliation(s)
- Zebin Fan
- Department of Psychiatry, National Clinical Research Center for Mental Disorders, and National Center for Mental Disorders, The Second Xiangya Hospital of Central South University, Changsha 410011, China; (Z.F.)
| | - Zhening Liu
- Department of Psychiatry, National Clinical Research Center for Mental Disorders, and National Center for Mental Disorders, The Second Xiangya Hospital of Central South University, Changsha 410011, China; (Z.F.)
| | - Jie Yang
- Department of Psychiatry, National Clinical Research Center for Mental Disorders, and National Center for Mental Disorders, The Second Xiangya Hospital of Central South University, Changsha 410011, China; (Z.F.)
| | - Jun Yang
- Department of Psychiatry, National Clinical Research Center for Mental Disorders, and National Center for Mental Disorders, The Second Xiangya Hospital of Central South University, Changsha 410011, China; (Z.F.)
| | - Fuping Sun
- Department of Psychiatry, National Clinical Research Center for Mental Disorders, and National Center for Mental Disorders, The Second Xiangya Hospital of Central South University, Changsha 410011, China; (Z.F.)
| | - Shixiong Tang
- Department of Radiology, The Second Xiangya Hospital of Central South University, Changsha 410011, China
| | - Guowei Wu
- Department of Psychiatry, National Clinical Research Center for Mental Disorders, and National Center for Mental Disorders, The Second Xiangya Hospital of Central South University, Changsha 410011, China; (Z.F.)
| | - Shuixia Guo
- Key Laboratory of Computing and Stochastic Mathematics (Ministry of Education), School of Mathematics and Statistics, Hunan Normal University, Changsha 410006, China
- Key Laboratory of Applied Statistics and Data Science, College of Hunan Province, Hunan Normal University, Changsha 410006, China
| | - Xuan Ouyang
- Department of Psychiatry, National Clinical Research Center for Mental Disorders, and National Center for Mental Disorders, The Second Xiangya Hospital of Central South University, Changsha 410011, China; (Z.F.)
| | - Haojuan Tao
- Department of Psychiatry, National Clinical Research Center for Mental Disorders, and National Center for Mental Disorders, The Second Xiangya Hospital of Central South University, Changsha 410011, China; (Z.F.)
| |
Collapse
|
43
|
Genkin M, Shenoy KV, Chandrasekaran C, Engel TA. The dynamics and geometry of choice in premotor cortex. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2023:2023.07.22.550183. [PMID: 37546748 PMCID: PMC10401920 DOI: 10.1101/2023.07.22.550183] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 08/08/2023]
Abstract
The brain represents sensory variables in the coordinated activity of neural populations, in which tuning curves of single neurons define the geometry of the population code. Whether the same coding principle holds for dynamic cognitive variables remains unknown because internal cognitive processes unfold with a unique time course on single trials observed only in the irregular spiking of heterogeneous neural populations. Here we show the existence of such a population code for the dynamics of choice formation in the primate premotor cortex. We developed an approach to simultaneously infer population dynamics and tuning functions of single neurons to the population state. Applied to spike data recorded during decision-making, our model revealed that populations of neurons encoded the same dynamic variable predicting choices, and heterogeneous firing rates resulted from the diverse tuning of single neurons to this decision variable. The inferred dynamics indicated an attractor mechanism for decision computation. Our results reveal a common geometric principle for neural encoding of sensory and dynamic cognitive variables.
Collapse
Affiliation(s)
| | - Krishna V Shenoy
- Howard Hughes Medical Institute, Stanford University, Stanford, CA
- Department of Electrical Engineering, Stanford University, Stanford, CA
| | - Chandramouli Chandrasekaran
- Department of Anatomy & Neurobiology, Boston University, Boston, MA
- Department of Psychological and Brain Sciences, Boston University, Boston, MA
- Center for Systems Neuroscience, Boston University, Boston, MA
| | - Tatiana A Engel
- Cold Spring Harbor Laboratory, Cold Spring Harbor, NY
- Princeton Neuroscience Institute, Princeton University, Princeton, NJ
| |
Collapse
|
44
|
Deen B, Schwiedrzik CM, Sliwa J, Freiwald WA. Specialized Networks for Social Cognition in the Primate Brain. Annu Rev Neurosci 2023; 46:381-401. [PMID: 37428602 PMCID: PMC11115357 DOI: 10.1146/annurev-neuro-102522-121410] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 07/12/2023]
Abstract
Primates have evolved diverse cognitive capabilities to navigate their complex social world. To understand how the brain implements critical social cognitive abilities, we describe functional specialization in the domains of face processing, social interaction understanding, and mental state attribution. Systems for face processing are specialized from the level of single cells to populations of neurons within brain regions to hierarchically organized networks that extract and represent abstract social information. Such functional specialization is not confined to the sensorimotor periphery but appears to be a pervasive theme of primate brain organization all the way to the apex regions of cortical hierarchies. Circuits processing social information are juxtaposed with parallel systems involved in processing nonsocial information, suggesting common computations applied to different domains. The emerging picture of the neural basis of social cognition is a set of distinct but interacting subnetworks involved in component processes such as face perception and social reasoning, traversing large parts of the primate brain.
Collapse
Affiliation(s)
- Ben Deen
- Psychology Department & Tulane Brain Institute, Tulane University, New Orleans, Louisiana, USA
| | - Caspar M Schwiedrzik
- Neural Circuits and Cognition Lab, European Neuroscience Institute Göttingen, A Joint Initiative of the University Medical Center Göttingen and the Max Planck Society; Perception and Plasticity Group, German Primate Center, Leibniz Institute for Primate Research; and Leibniz-Science Campus Primate Cognition, Göttingen, Germany
| | - Julia Sliwa
- Sorbonne Université, Institut du Cerveau, ICM, Inserm, CNRS, APHP, Hôpital de la Pitié Salpêtrière, Paris, France
| | - Winrich A Freiwald
- Laboratory of Neural Systems and The Price Family Center for the Social Brain, The Rockefeller University, New York, NY, USA;
- The Center for Brains, Minds and Machines, Cambridge, Massachusetts, USA
| |
Collapse
|
45
|
Sotomayor-Gómez B, Battaglia FP, Vinck M. SpikeShip: A method for fast, unsupervised discovery of high-dimensional neural spiking patterns. PLoS Comput Biol 2023; 19:e1011335. [PMID: 37523401 PMCID: PMC10414626 DOI: 10.1371/journal.pcbi.1011335] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/02/2023] [Revised: 08/10/2023] [Accepted: 07/07/2023] [Indexed: 08/02/2023] Open
Abstract
Neural coding and memory formation depend on temporal spiking sequences that span high-dimensional neural ensembles. The unsupervised discovery and characterization of these spiking sequences requires a suitable dissimilarity measure to spiking patterns, which can then be used for clustering and decoding. Here, we present a new dissimilarity measure based on optimal transport theory called SpikeShip, which compares multi-neuron spiking patterns based on all the relative spike-timing relationships among neurons. SpikeShip computes the optimal transport cost to make all the relative spike-timing relationships (across neurons) identical between two spiking patterns. We show that this transport cost can be decomposed into a temporal rigid translation term, which captures global latency shifts, and a vector of neuron-specific transport flows, which reflect inter-neuronal spike timing differences. SpikeShip can be effectively computed for high-dimensional neuronal ensembles, has a low (linear) computational cost that has the same order as the spike count, and is sensitive to higher-order correlations. Furthermore, SpikeShip is binless, can handle any form of spike time distributions, is not affected by firing rate fluctuations, can detect patterns with a low signal-to-noise ratio, and can be effectively combined with a sliding window approach. We compare the advantages and differences between SpikeShip and other measures like SPIKE and Victor-Purpura distance. We applied SpikeShip to large-scale Neuropixel recordings during spontaneous activity and visual encoding. We show that high-dimensional spiking sequences detected via SpikeShip reliably distinguish between different natural images and different behavioral states. These spiking sequences carried complementary information to conventional firing rate codes. SpikeShip opens new avenues for studying neural coding and memory consolidation by rapid and unsupervised detection of temporal spiking patterns in high-dimensional neural ensembles.
Collapse
Affiliation(s)
- Boris Sotomayor-Gómez
- Donders Centre for Neuroscience, Department of Neurophysics, Radboud University Nijmegen, Nijmegen, Netherlands
- Ernst Strüngmann Institute (ESI) for Neuroscience in Cooperation with Max Planck Society, Frankfurt, Germany
| | - Francesco P. Battaglia
- Donders Institute for Brain, Cognition and Behaviour, Radboud University Nijmegen, Nijmegen, Netherlands
| | - Martin Vinck
- Donders Centre for Neuroscience, Department of Neurophysics, Radboud University Nijmegen, Nijmegen, Netherlands
- Ernst Strüngmann Institute (ESI) for Neuroscience in Cooperation with Max Planck Society, Frankfurt, Germany
| |
Collapse
|
46
|
Li D, Chang L. Representational geometry of incomplete faces in macaque face patches. Cell Rep 2023; 42:112673. [PMID: 37342911 DOI: 10.1016/j.celrep.2023.112673] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/12/2022] [Revised: 04/23/2023] [Accepted: 06/06/2023] [Indexed: 06/23/2023] Open
Abstract
The neural code of faces has been intensively studied in the macaque face patch system. Although the majority of previous studies used complete faces as stimuli, faces are often seen partially in daily life. Here, we investigated how face-selective cells represent two types of incomplete faces: face fragments and occluded faces, with the location of the fragment/occluder and the facial features systematically varied. Contrary to popular belief, we found that the preferred face regions identified with two stimulus types are dissociated in many face cells. This dissociation can be explained by the nonlinear integration of information from different face parts and is closely related to a curved representation of face completeness in the state space, which allows a clear discrimination between different stimulus types. Furthermore, identity-related facial features are represented in a subspace orthogonal to the nonlinear dimension of face completeness, supporting a condition-general code of facial identity.
Collapse
Affiliation(s)
- Dongyuan Li
- Institute of Neuroscience, Key Laboratory of Primate Neurobiology, CAS Center for Excellence in Brain Science and Intelligence Technology, Chinese Academy of Sciences, Shanghai 200031, China; University of Chinese Academy of Sciences, Beijing 100049, China
| | - Le Chang
- Institute of Neuroscience, Key Laboratory of Primate Neurobiology, CAS Center for Excellence in Brain Science and Intelligence Technology, Chinese Academy of Sciences, Shanghai 200031, China; University of Chinese Academy of Sciences, Beijing 100049, China.
| |
Collapse
|
47
|
Ferraro S, Van de Maele T, Verbelen T, Dhoedt B. Symmetry and complexity in object-centric deep active inference models. Interface Focus 2023; 13:20220077. [PMID: 37065264 PMCID: PMC10102726 DOI: 10.1098/rsfs.2022.0077] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/02/2022] [Accepted: 03/01/2023] [Indexed: 04/18/2023] Open
Abstract
Humans perceive and interact with hundreds of objects every day. In doing so, they need to employ mental models of these objects and often exploit symmetries in the object's shape and appearance in order to learn generalizable and transferable skills. Active inference is a first principles approach to understanding and modelling sentient agents. It states that agents entertain a generative model of their environment, and learn and act by minimizing an upper bound on their surprisal, i.e. their free energy. The free energy decomposes into an accuracy and complexity term, meaning that agents favour the least complex model that can accurately explain their sensory observations. In this paper, we investigate how inherent symmetries of particular objects also emerge as symmetries in the latent state space of the generative model learnt under deep active inference. In particular, we focus on object-centric representations, which are trained from pixels to predict novel object views as the agent moves its viewpoint. First, we investigate the relation between model complexity and symmetry exploitation in the state space. Second, we do a principal component analysis to demonstrate how the model encodes the principal axis of symmetry of the object in the latent space. Finally, we also demonstrate how more symmetrical representations can be exploited for better generalization in the context of manipulation.
Collapse
Affiliation(s)
- Stefano Ferraro
- IDLab, Department of Information Technology, Ghent University–imec, Ghent, Belgium
| | - Toon Van de Maele
- IDLab, Department of Information Technology, Ghent University–imec, Ghent, Belgium
| | - Tim Verbelen
- IDLab, Department of Information Technology, Ghent University–imec, Ghent, Belgium
| | - Bart Dhoedt
- IDLab, Department of Information Technology, Ghent University–imec, Ghent, Belgium
| |
Collapse
|
48
|
Jones MW, Hunt T. Electromagnetic-field theories of qualia: can they improve upon standard neuroscience? Front Psychol 2023; 14:1015967. [PMID: 37325753 PMCID: PMC10267331 DOI: 10.3389/fpsyg.2023.1015967] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/31/2022] [Accepted: 04/24/2023] [Indexed: 06/17/2023] Open
Abstract
How do brains create all our different colors, pains, and other conscious qualities? These various qualia are the most essential aspects of consciousness. Yet standard neuroscience (primarily based on synaptic information processing) has not found the synaptic-firing codes, sometimes described as the "spike code," to account for how these qualia arise and how they unite to form complex perceptions, emotions, et cetera. Nor is it clear how to get from these abstract codes to the qualia we experience. But electromagnetic field (versus synaptic) approaches to how qualia arise have been offered in recent years by Pockett, McFadden, Jones, Bond, Ward and Guevera, Keppler and Shani, Hunt and Schooler, et cetera. These EM-field approaches show promise in offering more viable accounts of qualia. Yet, until now, they have not been evaluated together. We review various EM field theories of qualia, highlight their strengths and weaknesses, and contrast these theories with standard neuroscience approaches.
Collapse
Affiliation(s)
| | - Tam Hunt
- Department of Psychological and Brain Sciences, University of California, Santa Barbara, Santa Barbara, CA, United States
| |
Collapse
|
49
|
Chong I, Ramezanpour H, Thier P. Causal Manipulation of Gaze-Following in the Macaque Temporal Cortex. Prog Neurobiol 2023; 226:102466. [PMID: 37211234 DOI: 10.1016/j.pneurobio.2023.102466] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/23/2022] [Revised: 05/09/2023] [Accepted: 05/17/2023] [Indexed: 05/23/2023]
Abstract
Gaze-following, the ability to shift one's own attention to places or objects others are looking at, is essential for social interactions. Single unit recordings from the monkey cortex and neuroimaging work on the human and monkey brain suggest that a distinct region in the temporal cortex, the gaze-following patch (GFP), underpins this ability. Since previous studies of the GFP have relied on correlational techniques, it remains unclear whether gaze-following related activity in the GFP indicates a causal role rather than being just a reverberation of behaviorally relevant information produced elsewhere. To answer this question, we applied focal electrical and pharmacological perturbation to the GFP. Both approaches, when applied to the GFP, disrupted gaze-following if the monkeys had been instructed to follow gaze, along with the ability to suppress it if vetoed by the context. Hence the GFP is necessary for gaze-following as well as its cognitive control.
Collapse
Affiliation(s)
- Ian Chong
- Cognitive Neurology Laboratory, Hertie Institute for Clinical Brain Research, University of Tübingen, Tübingen, Germany.
| | - Hamidreza Ramezanpour
- Cognitive Neurology Laboratory, Hertie Institute for Clinical Brain Research, University of Tübingen, Tübingen, Germany; Centre for Vision Research, York University, Toronto, Ontario, Canada
| | - Peter Thier
- Cognitive Neurology Laboratory, Hertie Institute for Clinical Brain Research, University of Tübingen, Tübingen, Germany; Werner Reichardt Centre for Integrative Neuroscience, University of Tübingen, Tübingen, Germany.
| |
Collapse
|
50
|
Rolls ET, Rauschecker JP, Deco G, Huang CC, Feng J. Auditory cortical connectivity in humans. Cereb Cortex 2023; 33:6207-6227. [PMID: 36573464 PMCID: PMC10422925 DOI: 10.1093/cercor/bhac496] [Citation(s) in RCA: 14] [Impact Index Per Article: 14.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/25/2022] [Revised: 11/27/2022] [Accepted: 11/29/2022] [Indexed: 12/28/2022] Open
Abstract
To understand auditory cortical processing, the effective connectivity between 15 auditory cortical regions and 360 cortical regions was measured in 171 Human Connectome Project participants, and complemented with functional connectivity and diffusion tractography. 1. A hierarchy of auditory cortical processing was identified from Core regions (including A1) to Belt regions LBelt, MBelt, and 52; then to PBelt; and then to HCP A4. 2. A4 has connectivity to anterior temporal lobe TA2, and to HCP A5, which connects to dorsal-bank superior temporal sulcus (STS) regions STGa, STSda, and STSdp. These STS regions also receive visual inputs about moving faces and objects, which are combined with auditory information to help implement multimodal object identification, such as who is speaking, and what is being said. Consistent with this being a "what" ventral auditory stream, these STS regions then have effective connectivity to TPOJ1, STV, PSL, TGv, TGd, and PGi, which are language-related semantic regions connecting to Broca's area, especially BA45. 3. A4 and A5 also have effective connectivity to MT and MST, which connect to superior parietal regions forming a dorsal auditory "where" stream involved in actions in space. Connections of PBelt, A4, and A5 with BA44 may form a language-related dorsal stream.
Collapse
Affiliation(s)
- Edmund T Rolls
- Oxford Centre for Computational Neuroscience, Oxford, UK
- Department of Computer Science, University of Warwick, Coventry CV4 7AL, UK
- Shanghai Key Laboratory of Brain Functional Genomics (Ministry of Education), School of Psychology and Cognitive Science, East China Normal University, Shanghai 200602, China
| | - Josef P Rauschecker
- Department of Neuroscience, Georgetown University Medical Center, Washington, DC 20057, USA
- Institute for Advanced Study, Technical University, Munich, Germany
| | - Gustavo Deco
- Center for Brain and Cognition, Computational Neuroscience Group, Department of Information and Communication Technologies, Universitat Pompeu Fabra, Roc Boronat 138, Brain and Cognition, Pompeu Fabra University, Barcelona 08018, Spain
- Institució Catalana de la Recerca i Estudis Avançats (ICREA), Universitat Pompeu Fabra, Passeig Lluís Companys 23, Barcelona 08010, Spain
| | - Chu-Chung Huang
- Shanghai Key Laboratory of Brain Functional Genomics (Ministry of Education), School of Psychology and Cognitive Science, East China Normal University, Shanghai 200602, China
| | - Jianfeng Feng
- Department of Computer Science, University of Warwick, Coventry CV4 7AL, UK
- Institute of Science and Technology for Brain Inspired Intelligence, Fudan University, Shanghai 200403, China
| |
Collapse
|