1
|
Monteiro LCP, da Costa Brito FA, da Costa Brito Lacerda EM, Goulart PRK, Miquilini L, Costa MF, Ripardo RC, Ventura DF, Souza GS. Increased color preference through the introduction of luminance noise in chromatic stimuli. Sci Rep 2024; 14:18711. [PMID: 39134609 PMCID: PMC11319817 DOI: 10.1038/s41598-024-69690-z] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/21/2023] [Accepted: 08/07/2024] [Indexed: 08/15/2024] Open
Abstract
Humans exhibit consistent color preferences that are often described as a curvilinear pattern across hues. The recent literature posits that color preference is linked to the preference for objects or other entities associated with those colors. However, many studies examine this preference using isoluminant colors, which don't reflect the natural viewing experience typically influenced by different light intensities. The inclusion of random luminance levels (luminance noise) in chromatic stimuli may provide an initial step towards assessing color preference as it is presented in the real world. Employing mosaic stimuli, this study aimed to evaluate the influence of luminance noise on human color preference. Thirty normal trichromats engaged in a two-alternative forced-choice paradigm, indicating their color preferences between presented pairs. The chromatic stimuli included saturated versions of 8 standard hues, presented in mosaics with varying diameters under different luminance noise conditions. Results indicated that the inclusion of luminance noise increased color preference across all hues, specifically under the high luminance noise range, while the curvilinear pattern remained unchanged. Finally, women exhibit a greater sensitivity to the presence of luminance noise than men, potentially due to differences between men and women in aesthetic evaluation strategies.
Collapse
Affiliation(s)
| | - Felipe André da Costa Brito
- Instituto de Ciências Biológicas, Universidade Federal Do Pará, Av. Perimetral, 2-224, Belém, 66077830, Brazil
| | | | | | - Letícia Miquilini
- Núcleo de Teoria E Pesquisa Do Comportamento, Universidade Federal Do Pará, Belém, Brazil
| | | | - Rachel Coelho Ripardo
- Núcleo de Teoria E Pesquisa Do Comportamento, Universidade Federal Do Pará, Belém, Brazil
| | - Dora Fix Ventura
- Instituto de Psicologia, Universidade de São Paulo, São Paulo, Brazil
| | - Givago Silva Souza
- Instituto de Ciências Biológicas, Universidade Federal Do Pará, Av. Perimetral, 2-224, Belém, 66077830, Brazil
- Núcleo de Medicina Tropical, Universidade Federal Do Pará, Belém, Brazil
| |
Collapse
|
2
|
Stanojevic A, Woźniak S, Bellec G, Cherubini G, Pantazi A, Gerstner W. High-performance deep spiking neural networks with 0.3 spikes per neuron. Nat Commun 2024; 15:6793. [PMID: 39122775 PMCID: PMC11315905 DOI: 10.1038/s41467-024-51110-5] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/20/2023] [Accepted: 07/29/2024] [Indexed: 08/12/2024] Open
Abstract
Communication by rare, binary spikes is a key factor for the energy efficiency of biological brains. However, it is harder to train biologically-inspired spiking neural networks than artificial neural networks. This is puzzling given that theoretical results provide exact mapping algorithms from artificial to spiking neural networks with time-to-first-spike coding. In this paper we analyze in theory and simulation the learning dynamics of time-to-first-spike-networks and identify a specific instance of the vanishing-or-exploding gradient problem. While two choices of spiking neural network mappings solve this problem at initialization, only the one with a constant slope of the neuron membrane potential at threshold guarantees the equivalence of the training trajectory between spiking and artificial neural networks with rectified linear units. For specific image classification architectures comprising feed-forward dense or convolutional layers, we demonstrate that deep spiking neural network models can be effectively trained from scratch on MNIST and Fashion-MNIST datasets, or fine-tuned on large-scale datasets, such as CIFAR10, CIFAR100 and PLACES365, to achieve the exact same performance as that of artificial neural networks, surpassing previous spiking neural networks. Our approach accomplishes high-performance classification with less than 0.3 spikes per neuron, lending itself for an energy-efficient implementation. We also show that fine-tuning spiking neural networks with our robust gradient descent algorithm enables their optimization for hardware implementations with low latency and resilience to noise and quantization.
Collapse
Affiliation(s)
- Ana Stanojevic
- IBM Research Europe - Zurich, Rüschlikon, Switzerland
- School of Computer and Communication Sciences, École Polytechnique Fédérale de Lausanne, Lausanne, Switzerland
| | | | - Guillaume Bellec
- School of Computer and Communication Sciences, École Polytechnique Fédérale de Lausanne, Lausanne, Switzerland
- School of Life Sciences, École Polytechnique Fédérale de Lausanne, Lausanne, Switzerland
| | | | | | - Wulfram Gerstner
- School of Computer and Communication Sciences, École Polytechnique Fédérale de Lausanne, Lausanne, Switzerland
- School of Life Sciences, École Polytechnique Fédérale de Lausanne, Lausanne, Switzerland
| |
Collapse
|
3
|
Motlagh SC, Joanisse M, Wang B, Mohsenzadeh Y. Unveiling the neural dynamics of conscious perception in rapid object recognition. Neuroimage 2024; 296:120668. [PMID: 38848982 DOI: 10.1016/j.neuroimage.2024.120668] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/01/2023] [Revised: 05/23/2024] [Accepted: 06/05/2024] [Indexed: 06/09/2024] Open
Abstract
Our brain excels at recognizing objects, even when they flash by in a rapid sequence. However, the neural processes determining whether a target image in a rapid sequence can be recognized or not remains elusive. We used electroencephalography (EEG) to investigate the temporal dynamics of brain processes that shape perceptual outcomes in these challenging viewing conditions. Using naturalistic images and advanced multivariate pattern analysis (MVPA) techniques, we probed the brain dynamics governing conscious object recognition. Our results show that although initially similar, the processes for when an object can or cannot be recognized diverge around 180 ms post-appearance, coinciding with feedback neural processes. Decoding analyses indicate that gist perception (partial conscious perception) can occur at ∼120 ms through feedforward mechanisms. In contrast, object identification (full conscious perception of the image) is resolved at ∼190 ms after target onset, suggesting involvement of recurrent processing. These findings underscore the importance of recurrent neural connections in object recognition and awareness in rapid visual presentations.
Collapse
Affiliation(s)
- Saba Charmi Motlagh
- Western Center for Brain and Mind, Western University, London, Ontario, Canada; Vector Institute for Artificial Intelligence, Toronto, Ontario, Canada
| | - Marc Joanisse
- Western Center for Brain and Mind, Western University, London, Ontario, Canada; Department of Psychology, Western University, London, Ontario, Canada
| | - Boyu Wang
- Western Center for Brain and Mind, Western University, London, Ontario, Canada; Vector Institute for Artificial Intelligence, Toronto, Ontario, Canada; Department of Computer Science, Western University, London, Ontario, Canada
| | - Yalda Mohsenzadeh
- Western Center for Brain and Mind, Western University, London, Ontario, Canada; Vector Institute for Artificial Intelligence, Toronto, Ontario, Canada; Department of Computer Science, Western University, London, Ontario, Canada.
| |
Collapse
|
4
|
Wang T, Lee TS, Yao H, Hong J, Li Y, Jiang H, Andolina IM, Tang S. Large-scale calcium imaging reveals a systematic V4 map for encoding natural scenes. Nat Commun 2024; 15:6401. [PMID: 39080309 PMCID: PMC11289446 DOI: 10.1038/s41467-024-50821-z] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/15/2023] [Accepted: 07/22/2024] [Indexed: 08/02/2024] Open
Abstract
Biological visual systems have evolved to process natural scenes. A full understanding of visual cortical functions requires a comprehensive characterization of how neuronal populations in each visual area encode natural scenes. Here, we utilized widefield calcium imaging to record V4 cortical response to tens of thousands of natural images in male macaques. Using this large dataset, we developed a deep-learning digital twin of V4 that allowed us to map the natural image preferences of the neural population at 100-µm scale. This detailed map revealed a diverse set of functional domains in V4, each encoding distinct natural image features. We validated these model predictions using additional widefield imaging and single-cell resolution two-photon imaging. Feature attribution analysis revealed that these domains lie along a continuum from preferring spatially localized shape features to preferring spatially dispersed surface features. These results provide insights into the organizing principles that govern natural scene encoding in V4.
Collapse
Affiliation(s)
- Tianye Wang
- Peking University School of Life Sciences, Beijing, 100871, China
- Peking-Tsinghua Center for Life Sciences, Beijing, 100871, China
- IDG/McGovern Institute for Brain Research at Peking University, Beijing, 100871, China
- Key Laboratory of Machine Perception (Ministry of Education), Peking University, Beijing, 100871, China
| | - Tai Sing Lee
- Computer Science Department and Neuroscience Institute, Carnegie Mellon University, Pittsburgh, PA, 15213, USA
| | - Haoxuan Yao
- Peking University School of Life Sciences, Beijing, 100871, China
- Peking-Tsinghua Center for Life Sciences, Beijing, 100871, China
- IDG/McGovern Institute for Brain Research at Peking University, Beijing, 100871, China
- Key Laboratory of Machine Perception (Ministry of Education), Peking University, Beijing, 100871, China
| | - Jiayi Hong
- Peking University School of Life Sciences, Beijing, 100871, China
| | - Yang Li
- Peking University School of Life Sciences, Beijing, 100871, China
- Peking-Tsinghua Center for Life Sciences, Beijing, 100871, China
- IDG/McGovern Institute for Brain Research at Peking University, Beijing, 100871, China
- Key Laboratory of Machine Perception (Ministry of Education), Peking University, Beijing, 100871, China
| | - Hongfei Jiang
- Peking University School of Life Sciences, Beijing, 100871, China
- Peking-Tsinghua Center for Life Sciences, Beijing, 100871, China
- IDG/McGovern Institute for Brain Research at Peking University, Beijing, 100871, China
- Key Laboratory of Machine Perception (Ministry of Education), Peking University, Beijing, 100871, China
| | - Ian Max Andolina
- The Center for Excellence in Brain Science and Intelligence Technology, State Key Laboratory of Neuroscience, Key Laboratory of Primate Neurobiology, Institute of Neuroscience, Chinese Academy of Sciences, Shanghai, 200031, China
| | - Shiming Tang
- Peking University School of Life Sciences, Beijing, 100871, China.
- Peking-Tsinghua Center for Life Sciences, Beijing, 100871, China.
- IDG/McGovern Institute for Brain Research at Peking University, Beijing, 100871, China.
- Key Laboratory of Machine Perception (Ministry of Education), Peking University, Beijing, 100871, China.
| |
Collapse
|
5
|
Nau M, Schmid AC, Kaplan SM, Baker CI, Kravitz DJ. Centering cognitive neuroscience on task demands and generalization. Nat Neurosci 2024:10.1038/s41593-024-01711-6. [PMID: 39075326 DOI: 10.1038/s41593-024-01711-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/05/2023] [Accepted: 06/17/2024] [Indexed: 07/31/2024]
Abstract
Cognitive neuroscience seeks generalizable theories explaining the relationship between behavioral, physiological and mental states. In pursuit of such theories, we propose a theoretical and empirical framework that centers on understanding task demands and the mutual constraints they impose on behavior and neural activity. Task demands emerge from the interaction between an agent's sensory impressions, goals and behavior, which jointly shape the activity and structure of the nervous system on multiple spatiotemporal scales. Understanding this interaction requires multitask studies that vary more than one experimental component (for example, stimuli and instructions) combined with dense behavioral and neural sampling and explicit testing for generalization across tasks and data modalities. By centering task demands rather than mental processes that tasks are assumed to engage, this framework paves the way for the discovery of new generalizable concepts unconstrained by existing taxonomies, and moves cognitive neuroscience toward an action-oriented, dynamic and integrated view of the brain.
Collapse
Affiliation(s)
- Matthias Nau
- Laboratory of Brain and Cognition, National Institutes of Health, Bethesda, MD, USA.
| | - Alexandra C Schmid
- Laboratory of Brain and Cognition, National Institutes of Health, Bethesda, MD, USA
| | - Simon M Kaplan
- Department of Psychological & Brain Sciences, The George Washington University, Washington, DC, USA
| | - Chris I Baker
- Laboratory of Brain and Cognition, National Institutes of Health, Bethesda, MD, USA.
| | - Dwight J Kravitz
- Department of Psychological & Brain Sciences, The George Washington University, Washington, DC, USA.
- Division of Behavioral and Cognitive Sciences, Directorate for Social, Behavioral, and Economic Sciences, US National Science Foundation, Arlington, VA, USA.
| |
Collapse
|
6
|
Lee GM, Rodríguez Deliz CL, Bushnell BN, Majaj NJ, Movshon JA, Kiorpes L. Developmentally stable representations of naturalistic image structure in macaque visual cortex. Cell Rep 2024; 43:114534. [PMID: 39067025 DOI: 10.1016/j.celrep.2024.114534] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/01/2024] [Revised: 06/17/2024] [Accepted: 07/09/2024] [Indexed: 07/30/2024] Open
Abstract
To determine whether post-natal improvements in form vision result from changes in mid-level visual cortex, we studied neuronal and behavioral responses to texture stimuli that were matched in local spectral content but varied in "naturalistic" structure. We made longitudinal measurements of visual behavior from 16 to 95 weeks of age, and of neural responses from 20 to 56 weeks. We also measured behavioral and neural responses in near-adult animals more than 3 years old. Behavioral sensitivity reached half-maximum around 25 weeks of age, but neural sensitivities remained stable through all ages tested. Neural sensitivity to naturalistic structure was highest in V4, lower in V2 and inferotemporal cortex (IT), and barely discernible in V1. Our results show a dissociation between stable neural performance and improving behavioral performance, which may reflect improved processing capacity in circuits downstream of visual cortex.
Collapse
Affiliation(s)
- Gerick M Lee
- Center for Neural Science, New York University, New York, NY 10003, USA
| | | | | | - Najib J Majaj
- Center for Neural Science, New York University, New York, NY 10003, USA
| | - J Anthony Movshon
- Center for Neural Science, New York University, New York, NY 10003, USA.
| | - Lynne Kiorpes
- Center for Neural Science, New York University, New York, NY 10003, USA.
| |
Collapse
|
7
|
Lahner B, Dwivedi K, Iamshchinina P, Graumann M, Lascelles A, Roig G, Gifford AT, Pan B, Jin S, Ratan Murty NA, Kay K, Oliva A, Cichy R. Modeling short visual events through the BOLD moments video fMRI dataset and metadata. Nat Commun 2024; 15:6241. [PMID: 39048577 PMCID: PMC11269733 DOI: 10.1038/s41467-024-50310-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/14/2023] [Accepted: 07/04/2024] [Indexed: 07/27/2024] Open
Abstract
Studying the neural basis of human dynamic visual perception requires extensive experimental data to evaluate the large swathes of functionally diverse brain neural networks driven by perceiving visual events. Here, we introduce the BOLD Moments Dataset (BMD), a repository of whole-brain fMRI responses to over 1000 short (3 s) naturalistic video clips of visual events across ten human subjects. We use the videos' extensive metadata to show how the brain represents word- and sentence-level descriptions of visual events and identify correlates of video memorability scores extending into the parietal cortex. Furthermore, we reveal a match in hierarchical processing between cortical regions of interest and video-computable deep neural networks, and we showcase that BMD successfully captures temporal dynamics of visual events at second resolution. With its rich metadata, BMD offers new perspectives and accelerates research on the human brain basis of visual event perception.
Collapse
Affiliation(s)
- Benjamin Lahner
- Computer Science and Artificial Intelligence Laboratory, MIT, Cambridge, MA, USA.
| | - Kshitij Dwivedi
- Department of Education and Psychology, Freie Universität Berlin, Berlin, Germany
- Department of Computer Science, Goethe University Frankfurt, Frankfurt am Main, Germany
| | - Polina Iamshchinina
- Department of Education and Psychology, Freie Universität Berlin, Berlin, Germany
| | - Monika Graumann
- Department of Education and Psychology, Freie Universität Berlin, Berlin, Germany
| | - Alex Lascelles
- Computer Science and Artificial Intelligence Laboratory, MIT, Cambridge, MA, USA
| | - Gemma Roig
- Department of Computer Science, Goethe University Frankfurt, Frankfurt am Main, Germany
- The Hessian Center for AI (hessian.AI), Darmstadt, Germany
| | | | - Bowen Pan
- Computer Science and Artificial Intelligence Laboratory, MIT, Cambridge, MA, USA
| | - SouYoung Jin
- Computer Science and Artificial Intelligence Laboratory, MIT, Cambridge, MA, USA
| | - N Apurva Ratan Murty
- Department of Brain and Cognitive Science, MIT, Cambridge, MA, USA
- School of Psychology, Georgia Institute of Technology, Atlanta, GA, USA
| | - Kendrick Kay
- Center for Magnetic Resonance Research (CMRR), Department of Radiology, University of Minnesota, Minneapolis, MN, USA
| | - Aude Oliva
- Computer Science and Artificial Intelligence Laboratory, MIT, Cambridge, MA, USA
| | - Radoslaw Cichy
- Department of Education and Psychology, Freie Universität Berlin, Berlin, Germany
| |
Collapse
|
8
|
Bravo F, Glogowski J, Stamatakis EA, Herfert K. Dissonant music engages early visual processing. Proc Natl Acad Sci U S A 2024; 121:e2320378121. [PMID: 39008675 PMCID: PMC11287129 DOI: 10.1073/pnas.2320378121] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/01/2023] [Accepted: 06/04/2024] [Indexed: 07/17/2024] Open
Abstract
The neuroscientific examination of music processing in audio-visual contexts offers a valuable framework to assess how auditory information influences the emotional encoding of visual information. Using fMRI during naturalistic film viewing, we investigated the neural mechanisms underlying the effect of music on valence inferences during mental state attribution. Thirty-eight participants watched the same short-film accompanied by systematically controlled consonant or dissonant music. Subjects were instructed to think about the main character's intentions. The results revealed that increasing levels of dissonance led to more negatively valenced inferences, displaying the profound emotional impact of musical dissonance. Crucially, at the neuroscientific level and despite music being the sole manipulation, dissonance evoked the response of the primary visual cortex (V1). Functional/effective connectivity analysis showed a stronger coupling between the auditory ventral stream (AVS) and V1 in response to tonal dissonance and demonstrated the modulation of early visual processing via top-down feedback inputs from the AVS to V1. These V1 signal changes indicate the influence of high-level contextual representations associated with tonal dissonance on early visual cortices, serving to facilitate the emotional interpretation of visual information. Our results highlight the significance of employing systematically controlled music, which can isolate emotional valence from the arousal dimension, to elucidate the brain's sound-to-meaning interface and its distributive crossmodal effects on early visual encoding during naturalistic film viewing.
Collapse
Affiliation(s)
- Fernando Bravo
- Department of Preclinical Imaging and Radiopharmacy, University of Tübingen, Tübingen72076, Germany
- Cognition and Consciousness Imaging Group, Division of Anaesthesia, Department of Medicine, University of Cambridge, Addenbrooke’s Hospital, CambridgeCB2 0SP, United Kingdom
- Department of Clinical Neurosciences, University of Cambridge, Addenbrooke’s Hospital, CambridgeCB2 0SP, United Kingdom
- Institut für Kunst- und Musikwissenschaft, Division of Musicology, Technische Universität Dresden, Dresden01219, Germany
| | - Jana Glogowski
- Department of Psychology, Humboldt-Universität zu Berlin, Berlin12489, Germany
| | - Emmanuel Andreas Stamatakis
- Cognition and Consciousness Imaging Group, Division of Anaesthesia, Department of Medicine, University of Cambridge, Addenbrooke’s Hospital, CambridgeCB2 0SP, United Kingdom
- Department of Clinical Neurosciences, University of Cambridge, Addenbrooke’s Hospital, CambridgeCB2 0SP, United Kingdom
| | - Kristina Herfert
- Department of Preclinical Imaging and Radiopharmacy, University of Tübingen, Tübingen72076, Germany
| |
Collapse
|
9
|
Qi Z, Xiong H, Zhuo J, Cao D, Liu H, Shi W, Lang Y, Liu Y, Zhang G, Jiang T. Intracranial EEGs evidenced visual object processing in the human medial temporal lobe subregions. Neuroscience 2024; 555:205-212. [PMID: 39053670 DOI: 10.1016/j.neuroscience.2024.07.030] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/30/2023] [Revised: 07/10/2024] [Accepted: 07/18/2024] [Indexed: 07/27/2024]
Abstract
The perirhinal cortex (PRC) and parahippocampal cortex (PHC) are core regions along the visual dual-stream. The specific functional roles of the PRC and PHC and their interactions with the downstream hippocampus cortex (HPC) are crucial for understanding visual memory. Our research used human intracranial EEGs to study the neural mechanism of the PRC, PHC, and HPC in visual object encoding. Single-regional function analyses found evidence that the PRC, PHC, and HPC are activated ∼100 ms within the broad-gamma band and that the PRC was more strongly activated than either the PHC or the HPC after an object stimulus. Inter-regional analyses showed strong bidirectional interactions of the PRC with both the PHC and HPC in the low-frequency band, whereas the interactions between the PHC and HPC were not significant. These findings demonstrated the core role of the PRC in encoding visual object information and supported the hypothesis of PRC-HPC-ventral object pathway. The recruitment of the PHC and its interaction with the PRC in visual object encoding also provide new insights beyond the traditional dorsal-stream hypothesis.
Collapse
Affiliation(s)
- Zihui Qi
- Brainnetome Center, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China; University of Chinese Academy of Sciences, Beijing 100049, China
| | - Hui Xiong
- Brainnetome Center, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China
| | - Junjie Zhuo
- Key Laboratory of Biomedical Engineering of Hainan Province, School of Biomedical Engineering, Hainan University, Hainan 570228, China
| | - Dan Cao
- Brainnetome Center, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China
| | - Hao Liu
- Brainnetome Center, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China; University of Chinese Academy of Sciences, Beijing 100049, China
| | - Weiyang Shi
- Brainnetome Center, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China
| | - Yongcui Lang
- Department of Neurosurgery, Epilepsy Center, Aviation General Hospital, No. 3 Beiyuan Road, Chaoyang District, Beijing 100012, China
| | - Yaoling Liu
- Department of Neurosurgery, Epilepsy Center, Aviation General Hospital, No. 3 Beiyuan Road, Chaoyang District, Beijing 100012, China
| | - Guangming Zhang
- Department of Neurosurgery, Epilepsy Center, Aviation General Hospital, No. 3 Beiyuan Road, Chaoyang District, Beijing 100012, China.
| | - Tianzi Jiang
- Brainnetome Center, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China; University of Chinese Academy of Sciences, Beijing 100049, China; Xiaoxiang Institute for Brain Health and Yongzhou Central Hospital, Yongzhou 425000, Hunan Province, China.
| |
Collapse
|
10
|
Tye KM, Miller EK, Taschbach FH, Benna MK, Rigotti M, Fusi S. Mixed selectivity: Cellular computations for complexity. Neuron 2024; 112:2289-2303. [PMID: 38729151 PMCID: PMC11257803 DOI: 10.1016/j.neuron.2024.04.017] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/11/2023] [Revised: 03/08/2024] [Accepted: 04/12/2024] [Indexed: 05/12/2024]
Abstract
The property of mixed selectivity has been discussed at a computational level and offers a strategy to maximize computational power by adding versatility to the functional role of each neuron. Here, we offer a biologically grounded implementational-level mechanistic explanation for mixed selectivity in neural circuits. We define pure, linear, and nonlinear mixed selectivity and discuss how these response properties can be obtained in simple neural circuits. Neurons that respond to multiple, statistically independent variables display mixed selectivity. If their activity can be expressed as a weighted sum, then they exhibit linear mixed selectivity; otherwise, they exhibit nonlinear mixed selectivity. Neural representations based on diverse nonlinear mixed selectivity are high dimensional; hence, they confer enormous flexibility to a simple downstream readout neural circuit. However, a simple neural circuit cannot possibly encode all possible mixtures of variables simultaneously, as this would require a combinatorially large number of mixed selectivity neurons. Gating mechanisms like oscillations and neuromodulation can solve this problem by dynamically selecting which variables are mixed and transmitted to the readout.
Collapse
Affiliation(s)
- Kay M Tye
- Salk Institute for Biological Studies, La Jolla, CA, USA; Howard Hughes Medical Institute, La Jolla, CA; Department of Neurobiology, School of Biological Sciences, University of California, San Diego, La Jolla, CA 92093, USA; Kavli Institute for Brain and Mind, San Diego, CA, USA.
| | - Earl K Miller
- The Picower Institute for Learning and Memory, Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA 02139, USA.
| | - Felix H Taschbach
- Salk Institute for Biological Studies, La Jolla, CA, USA; Biological Science Graduate Program, University of California, San Diego, La Jolla, CA 92093, USA; Department of Neurobiology, School of Biological Sciences, University of California, San Diego, La Jolla, CA 92093, USA.
| | - Marcus K Benna
- Department of Neurobiology, School of Biological Sciences, University of California, San Diego, La Jolla, CA 92093, USA.
| | | | - Stefano Fusi
- Center for Theoretical Neuroscience, Columbia University, New York, NY, USA; Zuckerman Mind Brain Behavior Institute, Columbia University, New York, NY, USA; Department of Neuroscience, Columbia University, New York, NY, USA; Kavli Institute for Brain Science, Columbia University, New York, NY, USA.
| |
Collapse
|
11
|
Quaia C, Krauzlis RJ. Object recognition in primates: what can early visual areas contribute? Front Behav Neurosci 2024; 18:1425496. [PMID: 39070778 PMCID: PMC11272660 DOI: 10.3389/fnbeh.2024.1425496] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/29/2024] [Accepted: 07/01/2024] [Indexed: 07/30/2024] Open
Abstract
Introduction If neuroscientists were asked which brain area is responsible for object recognition in primates, most would probably answer infero-temporal (IT) cortex. While IT is likely responsible for fine discriminations, and it is accordingly dominated by foveal visual inputs, there is more to object recognition than fine discrimination. Importantly, foveation of an object of interest usually requires recognizing, with reasonable confidence, its presence in the periphery. Arguably, IT plays a secondary role in such peripheral recognition, and other visual areas might instead be more critical. Methods To investigate how signals carried by early visual processing areas (such as LGN and V1) could be used for object recognition in the periphery, we focused here on the task of distinguishing faces from non-faces. We tested how sensitive various models were to nuisance parameters, such as changes in scale and orientation of the image, and the type of image background. Results We found that a model of V1 simple or complex cells could provide quite reliable information, resulting in performance better than 80% in realistic scenarios. An LGN model performed considerably worse. Discussion Because peripheral recognition is both crucial to enable fine recognition (by bringing an object of interest on the fovea), and probably sufficient to account for a considerable fraction of our daily recognition-guided behavior, we think that the current focus on area IT and foveal processing is too narrow. We propose that rather than a hierarchical system with IT-like properties as its primary aim, object recognition should be seen as a parallel process, with high-accuracy foveal modules operating in parallel with lower-accuracy and faster modules that can operate across the visual field.
Collapse
Affiliation(s)
- Christian Quaia
- Laboratory of Sensorimotor Research, National Eye Institute, NIH, Bethesda, MD, United States
| | | |
Collapse
|
12
|
Contier O, Baker CI, Hebart MN. Distributed representations of behavior-derived object dimensions in the human visual system. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2024:2023.08.23.553812. [PMID: 37662312 PMCID: PMC10473665 DOI: 10.1101/2023.08.23.553812] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 09/05/2023]
Abstract
Object vision is commonly thought to involve a hierarchy of brain regions processing increasingly complex image features, with high-level visual cortex supporting object recognition and categorization. However, object vision supports diverse behavioral goals, suggesting basic limitations of this category-centric framework. To address these limitations, we mapped a series of dimensions derived from a large-scale analysis of human similarity judgments directly onto the brain. Our results reveal broadly distributed representations of behaviorally-relevant information, demonstrating selectivity to a wide variety of novel dimensions while capturing known selectivities for visual features and categories. Behavior-derived dimensions were superior to categories at predicting brain responses, yielding mixed selectivity in much of visual cortex and sparse selectivity in category-selective clusters. This framework reconciles seemingly disparate findings regarding regional specialization, explaining category selectivity as a special case of sparse response profiles among representational dimensions, suggesting a more expansive view on visual processing in the human brain.
Collapse
Affiliation(s)
- O Contier
- Vision and Computational Cognition Group, Max Planck Institute for Human Cognitive and Brain Sciences, Leipzig, Germany
- Max Planck School of Cognition, Leipzig, Germany
| | - C I Baker
- Laboratory of Brain and Cognition, National Institute of Mental Health, National Institutes of Health, Bethesda MD, USA
| | - M N Hebart
- Vision and Computational Cognition Group, Max Planck Institute for Human Cognitive and Brain Sciences, Leipzig, Germany
- Department of Medicine, Justus Liebig University Giessen, Giessen, Germany
| |
Collapse
|
13
|
Hirabayashi T, Nagai Y, Hori Y, Hori Y, Oyama K, Mimura K, Miyakawa N, Iwaoki H, Inoue KI, Suhara T, Takada M, Higuchi M, Minamimoto T. Multiscale chemogenetic dissection of fronto-temporal top-down regulation for object memory in primates. Nat Commun 2024; 15:5369. [PMID: 38987235 PMCID: PMC11237144 DOI: 10.1038/s41467-024-49570-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/17/2023] [Accepted: 06/07/2024] [Indexed: 07/12/2024] Open
Abstract
Visual object memory is a fundamental element of various cognitive abilities, and the underlying neural mechanisms have been extensively examined especially in the anterior temporal cortex of primates. However, both macroscopic large-scale functional network in which this region is embedded and microscopic neuron-level dynamics of top-down regulation it receives for object memory remains elusive. Here, we identified the orbitofrontal node as a critical partner of the anterior temporal node for object memory by combining whole-brain functional imaging during rest and a short-term object memory task in male macaques. Focal chemogenetic silencing of the identified orbitofrontal node downregulated both the local orbitofrontal and remote anterior temporal nodes during the task, in association with deteriorated mnemonic, but not perceptual, performance. Furthermore, imaging-guided neuronal recordings in the same monkeys during the same task causally revealed that orbitofrontal top-down modulation enhanced stimulus-selective mnemonic signal in individual anterior temporal neurons while leaving bottom-up perceptual signal unchanged. Furthermore, similar activity difference was also observed between correct and mnemonic error trials before silencing, suggesting its behavioral relevance. These multifaceted but convergent results provide a multiscale causal understanding of dynamic top-down regulation of the anterior temporal cortex along the ventral fronto-temporal network underpinning short-term object memory in primates.
Collapse
Affiliation(s)
- Toshiyuki Hirabayashi
- Advanced Neuroimaging Center, National Institutes for Quantum Science and Technology, Chiba, 263-8555, Japan.
| | - Yuji Nagai
- Advanced Neuroimaging Center, National Institutes for Quantum Science and Technology, Chiba, 263-8555, Japan
| | - Yuki Hori
- Advanced Neuroimaging Center, National Institutes for Quantum Science and Technology, Chiba, 263-8555, Japan
| | - Yukiko Hori
- Advanced Neuroimaging Center, National Institutes for Quantum Science and Technology, Chiba, 263-8555, Japan
| | - Kei Oyama
- Advanced Neuroimaging Center, National Institutes for Quantum Science and Technology, Chiba, 263-8555, Japan
| | - Koki Mimura
- Advanced Neuroimaging Center, National Institutes for Quantum Science and Technology, Chiba, 263-8555, Japan
| | - Naohisa Miyakawa
- Advanced Neuroimaging Center, National Institutes for Quantum Science and Technology, Chiba, 263-8555, Japan
| | - Haruhiko Iwaoki
- Advanced Neuroimaging Center, National Institutes for Quantum Science and Technology, Chiba, 263-8555, Japan
| | - Ken-Ichi Inoue
- Center for the Evolutionary Origins of Human Behavior, Kyoto University, Inuyama, Aichi, 484-8506, Japan
| | - Tetsuya Suhara
- Advanced Neuroimaging Center, National Institutes for Quantum Science and Technology, Chiba, 263-8555, Japan
| | - Masahiko Takada
- Center for the Evolutionary Origins of Human Behavior, Kyoto University, Inuyama, Aichi, 484-8506, Japan
| | - Makoto Higuchi
- Advanced Neuroimaging Center, National Institutes for Quantum Science and Technology, Chiba, 263-8555, Japan
| | - Takafumi Minamimoto
- Advanced Neuroimaging Center, National Institutes for Quantum Science and Technology, Chiba, 263-8555, Japan
| |
Collapse
|
14
|
Asaad WF, Sheth SA. What's the n? On sample size vs. subject number for brain-behavior neurophysiology and neuromodulation. Neuron 2024; 112:2086-2090. [PMID: 38781973 DOI: 10.1016/j.neuron.2024.04.033] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/01/2024] [Revised: 04/01/2024] [Accepted: 04/26/2024] [Indexed: 05/25/2024]
Abstract
Neurophysiology and neuromodulation strive to understand the neural basis of behavior through a one-to-one correspondence between a particular brain and its behavioral output. Within this framework, studies with few subjects but sufficient sample sizes can be both rigorous and impactful.
Collapse
Affiliation(s)
- Wael F Asaad
- Department of Neuroscience, Brown University, Providence, RI, USA; Department of Neurosurgery, Brown University Alpert Medical School, Providence, RI, USA; Carney Institute for Brain Science, Brown University, Providence, RI, USA; Norman Prince Neurosciences Institute, Rhode Island Hospital, Providence, RI, USA.
| | - Sameer A Sheth
- Department of Neurosurgery, Baylor College of Medicine, Houston, TX, USA; Jan and Dan Duncan Neurological Research Institute, Baylor College of Medicine, Houston, TX, USA
| |
Collapse
|
15
|
Ostojic S, Fusi S. Computational role of structure in neural activity and connectivity. Trends Cogn Sci 2024; 28:677-690. [PMID: 38553340 DOI: 10.1016/j.tics.2024.03.003] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/24/2023] [Revised: 02/29/2024] [Accepted: 03/07/2024] [Indexed: 07/05/2024]
Abstract
One major challenge of neuroscience is identifying structure in seemingly disorganized neural activity. Different types of structure have different computational implications that can help neuroscientists understand the functional role of a particular brain area. Here, we outline a unified approach to characterize structure by inspecting the representational geometry and the modularity properties of the recorded activity and show that a similar approach can also reveal structure in connectivity. We start by setting up a general framework for determining geometry and modularity in activity and connectivity and relating these properties with computations performed by the network. We then use this framework to review the types of structure found in recent studies of model networks performing three classes of computations.
Collapse
Affiliation(s)
- Srdjan Ostojic
- Laboratoire de Neurosciences Cognitives et Computationnelles, INSERM U960, Ecole Normale Superieure - PSL Research University, 75005 Paris, France.
| | - Stefano Fusi
- Center for Theoretical Neuroscience, Columbia University, New York, NY, USA; Zuckerman Mind Brain Behavior Institute, Columbia University, New York, NY, USA; Department of Neuroscience, Columbia University, New York, NY, USA; Kavli Institute for Brain Science, Columbia University, New York, NY, USA
| |
Collapse
|
16
|
He D, Ogmen H. The roles of symmetry and elongation in developing reference frames. Front Psychol 2024; 15:1402156. [PMID: 39011287 PMCID: PMC11249022 DOI: 10.3389/fpsyg.2024.1402156] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/17/2024] [Accepted: 06/11/2024] [Indexed: 07/17/2024] Open
Abstract
Previous studies showed that elongation and symmetry (two ubiquitous aspects of natural stimuli) are important attributes in object perception and recognition, which in turn suggests that these geometrical factors may contribute to the selection of perceptual reference-frames. However, whether and how these attributes guide the selection of reference-frames is still poorly understood. The goal of this study was to examine systematically the roles of elongation and symmetry, as well as their combination, in the selection of reference axis and how these axes are developed for unfamiliar objects. We designed our experiments to eliminate two potential confounding factors: (i) extraneous environmental cues, such as edges of the screen, etc. (by using VR) and (ii) pre-learned cues for familiar objects and shapes (by using reinforcement learning of novel shapes). We used algorithmically generated textures with different orientations having specified levels of symmetry and elongation as the stimuli. In each trial, we presented only one stimulus and asked observers to report if the stimulus was in its original form or a flipped (mirror-image) one. Feedback was provided at the end of each trial. Based on previous studies on mental rotation, we hypothesized that the selection of a reference-frame defined by symmetry and/or elongation would be revealed by a linear relationship between reaction-times and the angular-deviation from either the most symmetrical or the most elongated orientation. Our results are consistent with this hypothesis. We found that subjects performed mental rotation to transform images to their reference axes and used the most symmetrical or elongated orientation as the reference axis when only one factor was presented, and they used a "winner-take-all" strategy when both factors were presented, with elongation being more dominant than symmetry. We discuss theoretical implications of these findings, in particular in the context of "canonical sensorimotor theory."
Collapse
Affiliation(s)
- Dongcheng He
- Department of Electrical and Computer Engineering, Ritchie School of Engineering and Computer Science, University of Denver, Denver, CO, United States
- Herbert Wertheim School of Optometry and Vision Science, University of California, Berkeley, Berkeley, CA, United States
| | - Haluk Ogmen
- Department of Electrical and Computer Engineering, Ritchie School of Engineering and Computer Science, University of Denver, Denver, CO, United States
| |
Collapse
|
17
|
Jahani Yekta MM. Biological direct-shortcut deep residual learning for sparse visual processing. Sci Rep 2024; 14:14066. [PMID: 38890361 PMCID: PMC11189431 DOI: 10.1038/s41598-024-62756-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/05/2023] [Accepted: 05/21/2024] [Indexed: 06/20/2024] Open
Abstract
We show, based on the following three grounds, that the primary visual cortex (V1) is a biological direct-shortcut deep residual learning neural network (ResNet) for sparse visual processing: (1) We first highlight that Gabor-like sets of basis functions, which are similar to the receptive fields of simple cells in the primary visual cortex (V1), are excellent candidates for sparse representation of natural images; i.e., images from the natural world, affirming the brain to be optimized for this. (2) We then prove that the intra-layer synaptic weight matrices of this region can be reasonably first-order approximated by identity mappings, and are thus sparse themselves. (3) Finally, we point out that intra-layer weight matrices having identity mapping as their initial approximation, irrespective of this approximation being also a reasonable first-order one or not, resemble the building blocks of direct-shortcut digital ResNets, which completes the grounds. This biological ResNet interconnects the sparsity of the final representation of the image to that of its intra-layer weights. Further exploration of this ResNet, and understanding the joint effects of its architecture and learning rules, e.g. on its inductive bias, could lead to major advancements in the area of bio-inspired digital ResNets. One immediate line of research in this context, for instance, is to study the impact of forcing the direct-shortcuts to be good first-order approximations of each building block. For this, along with the ℓ 1 -minimization posed on the basis function coefficients the ResNet finally provides at its output, another parallel one could e.g. also be posed on the weights of its residual layers.
Collapse
|
18
|
Li Y, Yang H, Gu S. Enhancing neural encoding models for naturalistic perception with a multi-level integration of deep neural networks and cortical networks. Sci Bull (Beijing) 2024; 69:1738-1747. [PMID: 38490889 DOI: 10.1016/j.scib.2024.02.035] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/01/2023] [Revised: 06/27/2023] [Accepted: 02/23/2024] [Indexed: 03/17/2024]
Abstract
Cognitive neuroscience aims to develop computational models that can accurately predict and explain neural responses to sensory inputs in the cortex. Recent studies attempt to leverage the representation power of deep neural networks (DNNs) to predict the brain response and suggest a correspondence between artificial and biological neural networks in their feature representations. However, typical voxel-wise encoding models tend to rely on specific networks designed for computer vision tasks, leading to suboptimal brain-wide correspondence during cognitive tasks. To address this challenge, this work proposes a novel approach that upgrades voxel-wise encoding models through multi-level integration of features from DNNs and information from brain networks. Our approach combines DNN feature-level ensemble learning and brain atlas-level model integration, resulting in significant improvements in predicting whole-brain neural activity during naturalistic video perception. Furthermore, this multi-level integration framework enables a deeper understanding of the brain's neural representation mechanism, accurately predicting the neural response to complex visual concepts. We demonstrate that neural encoding models can be optimized by leveraging a framework that integrates both data-driven approaches and theoretical insights into the functional structure of the cortical networks.
Collapse
Affiliation(s)
- Yuanning Li
- School of Biomedical Engineering & State Key Laboratory of Advanced Medical Materials and Devices, ShanghaiTech University, Shanghai 201210, China.
| | - Huzheng Yang
- School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu 611731, China; Department of Computer and Information Science, University of Pennsylvania, Philadelphia, PA 19104, USA
| | - Shi Gu
- School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu 611731, China; Shenzhen Institute for Advanced Study, University of Electronic Science and Technology of China, Shenzhen 518110, China.
| |
Collapse
|
19
|
De Agrò M, Rößler DC, Shamble PS. Eye-specific detection and a multi-eye integration model of biological motion perception. J Exp Biol 2024; 227:jeb247061. [PMID: 38752337 DOI: 10.1242/jeb.247061] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/21/2023] [Accepted: 05/07/2024] [Indexed: 06/27/2024]
Abstract
'Biological motion' refers to the distinctive kinematics observed in many living organisms, where visually perceivable points on the animal move at fixed distances from each other. Across the animal kingdom, many species have developed specialized visual circuitry to recognize such biological motion and to discriminate it from other patterns. Recently, this ability has been observed in the distributed visual system of jumping spiders. These eight-eyed animals use six eyes to perceive motion, while the remaining two (the principal anterior medial eyes) are shifted across the visual scene to further inspect detected objects. When presented with a biologically moving stimulus and a random one, jumping spiders turn to face the latter, clearly demonstrating the ability to discriminate between them. However, it remains unclear whether the principal eyes are necessary for this behavior, whether all secondary eyes can perform this discrimination, or whether a single eye-pair is specialized for this task. Here, we systematically tested the ability of jumping spiders to discriminate between biological and random visual stimuli by testing each eye-pair alone. Spiders were able to discriminate stimuli only when the anterior lateral eyes were unblocked, and performed at chance levels in other configurations. Interestingly, spiders showed a preference for biological motion over random stimuli - unlike in past work. We therefore propose a new model describing how specialization of the anterior lateral eyes for detecting biological motion contributes to multi-eye integration in this system. This integration generates more complex behavior through the combination of simple, single-eye responses. We posit that this in-built modularity may be a solution to the limited resources of these invertebrates' brains, constituting a novel approach to visual processing.
Collapse
Affiliation(s)
- Massimo De Agrò
- Faculty of Biology, University of Regensburg, 93053 Regensburg, Germany
- Department of Biology, University of Florence, 50121 Firenze, Italy
- The BioRobotics Institute, Sant'Anna School of Advanced Studies, 56127 Pisa, Italy
| | - Daniela C Rößler
- Zukunftskolleg, Konstanz University, 78464 Konstanz, Germany
- Department of Biology, Konstanz University, 78464 Konstanz, Germany
- Department of Ecology of Animal Societies, Max Planck Institute of Animal Behavior, 78464 Konstanz, Germany
| | - Paul S Shamble
- Kavli Institute for Neuroscience, Department of Neuroscience, Yale University School of Medicine, New Haven, CT 06510, USA
| |
Collapse
|
20
|
Kim JH, Daie K, Li N. A combinatorial neural code for long-term motor memory. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2024:2024.06.05.597627. [PMID: 38895416 PMCID: PMC11185691 DOI: 10.1101/2024.06.05.597627] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/21/2024]
Abstract
Motor skill repertoire can be stably retained over long periods, but the neural mechanism underlying stable memory storage remains poorly understood. Moreover, it is unknown how existing motor memories are maintained as new motor skills are continuously acquired. Here we tracked neural representation of learned actions throughout a significant portion of a mouse's lifespan, and we show that learned actions are stably retained in motor memory in combination with context, which protects existing memories from erasure during new motor learning. We used automated home-cage training to establish a continual learning paradigm in which mice learned to perform directional licking in different task contexts. We combined this paradigm with chronic two-photon imaging of motor cortex activity for up to 6 months. Within the same task context, activity driving directional licking was stable over time with little representational drift. When learning new task contexts, new preparatory activity emerged to drive the same licking actions. Learning created parallel new motor memories while retaining the previous memories. Re-learning to make the same actions in the previous task context re-activated the previous preparatory activity, even months later. At the same time, continual learning of new task contexts kept creating new preparatory activity patterns. Context-specific memories, as we observed in the motor system, may provide a solution for stable memory storage throughout continual learning. Learning in new contexts produces parallel new representations instead of modifying existing representations, thus protecting existing motor repertoire from erasure.
Collapse
|
21
|
Srinath R, Ni AM, Marucci C, Cohen MR, Brainard DH. Orthogonal neural representations support perceptual judgements of natural stimuli. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2024:2024.02.14.580134. [PMID: 38464018 PMCID: PMC10925131 DOI: 10.1101/2024.02.14.580134] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 03/12/2024]
Abstract
In natural behavior, observers must separate relevant information from a barrage of irrelevant information. Many studies have investigated the neural underpinnings of this ability using artificial stimuli presented on simple backgrounds. Natural viewing, however, carries a set of challenges that are inaccessible using artificial stimuli, including neural responses to background objects that are task-irrelevant. An emerging body of evidence suggests that the visual abilities of humans and animals can be modeled through the linear decoding of task-relevant information from visual cortex. This idea suggests the hypothesis that irrelevant features of a natural scene should impair performance on a visual task only if their neural representations intrude on the linear readout of the task relevant feature, as would occur if the representations of task-relevant and irrelevant features are not orthogonal in the underlying neural population. We tested this hypothesis using human psychophysics and monkey neurophysiology, in response to parametrically variable naturalistic stimuli. We demonstrate that 1) the neural representation of one feature (the position of a central object) in visual area V4 is orthogonal to those of several background features, 2) the ability of human observers to precisely judge object position was largely unaffected by task-irrelevant variation in those background features, and 3) many features of the object and the background are orthogonally represented by V4 neural responses. Our observations are consistent with the hypothesis that orthogonal neural representations can support stable perception of objects and features despite the tremendous richness of natural visual scenes.
Collapse
Affiliation(s)
- Ramanujan Srinath
- equal contribution
- Department of Neurobiology and Neuroscience Institute, The University of Chicago, Chicago, IL 60637, USA
| | - Amy M. Ni
- equal contribution
- Department of Neurobiology and Neuroscience Institute, The University of Chicago, Chicago, IL 60637, USA
- Department of Psychology, University of Pennsylvania, Philadelphia, PA 19104, USA
| | - Claire Marucci
- Department of Psychology, University of Pennsylvania, Philadelphia, PA 19104, USA
| | - Marlene R. Cohen
- Department of Neurobiology and Neuroscience Institute, The University of Chicago, Chicago, IL 60637, USA
- equal contribution
| | - David H. Brainard
- Department of Psychology, University of Pennsylvania, Philadelphia, PA 19104, USA
- equal contribution
| |
Collapse
|
22
|
De Rosa M, Vignali L, D’Urso A, Ktori M, Bottini R, Crepaldi D. Selective Neural Entrainment Reveals Hierarchical Tuning to Linguistic Regularities in Reading. NEUROBIOLOGY OF LANGUAGE (CAMBRIDGE, MASS.) 2024; 5:528-552. [PMID: 38911459 PMCID: PMC11192515 DOI: 10.1162/nol_a_00145] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 07/14/2023] [Accepted: 03/20/2024] [Indexed: 06/25/2024]
Abstract
Reading is both a visual and a linguistic task, and as such it relies on both general-purpose, visual mechanisms and more abstract, meaning-oriented processes. Disentangling the roles of these resources is of paramount importance in reading research. The present study capitalizes on the coupling of fast periodic visual stimulation and MEG recordings to address this issue and investigate the role of different kinds of visual and linguistic units in the visual word identification system. We compared strings of pseudo-characters; strings of consonants (e.g., sfcl); readable, but unattested strings (e.g., amsi); frequent, but non-meaningful chunks (e.g., idge); suffixes (e.g., ment); and words (e.g., vibe); and looked for discrimination responses with a particular focus on the ventral, occipito-temporal regions. The results revealed sensitivity to alphabetic, readable, familiar, and lexical stimuli. Interestingly, there was no discrimination between suffixes and equally frequent, but meaningless endings, thus highlighting a lack of sensitivity to semantics. Taken together, the data suggest that the visual word identification system, at least in its early processing stages, is particularly tuned to form-based regularities, most likely reflecting its reliance on general-purpose, statistical learning mechanisms that are a core feature of the visual system as implemented in the ventral stream.
Collapse
Affiliation(s)
- Mara De Rosa
- Cognitive Neuroscience Department, International School for Advanced Studies, Trieste, Italy
| | - Lorenzo Vignali
- Center for Mind/Brain Sciences (CIMeC), University of Trento, Mattarello, Trento, Italy
| | - Anna D’Urso
- Center for Mind/Brain Sciences (CIMeC), University of Trento, Mattarello, Trento, Italy
| | - Maria Ktori
- Cognitive Neuroscience Department, International School for Advanced Studies, Trieste, Italy
| | - Roberto Bottini
- Center for Mind/Brain Sciences (CIMeC), University of Trento, Mattarello, Trento, Italy
| | - Davide Crepaldi
- Cognitive Neuroscience Department, International School for Advanced Studies, Trieste, Italy
| |
Collapse
|
23
|
Ahn S, Adeli H, Zelinsky GJ. The attentive reconstruction of objects facilitates robust object recognition. PLoS Comput Biol 2024; 20:e1012159. [PMID: 38870125 PMCID: PMC11175536 DOI: 10.1371/journal.pcbi.1012159] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/08/2023] [Accepted: 05/11/2024] [Indexed: 06/15/2024] Open
Abstract
Humans are extremely robust in our ability to perceive and recognize objects-we see faces in tea stains and can recognize friends on dark streets. Yet, neurocomputational models of primate object recognition have focused on the initial feed-forward pass of processing through the ventral stream and less on the top-down feedback that likely underlies robust object perception and recognition. Aligned with the generative approach, we propose that the visual system actively facilitates recognition by reconstructing the object hypothesized to be in the image. Top-down attention then uses this reconstruction as a template to bias feedforward processing to align with the most plausible object hypothesis. Building on auto-encoder neural networks, our model makes detailed hypotheses about the appearance and location of the candidate objects in the image by reconstructing a complete object representation from potentially incomplete visual input due to noise and occlusion. The model then leverages the best object reconstruction, measured by reconstruction error, to direct the bottom-up process of selectively routing low-level features, a top-down biasing that captures a core function of attention. We evaluated our model using the MNIST-C (handwritten digits under corruptions) and ImageNet-C (real-world objects under corruptions) datasets. Not only did our model achieve superior performance on these challenging tasks designed to approximate real-world noise and occlusion viewing conditions, but also better accounted for human behavioral reaction times and error patterns than a standard feedforward Convolutional Neural Network. Our model suggests that a complete understanding of object perception and recognition requires integrating top-down and attention feedback, which we propose is an object reconstruction.
Collapse
Affiliation(s)
- Seoyoung Ahn
- Department of Molecular and Cell Biology, University of California, Berkeley, California, United States of America
| | - Hossein Adeli
- Zuckerman Mind Brain Behavior Institute, Columbia University, New York City, New York, United States of America
| | - Gregory J. Zelinsky
- Department of Psychology, Stony Brook University, Stony Brook, New York, United States of America
- Department of Computer Science, Stony Brook University, Stony Brook, New York, United States of America
| |
Collapse
|
24
|
Lim HY, Lee I. Subpopulations of neurons in the perirhinal cortex enable both modality-specific and modality-invariant recognition of objects. PLoS Biol 2024; 22:e3002713. [PMID: 38924050 PMCID: PMC11233021 DOI: 10.1371/journal.pbio.3002713] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/22/2023] [Revised: 07/09/2024] [Accepted: 06/14/2024] [Indexed: 06/28/2024] Open
Abstract
The perirhinal cortex (PER) supports multimodal object recognition, but how multimodal information of objects is integrated within the PER remains unknown. Here, we recorded single units within the PER while rats performed a PER-dependent multimodal object-recognition task. In this task, audiovisual cues were presented simultaneously (multimodally) or separately (unimodally). We identified 2 types of object-selective neurons in the PER: crossmodal cells, showing constant firing patterns for an object irrespective of its modality, and unimodal cells, showing a preference for a specific modality. Unimodal cells further dissociated unimodal and multimodal versions of the object by modulating their firing rates according to the modality condition. A population-decoding analysis confirmed that the PER could perform both modality-invariant and modality-specific object decoding-the former for recognizing an object as the same in various conditions and the latter for remembering modality-specific experiences of the same object.
Collapse
Affiliation(s)
- Heung-Yeol Lim
- Department of Brain and Cognitive Sciences, Seoul National University, Seoul, Korea
| | - Inah Lee
- Department of Brain and Cognitive Sciences, Seoul National University, Seoul, Korea
| |
Collapse
|
25
|
Raja V. The motifs of radical embodied neuroscience. Eur J Neurosci 2024. [PMID: 38816952 DOI: 10.1111/ejn.16434] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/26/2024] [Revised: 04/05/2024] [Accepted: 05/20/2024] [Indexed: 06/01/2024]
Abstract
In this paper, I analyse how the emerging scientific framework of radical embodied neuroscience is different from contemporary mainstream cognitive neuroscience. To do so, I propose the notion of motif to enrich the philosophical toolkit of cognitive neuroscience. This notion can be used to characterize the guiding ideas of any given scientific framework in psychology and neuroscience. Motifs are highly unconstrained, open-ended concepts that support equally open-ended families of explanations. Different scientific frameworks-e.g., psychophysics or cognitive neuroscience-provide these motifs to answer the overarching themes of these disciplines, such as the relationship between stimuli and sensations or the proper methods of the sciences of the mind. Some motifs of mainstream cognitive neuroscience are the motif of encoding, the motif of input-output systems, and the motif of algorithms. The two first ones answer the question about the relationship between stimuli, sensations and experience (e.g., stimuli are input and are encoded by brain structures). The latter one answers the question regarding the mechanism of cognition and experience. The three of them are equally unconstrained and open-ended, and they serve as an umbrella for different kinds of explanation-i.e., different positions regarding what counts as a code or as an input. Along with the articulation of the notion of motif, the main aim of this article is to present three motifs for radical embodied neuroscience: the motif of complex stimulation, the motif of organic behaviour and the motif of resonance.
Collapse
Affiliation(s)
- Vicente Raja
- Department of Philosophy, Universidad de Murcia, Murcia, Spain
- Rotman Institute of Philosophy, Western University, London, Canada
| |
Collapse
|
26
|
Sharma H, Azouz R. Reliability and stability of tactile perception in the whisker somatosensory system. Front Neurosci 2024; 18:1344758. [PMID: 38872944 PMCID: PMC11169650 DOI: 10.3389/fnins.2024.1344758] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/26/2023] [Accepted: 05/14/2024] [Indexed: 06/15/2024] Open
Abstract
Rodents rely on their whiskers as vital sensory tools for tactile perception, enabling them to distinguish textures and shapes. Ensuring the reliability and constancy of tactile perception under varying stimulus conditions remains a fascinating and fundamental inquiry. This study explores the impact of stimulus configurations, including whisker movement velocity and object spatial proximity, on texture discrimination and stability in rats. To address this issue, we employed three distinct approaches for our investigation. Stimulus configurations notably affected tactile inputs, altering whisker vibration's kinetic and kinematic aspects with consistent effects across various textures. Through a texture discrimination task, rats exhibited consistent discrimination performance irrespective of changes in stimulus configuration. However, alterations in stimulus configuration significantly affected the rats' ability to maintain stability in texture perception. Additionally, we investigated the influence of stimulus configurations on cortical neuronal responses by manipulating them experimentally. Notably, cortical neurons demonstrated substantial and intricate changes in firing rates without compromising the ability to discriminate between textures. Nevertheless, these changes resulted in a reduction in texture neuronal response stability. Stimulating multiple whiskers led to improved neuronal texture discrimination and maintained coding stability. These findings emphasize the importance of considering numerous factors and their interactions when studying the impact of stimulus configuration on neuronal responses and behavior.
Collapse
Affiliation(s)
| | - Rony Azouz
- Department of Physiology and Cell Biology, Zlotowski Center for Neuroscience, Ben-Gurion University of the Negev, Be’er Sheva, Israel
| |
Collapse
|
27
|
Hassanpour MS, Merlin S, Federer F, Zaidi Q, Angelucci A. Primate V2 Receptive Fields Derived from Anatomically Identified Large-Scale V1 Inputs. RESEARCH SQUARE 2024:rs.3.rs-4139501. [PMID: 38798339 PMCID: PMC11118708 DOI: 10.21203/rs.3.rs-4139501/v1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/29/2024]
Abstract
In the primate visual system, visual object recognition involves a series of cortical areas arranged hierarchically along the ventral visual pathway. As information flows through this hierarchy, neurons become progressively tuned to more complex image features. The circuit mechanisms and computations underlying the increasing complexity of these receptive fields (RFs) remain unidentified. To understand how this complexity emerges in the secondary visual area (V2), we investigated the functional organization of inputs from the primary visual cortex (V1) to V2 by combining retrograde anatomical tracing of these inputs with functional imaging of feature maps in macaque monkey V1 and V2. We found that V1 neurons sending inputs to single V2 orientation columns have a broad range of preferred orientations, but are strongly biased towards the orientation represented at the injected V2 site. For each V2 site, we then constructed a feedforward model based on the linear combination of its anatomically-identified large-scale V1 inputs, and studied the response proprieties of the generated V2 RFs. We found that V2 RFs derived from the linear feedforward model were either elongated versions of V1 filters or had spatially complex structures. These modeled RFs predicted V2 neuron responses to oriented grating stimuli with high accuracy. Remarkably, this simple model also explained the greater selectivity to naturalistic textures of V2 cells compared to their V1 input cells. Our results demonstrate that simple linear combinations of feedforward inputs can account for the orientation selectivity and texture sensitivity of V2 RFs.
Collapse
Affiliation(s)
- Mahlega S Hassanpour
- Dept. of Ophthalmology and Visual Science, Moran Eye Institute, University of Utah
| | - Sam Merlin
- Dept. of Ophthalmology and Visual Science, Moran Eye Institute, University of Utah
- Present address: Dept of Medical Science, School of Science, Western Sydney University
| | - Frederick Federer
- Dept. of Ophthalmology and Visual Science, Moran Eye Institute, University of Utah
| | - Qasim Zaidi
- Graduate Center for Vision Research, State University of New York, College of Optometry
| | - Alessandra Angelucci
- Dept. of Ophthalmology and Visual Science, Moran Eye Institute, University of Utah
| |
Collapse
|
28
|
Chen PB, Chen R, LaPierre N, Chen Z, Mefford J, Marcus E, Heffel MG, Soto DC, Ernst J, Luo C, Flint J. Complementation testing identifies genes mediating effects at quantitative trait loci underlying fear-related behavior. CELL GENOMICS 2024; 4:100545. [PMID: 38697120 PMCID: PMC11099346 DOI: 10.1016/j.xgen.2024.100545] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/03/2024] [Revised: 02/23/2024] [Accepted: 04/04/2024] [Indexed: 05/04/2024]
Abstract
Knowing the genes involved in quantitative traits provides an entry point to understanding the biological bases of behavior, but there are very few examples where the pathway from genetic locus to behavioral change is known. To explore the role of specific genes in fear behavior, we mapped three fear-related traits, tested fourteen genes at six quantitative trait loci (QTLs) by quantitative complementation, and identified six genes. Four genes, Lamp, Ptprd, Nptx2, and Sh3gl, have known roles in synapse function; the fifth, Psip1, was not previously implicated in behavior; and the sixth is a long non-coding RNA, 4933413L06Rik, of unknown function. Variation in transcriptome and epigenetic modalities occurred preferentially in excitatory neurons, suggesting that genetic variation is more permissible in excitatory than inhibitory neuronal circuits. Our results relieve a bottleneck in using genetic mapping of QTLs to uncover biology underlying behavior and prompt a reconsideration of expected relationships between genetic and functional variation.
Collapse
Affiliation(s)
- Patrick B Chen
- Department of Psychiatry and Biobehavioral Sciences, David Geffen School of Medicine, University of California, Los Angeles, Los Angeles, CA, USA
| | - Rachel Chen
- Department of Psychiatry and Biobehavioral Sciences, David Geffen School of Medicine, University of California, Los Angeles, Los Angeles, CA, USA
| | - Nathan LaPierre
- Department of Computer Science, Samueli School of Engineering, University of California, Los Angeles, Los Angeles, CA, USA; Department of Human Genetics, David Geffen School of Medicine, University of California, Los Angeles, Los Angeles, CA, USA
| | - Zeyuan Chen
- Department of Computer Science, Samueli School of Engineering, University of California, Los Angeles, Los Angeles, CA, USA
| | - Joel Mefford
- Department of Human Genetics, David Geffen School of Medicine, University of California, Los Angeles, Los Angeles, CA, USA
| | - Emilie Marcus
- Department of Biological Chemistry, University of California, Los Angeles, Los Angeles, CA, USA
| | - Matthew G Heffel
- Department of Human Genetics, David Geffen School of Medicine, University of California, Los Angeles, Los Angeles, CA, USA
| | - Daniela C Soto
- Department of Psychiatry and Biobehavioral Sciences, David Geffen School of Medicine, University of California, Los Angeles, Los Angeles, CA, USA
| | - Jason Ernst
- Department of Computer Science, Samueli School of Engineering, University of California, Los Angeles, Los Angeles, CA, USA; Department of Biological Chemistry, University of California, Los Angeles, Los Angeles, CA, USA
| | - Chongyuan Luo
- Department of Human Genetics, David Geffen School of Medicine, University of California, Los Angeles, Los Angeles, CA, USA
| | - Jonathan Flint
- Department of Psychiatry and Biobehavioral Sciences, David Geffen School of Medicine, University of California, Los Angeles, Los Angeles, CA, USA.
| |
Collapse
|
29
|
Faghel-Soubeyrand S, Richoz AR, Waeber D, Woodhams J, Caldara R, Gosselin F, Charest I. Neural computations in prosopagnosia. Cereb Cortex 2024; 34:bhae211. [PMID: 38795358 PMCID: PMC11127037 DOI: 10.1093/cercor/bhae211] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/27/2022] [Revised: 04/30/2024] [Accepted: 05/03/2024] [Indexed: 05/27/2024] Open
Abstract
We report an investigation of the neural processes involved in the processing of faces and objects of brain-lesioned patient PS, a well-documented case of pure acquired prosopagnosia. We gathered a substantial dataset of high-density electrophysiological recordings from both PS and neurotypicals. Using representational similarity analysis, we produced time-resolved brain representations in a format that facilitates direct comparisons across time points, different individuals, and computational models. To understand how the lesions in PS's ventral stream affect the temporal evolution of her brain representations, we computed the temporal generalization of her brain representations. We uncovered that PS's early brain representations exhibit an unusual similarity to later representations, implying an excessive generalization of early visual patterns. To reveal the underlying computational deficits, we correlated PS' brain representations with those of deep neural networks (DNN). We found that the computations underlying PS' brain activity bore a closer resemblance to early layers of a visual DNN than those of controls. However, the brain representations in neurotypicals became more akin to those of the later layers of the model compared to PS. We confirmed PS's deficits in high-level brain representations by demonstrating that her brain representations exhibited less similarity with those of a DNN of semantics.
Collapse
Affiliation(s)
- Simon Faghel-Soubeyrand
- Département de psychologie, Université de Montréal, 90 av. Vincent D’indy, Montreal, H2V 2S9, Canada
- Department of Experimental Psychology, University of Oxford, Anna Watts Building, Woodstock Rd, Oxford OX2 6GG
| | - Anne-Raphaelle Richoz
- Département de psychologie, Université de Fribourg, RM 01 bu. C-3.117Rue P.A. de Faucigny 21700 Fribourg, Switzerland
| | - Delphine Waeber
- Département de psychologie, Université de Fribourg, RM 01 bu. C-3.117Rue P.A. de Faucigny 21700 Fribourg, Switzerland
| | - Jessica Woodhams
- School of Psychology, University of Birmingham, Hills Building, Edgbaston Park Rd, Birmingham B15 2TT, UK
| | - Roberto Caldara
- Département de psychologie, Université de Fribourg, RM 01 bu. C-3.117Rue P.A. de Faucigny 21700 Fribourg, Switzerland
| | - Frédéric Gosselin
- Département de psychologie, Université de Montréal, 90 av. Vincent D’indy, Montreal, H2V 2S9, Canada
| | - Ian Charest
- Département de psychologie, Université de Montréal, 90 av. Vincent D’indy, Montreal, H2V 2S9, Canada
| |
Collapse
|
30
|
Ren Y, Bashivan P. How well do models of visual cortex generalize to out of distribution samples? PLoS Comput Biol 2024; 20:e1011145. [PMID: 38820563 PMCID: PMC11216589 DOI: 10.1371/journal.pcbi.1011145] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/02/2023] [Revised: 07/01/2024] [Accepted: 04/29/2024] [Indexed: 06/02/2024] Open
Abstract
Unit activity in particular deep neural networks (DNNs) are remarkably similar to the neuronal population responses to static images along the primate ventral visual cortex. Linear combinations of DNN unit activities are widely used to build predictive models of neuronal activity in the visual cortex. Nevertheless, prediction performance in these models is often investigated on stimulus sets consisting of everyday objects under naturalistic settings. Recent work has revealed a generalization gap in how predicting neuronal responses to synthetically generated out-of-distribution (OOD) stimuli. Here, we investigated how the recent progress in improving DNNs' object recognition generalization, as well as various DNN design choices such as architecture, learning algorithm, and datasets have impacted the generalization gap in neural predictivity. We came to a surprising conclusion that the performance on none of the common computer vision OOD object recognition benchmarks is predictive of OOD neural predictivity performance. Furthermore, we found that adversarially robust models often yield substantially higher generalization in neural predictivity, although the degree of robustness itself was not predictive of neural predictivity score. These results suggest that improving object recognition behavior on current benchmarks alone may not lead to more general models of neurons in the primate ventral visual cortex.
Collapse
Affiliation(s)
- Yifei Ren
- Department of Computer Science, McGill University, Montreal, Canada
| | - Pouya Bashivan
- Department of Computer Science, McGill University, Montreal, Canada
- Department of Computer Physiology, McGill University, Montreal, Canada
- Mila, Université de Montréal, Montreal, Canada
| |
Collapse
|
31
|
Revsine C, Gonzalez-Castillo J, Merriam EP, Bandettini PA, Ramírez FM. A Unifying Model for Discordant and Concordant Results in Human Neuroimaging Studies of Facial Viewpoint Selectivity. J Neurosci 2024; 44:e0296232024. [PMID: 38438256 PMCID: PMC11044116 DOI: 10.1523/jneurosci.0296-23.2024] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/14/2023] [Revised: 02/06/2024] [Accepted: 02/15/2024] [Indexed: 03/06/2024] Open
Abstract
Recognizing faces regardless of their viewpoint is critical for social interactions. Traditional theories hold that view-selective early visual representations gradually become tolerant to viewpoint changes along the ventral visual hierarchy. Newer theories, based on single-neuron monkey electrophysiological recordings, suggest a three-stage architecture including an intermediate face-selective patch abruptly achieving invariance to mirror-symmetric face views. Human studies combining neuroimaging and multivariate pattern analysis (MVPA) have provided convergent evidence of view selectivity in early visual areas. However, contradictory conclusions have been reached concerning the existence in humans of a mirror-symmetric representation like that observed in macaques. We believe these contradictions arise from low-level stimulus confounds and data analysis choices. To probe for low-level confounds, we analyzed images from two face databases. Analyses of image luminance and contrast revealed biases across face views described by even polynomials-i.e., mirror-symmetric. To explain major trends across neuroimaging studies, we constructed a network model incorporating three constraints: cortical magnification, convergent feedforward projections, and interhemispheric connections. Given the identified low-level biases, we show that a gradual increase of interhemispheric connections across network-layers is sufficient to replicate view-tuning in early processing stages and mirror-symmetry in later stages. Data analysis decisions-pattern dissimilarity measure and data recentering-accounted for the inconsistent observation of mirror-symmetry across prior studies. Pattern analyses of human fMRI data (of either sex) revealed biases compatible with our model. The model provides a unifying explanation of MVPA studies of viewpoint selectivity and suggests observations of mirror-symmetry originate from ineffectively normalized signal imbalances across different face views.
Collapse
Affiliation(s)
- Cambria Revsine
- Laboratory of Brain and Cognition, National Institute of Mental Health, National Institutes of Health, Bethesda, Maryland 20892
- Department of Psychology, University of Chicago, Chicago, Illinois 60637
| | - Javier Gonzalez-Castillo
- Section on Functional Imaging Methods, Laboratory of Brain and Cognition, National Institute of Mental Health, National Institutes of Health, Bethesda, Maryland 20892
| | - Elisha P Merriam
- Laboratory of Brain and Cognition, National Institute of Mental Health, National Institutes of Health, Bethesda, Maryland 20892
| | - Peter A Bandettini
- Section on Functional Imaging Methods, Laboratory of Brain and Cognition, National Institute of Mental Health, National Institutes of Health, Bethesda, Maryland 20892
- Functional MRI Core Facility, National Institute of Mental Health, National Institutes of Health, Bethesda, Maryland 20892
| | - Fernando M Ramírez
- Laboratory of Brain and Cognition, National Institute of Mental Health, National Institutes of Health, Bethesda, Maryland 20892
- Section on Functional Imaging Methods, Laboratory of Brain and Cognition, National Institute of Mental Health, National Institutes of Health, Bethesda, Maryland 20892
| |
Collapse
|
32
|
Wang Y, Cao R, Wang S. Encoding of Visual Objects in the Human Medial Temporal Lobe. J Neurosci 2024; 44:e2135232024. [PMID: 38429107 PMCID: PMC11026346 DOI: 10.1523/jneurosci.2135-23.2024] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/14/2023] [Revised: 02/10/2024] [Accepted: 02/25/2024] [Indexed: 03/03/2024] Open
Abstract
The human medial temporal lobe (MTL) plays a crucial role in recognizing visual objects, a key cognitive function that relies on the formation of semantic representations. Nonetheless, it remains unknown how visual information of general objects is translated into semantic representations in the MTL. Furthermore, the debate about whether the human MTL is involved in perception has endured for a long time. To address these questions, we investigated three distinct models of neural object coding-semantic coding, axis-based feature coding, and region-based feature coding-in each subregion of the human MTL, using high-resolution fMRI in two male and six female participants. Our findings revealed the presence of semantic coding throughout the MTL, with a higher prevalence observed in the parahippocampal cortex (PHC) and perirhinal cortex (PRC), while axis coding and region coding were primarily observed in the earlier regions of the MTL. Moreover, we demonstrated that voxels exhibiting axis coding supported the transition to region coding and contained information relevant to semantic coding. Together, by providing a detailed characterization of neural object coding schemes and offering a comprehensive summary of visual coding information for each MTL subregion, our results not only emphasize a clear role of the MTL in perceptual processing but also shed light on the translation of perception-driven representations of visual features into memory-driven representations of semantics along the MTL processing pathway.
Collapse
Affiliation(s)
- Yue Wang
- Department of Radiology, Washington University in St. Louis, St. Louis, Missouri 63110
| | - Runnan Cao
- Department of Radiology, Washington University in St. Louis, St. Louis, Missouri 63110
| | - Shuo Wang
- Department of Radiology, Washington University in St. Louis, St. Louis, Missouri 63110
| |
Collapse
|
33
|
Heinen R, Bierbrauer A, Wolf OT, Axmacher N. Representational formats of human memory traces. Brain Struct Funct 2024; 229:513-529. [PMID: 37022435 PMCID: PMC10978732 DOI: 10.1007/s00429-023-02636-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/06/2022] [Accepted: 03/28/2023] [Indexed: 04/07/2023]
Abstract
Neural representations are internal brain states that constitute the brain's model of the external world or some of its features. In the presence of sensory input, a representation may reflect various properties of this input. When perceptual information is no longer available, the brain can still activate representations of previously experienced episodes due to the formation of memory traces. In this review, we aim at characterizing the nature of neural memory representations and how they can be assessed with cognitive neuroscience methods, mainly focusing on neuroimaging. We discuss how multivariate analysis techniques such as representational similarity analysis (RSA) and deep neural networks (DNNs) can be leveraged to gain insights into the structure of neural representations and their different representational formats. We provide several examples of recent studies which demonstrate that we are able to not only measure memory representations using RSA but are also able to investigate their multiple formats using DNNs. We demonstrate that in addition to slow generalization during consolidation, memory representations are subject to semantization already during short-term memory, by revealing a shift from visual to semantic format. In addition to perceptual and conceptual formats, we describe the impact of affective evaluations as an additional dimension of episodic memories. Overall, these studies illustrate how the analysis of neural representations may help us gain a deeper understanding of the nature of human memory.
Collapse
Affiliation(s)
- Rebekka Heinen
- Department of Neuropsychology, Institute of Cognitive Neuroscience, Faculty of Psychology, Ruhr University Bochum, Universitätsstraße 150, 44801, Bochum, Germany.
| | - Anne Bierbrauer
- Department of Neuropsychology, Institute of Cognitive Neuroscience, Faculty of Psychology, Ruhr University Bochum, Universitätsstraße 150, 44801, Bochum, Germany
- Institute for Systems Neuroscience, Medical Center Hamburg-Eppendorf, Martinistraße 52, 20251, Hamburg, Germany
| | - Oliver T Wolf
- Department of Cognitive Psychology, Institute of Cognitive Neuroscience, Faculty of Psychology, Ruhr University Bochum, Universitätsstraße 150, 44801, Bochum, Germany
| | - Nikolai Axmacher
- Department of Neuropsychology, Institute of Cognitive Neuroscience, Faculty of Psychology, Ruhr University Bochum, Universitätsstraße 150, 44801, Bochum, Germany
| |
Collapse
|
34
|
Campbell A, Tanaka JW. Fast saccades to faces during the feedforward sweep. J Vis 2024; 24:16. [PMID: 38630459 PMCID: PMC11037494 DOI: 10.1167/jov.24.4.16] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/10/2022] [Accepted: 09/19/2023] [Indexed: 04/19/2024] Open
Abstract
Saccadic choice tasks use eye movements as a response method, typically in a task where observers are asked to saccade as quickly as possible to an image of a prespecified target category. Using this approach, face-selective saccades have been observed within 100 ms poststimulus. When taking into account oculomotor processing, this suggests that faces can be detected in as little as 70 to 80 ms. It has therefore been suggested that face detection must occur during the initial feedforward sweep, since this latency leaves little time for feedback processing. In the current experiment, we tested this hypothesis using backward masking-a technique shown to primarily disrupt feedback processing while leaving feedforward activation relatively intact. Based on minimum saccadic reaction time, we found that face detection benefited from ultra-fast, accurate saccades within 110 to 160 ms and that these eye movements are obtainable even under extreme masking conditions that limit perceptual awareness. However, masking did significantly increase the median SRT for faces. In the manual responses, we found remarkable detection accuracy for faces and houses, even when participants indicated having no visual experience of the test images. These results provide evidence for the view that the saccadic bias to faces is initiated by coarse information used to categorize faces in the feedforward sweep but that, in most cases, additional processing is required to quickly reach the threshold for saccade initiation.
Collapse
Affiliation(s)
- Alison Campbell
- Department of Psychology, University of Victoria, Victoria, BC, Canada
- https://orcid.org/0000-0001-6891-8609
| | - James W Tanaka
- Department of Psychology, University of Victoria, Victoria, BC, Canada
- https://orcid.org/0000-0001-6559-0388
| |
Collapse
|
35
|
Goris RLT, Coen-Cagli R, Miller KD, Priebe NJ, Lengyel M. Response sub-additivity and variability quenching in visual cortex. Nat Rev Neurosci 2024; 25:237-252. [PMID: 38374462 DOI: 10.1038/s41583-024-00795-0] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 01/24/2024] [Indexed: 02/21/2024]
Abstract
Sub-additivity and variability are ubiquitous response motifs in the primary visual cortex (V1). Response sub-additivity enables the construction of useful interpretations of the visual environment, whereas response variability indicates the factors that limit the precision with which the brain can do this. There is increasing evidence that experimental manipulations that elicit response sub-additivity often also quench response variability. Here, we provide an overview of these phenomena and suggest that they may have common origins. We discuss empirical findings and recent model-based insights into the functional operations, computational objectives and circuit mechanisms underlying V1 activity. These different modelling approaches all predict that response sub-additivity and variability quenching often co-occur. The phenomenology of these two response motifs, as well as many of the insights obtained about them in V1, generalize to other cortical areas. Thus, the connection between response sub-additivity and variability quenching may be a canonical motif across the cortex.
Collapse
Affiliation(s)
- Robbe L T Goris
- Center for Perceptual Systems, University of Texas at Austin, Austin, TX, USA.
| | - Ruben Coen-Cagli
- Department of Systems and Computational Biology, Albert Einstein College of Medicine, Bronx, NY, USA
- Dominick P. Purpura Department of Neuroscience, Albert Einstein College of Medicine, Bronx, NY, USA
- Department of Ophthalmology and Visual Sciences, Albert Einstein College of Medicine, Bronx, NY, USA
| | - Kenneth D Miller
- Center for Theoretical Neuroscience, Columbia University, New York, NY, USA
- Kavli Institute for Brain Science, Columbia University, New York, NY, USA
- Dept. of Neuroscience, College of Physicians and Surgeons, Columbia University, New York, NY, USA
- Morton B. Zuckerman Mind Brain Behavior Institute, Columbia University, New York, NY, USA
- Swartz Program in Theoretical Neuroscience, Columbia University, New York, NY, USA
| | - Nicholas J Priebe
- Center for Learning and Memory, University of Texas at Austin, Austin, TX, USA
| | - Máté Lengyel
- Computational and Biological Learning Lab, Department of Engineering, University of Cambridge, Cambridge, UK
- Center for Cognitive Computation, Department of Cognitive Science, Central European University, Budapest, Hungary
| |
Collapse
|
36
|
Lee GM, Rodríguez-Deliz CL, Bushnell BN, Majaj NJ, Movshon JA, Kiorpes L. Developmentally stable representations of naturalistic image structure in macaque visual cortex. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2024:2024.02.24.581889. [PMID: 38463955 PMCID: PMC10925106 DOI: 10.1101/2024.02.24.581889] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 03/12/2024]
Abstract
We studied visual development in macaque monkeys using texture stimuli, matched in local spectral content but varying in "naturalistic" structure. In adult monkeys, naturalistic textures preferentially drive neurons in areas V2 and V4, but not V1. We paired behavioral measurements of naturalness sensitivity with separately-obtained neuronal population recordings from neurons in areas V1, V2, V4, and inferotemporal cortex (IT). We made behavioral measurements from 16 weeks of age and physiological measurements as early as 20 weeks, and continued through 56 weeks. Behavioral sensitivity reached half of maximum at roughly 25 weeks of age. Neural sensitivities remained stable from the earliest ages tested. As in adults, neural sensitivity to naturalistic structure increased from V1 to V2 to V4. While sensitivities in V2 and IT were similar, the dimensionality of the IT representation was more similar to V4's than to V2's.
Collapse
Affiliation(s)
- Gerick M. Lee
- Center for Neural Science New York University New York, NY, USA 10003
| | | | | | - Najib J. Majaj
- Center for Neural Science New York University New York, NY, USA 10003
| | | | - Lynne Kiorpes
- Center for Neural Science New York University New York, NY, USA 10003
| |
Collapse
|
37
|
Hassanpour MS, Merlin S, Federer F, Zaidi Q, Angelucci A. Primate V2 Receptive Fields Derived from Anatomically Identified Large-Scale V1 Inputs. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2024:2024.03.22.586002. [PMID: 38585792 PMCID: PMC10996519 DOI: 10.1101/2024.03.22.586002] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 04/09/2024]
Abstract
In the primate visual system, visual object recognition involves a series of cortical areas arranged hierarchically along the ventral visual pathway. As information flows through this hierarchy, neurons become progressively tuned to more complex image features. The circuit mechanisms and computations underlying the increasing complexity of these receptive fields (RFs) remain unidentified. To understand how this complexity emerges in the secondary visual area (V2), we investigated the functional organization of inputs from the primary visual cortex (V1) to V2 by combining retrograde anatomical tracing of these inputs with functional imaging of feature maps in macaque monkey V1 and V2. We found that V1 neurons sending inputs to single V2 orientation columns have a broad range of preferred orientations, but are strongly biased towards the orientation represented at the injected V2 site. For each V2 site, we then constructed a feedforward model based on the linear combination of its anatomically-identified large-scale V1 inputs, and studied the response proprieties of the generated V2 RFs. We found that V2 RFs derived from the linear feedforward model were either elongated versions of V1 filters or had spatially complex structures. These modeled RFs predicted V2 neuron responses to oriented grating stimuli with high accuracy. Remarkably, this simple model also explained the greater selectivity to naturalistic textures of V2 cells compared to their V1 input cells. Our results demonstrate that simple linear combinations of feedforward inputs can account for the orientation selectivity and texture sensitivity of V2 RFs.
Collapse
|
38
|
Kumar N, Srivastava R. Deep learning in structural bioinformatics: current applications and future perspectives. Brief Bioinform 2024; 25:bbae042. [PMID: 38701422 PMCID: PMC11066934 DOI: 10.1093/bib/bbae042] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/17/2023] [Revised: 01/05/2024] [Accepted: 01/18/2024] [Indexed: 05/05/2024] Open
Abstract
In this review article, we explore the transformative impact of deep learning (DL) on structural bioinformatics, emphasizing its pivotal role in a scientific revolution driven by extensive data, accessible toolkits and robust computing resources. As big data continue to advance, DL is poised to become an integral component in healthcare and biology, revolutionizing analytical processes. Our comprehensive review provides detailed insights into DL, featuring specific demonstrations of its notable applications in bioinformatics. We address challenges tailored for DL, spotlight recent successes in structural bioinformatics and present a clear exposition of DL-from basic shallow neural networks to advanced models such as convolution, recurrent, artificial and transformer neural networks. This paper discusses the emerging use of DL for understanding biomolecular structures, anticipating ongoing developments and applications in the realm of structural bioinformatics.
Collapse
Affiliation(s)
- Niranjan Kumar
- School of Computational and Integrative Sciences, Jawaharlal Nehru University, New Delhi, India
| | - Rakesh Srivastava
- Center for Computational Natural Sciences and Bioinformatics, International Institute of Information Technology, Hyderabad, India
| |
Collapse
|
39
|
McMullin MA, Kumar R, Higgins NC, Gygi B, Elhilali M, Snyder JS. Preliminary Evidence for Global Properties in Human Listeners During Natural Auditory Scene Perception. Open Mind (Camb) 2024; 8:333-365. [PMID: 38571530 PMCID: PMC10990578 DOI: 10.1162/opmi_a_00131] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/10/2023] [Accepted: 02/10/2024] [Indexed: 04/05/2024] Open
Abstract
Theories of auditory and visual scene analysis suggest the perception of scenes relies on the identification and segregation of objects within it, resembling a detail-oriented processing style. However, a more global process may occur while analyzing scenes, which has been evidenced in the visual domain. It is our understanding that a similar line of research has not been explored in the auditory domain; therefore, we evaluated the contributions of high-level global and low-level acoustic information to auditory scene perception. An additional aim was to increase the field's ecological validity by using and making available a new collection of high-quality auditory scenes. Participants rated scenes on 8 global properties (e.g., open vs. enclosed) and an acoustic analysis evaluated which low-level features predicted the ratings. We submitted the acoustic measures and average ratings of the global properties to separate exploratory factor analyses (EFAs). The EFA of the acoustic measures revealed a seven-factor structure explaining 57% of the variance in the data, while the EFA of the global property measures revealed a two-factor structure explaining 64% of the variance in the data. Regression analyses revealed each global property was predicted by at least one acoustic variable (R2 = 0.33-0.87). These findings were extended using deep neural network models where we examined correlations between human ratings of global properties and deep embeddings of two computational models: an object-based model and a scene-based model. The results support that participants' ratings are more strongly explained by a global analysis of the scene setting, though the relationship between scene perception and auditory perception is multifaceted, with differing correlation patterns evident between the two models. Taken together, our results provide evidence for the ability to perceive auditory scenes from a global perspective. Some of the acoustic measures predicted ratings of global scene perception, suggesting representations of auditory objects may be transformed through many stages of processing in the ventral auditory stream, similar to what has been proposed in the ventral visual stream. These findings and the open availability of our scene collection will make future studies on perception, attention, and memory for natural auditory scenes possible.
Collapse
Affiliation(s)
| | - Rohit Kumar
- Department of Electrical and Computer Engineering, Johns Hopkins University, Baltimore, MD, USA
| | - Nathan C. Higgins
- Department of Communication Sciences & Disorders, University of South Florida, Tampa, FL, USA
| | - Brian Gygi
- East Bay Institute for Research and Education, Martinez, CA, USA
| | - Mounya Elhilali
- Department of Electrical and Computer Engineering, Johns Hopkins University, Baltimore, MD, USA
| | - Joel S. Snyder
- Department of Psychology, University of Nevada, Las Vegas, Las Vegas, NV, USA
| |
Collapse
|
40
|
Grisoni L, Boux IP, Pulvermüller F. Predictive Brain Activity Shows Congruent Semantic Specificity in Language Comprehension and Production. J Neurosci 2024; 44:e1723232023. [PMID: 38267261 PMCID: PMC10957213 DOI: 10.1523/jneurosci.1723-23.2023] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/13/2023] [Revised: 12/21/2023] [Accepted: 12/25/2023] [Indexed: 01/26/2024] Open
Abstract
Sentence fragments strongly predicting a specific subsequent meaningful word elicit larger preword slow waves, prediction potentials (PPs), than unpredictive contexts. To test the current predictive processing models, 128-channel EEG data were collected from both sexes to examine whether (1) different semantic PPs are elicited in language comprehension and production and (2) whether these PPs originate from the same specific "prediction area(s)" or rather from widely distributed category-specific neuronal circuits reflecting the meaning of the predicted item. Slow waves larger after predictable than unpredictable contexts were present both before subjects heard the sentence-final word in the comprehension experiment and before they pronounced the sentence-final word in the production experiment. Crucially, cortical sources underlying the semantic PP were distributed across several cortical areas and differed between the semantic categories of the expected words. In both production and comprehension, the anticipation of animal words was reflected by sources in posterior visual areas, whereas predictable tool words were preceded by sources in the frontocentral sensorimotor cortex. For both modalities, PP size increased with higher cloze probability, thus further confirming that it reflects semantic prediction, and with shorter latencies with which participants completed sentence fragments. These results sit well with theories viewing distributed semantic category-specific circuits as the mechanistic basis of semantic prediction in the two modalities.
Collapse
Affiliation(s)
- Luigi Grisoni
- Brain Language Laboratory, Department of Philosophy and Humanities, Freie Universität Berlin, Berlin 14195, Germany
- Cluster of Excellence 'Matters of Activity, Image Space Material', Humboldt Universität zu Berlin, Berlin 10099, Germany
| | - Isabella P Boux
- Brain Language Laboratory, Department of Philosophy and Humanities, Freie Universität Berlin, Berlin 14195, Germany
- Berlin School of Mind and Brain, Humboldt Universität zu Berlin, Berlin 10099, Germany
- Einstein Center for Neurosciences, Berlin 10117, Germany
- Biological and Social Psychology, Institute of Psychology, RWTH Aachen University, Aachen 52056, Germany
| | - Friedemann Pulvermüller
- Brain Language Laboratory, Department of Philosophy and Humanities, Freie Universität Berlin, Berlin 14195, Germany
- Cluster of Excellence 'Matters of Activity, Image Space Material', Humboldt Universität zu Berlin, Berlin 10099, Germany
- Berlin School of Mind and Brain, Humboldt Universität zu Berlin, Berlin 10099, Germany
- Einstein Center for Neurosciences, Berlin 10117, Germany
| |
Collapse
|
41
|
Forti B. Approaching the nature of consciousness through a phenomenal analysis of early vision. What is the explanandum? Front Psychol 2024; 15:1329259. [PMID: 38562232 PMCID: PMC10982490 DOI: 10.3389/fpsyg.2024.1329259] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/28/2023] [Accepted: 03/07/2024] [Indexed: 04/04/2024] Open
Abstract
Loorits (2014) identifies the solution to the hard problem of consciousness in the possibility of fully analyzing seemingly non-structural aspects of consciousness in structural terms. However, research on consciousness conducted in recent decades has failed to bridge the explanatory gap between the brain and conscious mind. One reason why the explanatory gap cannot be filled, and consequently the problem remains hard, is that experience and neural structure are too different or "distant" to be directly compatible. Conversely, structural aspects of consciousness can be found in phenomenal experience. One possible alternative, therefore, is to seek the structure of seemingly non-structural aspects of consciousness not in the neural substrate, but within consciousness itself, through a phenomenal analysis of the qualitative aspects of experience, starting from its simplest forms. An essential premise is to reformulate the explanandum of consciousness, which is usually attributed to qualia and what it is like to be in a certain state. However, these properties do not allow us to identify the fundamental aspects of phenomenal experience. Sensations such as the redness of red or the painfulness of pain are inseparable from the context of the experience to which they belong, making qualia appear as phenomenal artifacts. Furthermore, the simplest qualitative aspects can be found in early vision. They are involved in perceptual organization and necessarily have relational significance. The unitary set of qualities found in early vision-such as those related to being an object, background or detail-constitutes the explanandum of the simplest forms of consciousness and seems to imply a justifying structure. Although early vision is characterized by interdependent qualitative components that form a unitary whole, we cannot find in it the structure of seemingly non-structural aspects of consciousness. Phenomenal appearance alone does not seem sufficient to identify a unitary structure of consciousness. However, the closeness of these characteristics to a unitary structure prompts us to delve into less explored territory, using the components of experience also as possible explanans.
Collapse
Affiliation(s)
- Bruno Forti
- Department of Mental Health, Azienda ULSS 1 Dolomiti, Belluno, Italy
| |
Collapse
|
42
|
Oleskiw TD, Lieber JD, Simoncelli EP, Movshon JA. Foundations of visual form selectivity for neurons in macaque V1 and V2. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2024:2024.03.04.583307. [PMID: 38496618 PMCID: PMC10942284 DOI: 10.1101/2024.03.04.583307] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 03/19/2024]
Abstract
We have measured the visually evoked activity of single neurons recorded in areas V1 and V2 of awake, fixating macaque monkeys, and captured their responses with a common computational model. We used a stimulus set composed of "droplets" of localized contrast, band-limited in orientation and spatial frequency; each brief stimulus contained a random superposition of droplets presented in and near the mapped receptive field. We accounted for neuronal responses with a 2-layer linear-nonlinear model, representing each receptive field by a combination of orientation- and scale-selective filters. We fit the data by jointly optimizing the model parameters to enforce sparsity and to prevent overfitting. We visualized and interpreted the fits in terms of an "afferent field" of nonlinearly combined inputs, dispersed in the 4 dimensions of space and spatial frequency. The resulting fits generally give a good account of the responses of neurons in both V1 and V2, capturing an average of 40% of the explainable variance in neuronal firing. Moreover, the resulting models predict neuronal responses to image families outside the test set, such as gratings of different orientations and spatial frequencies. Our results offer a common framework for understanding processing in the early visual cortex, and also demonstrate the ways in which the distributions of neuronal responses in V1 and V2 are similar but not identical.
Collapse
Affiliation(s)
- Timothy D Oleskiw
- Center for Neural Science, New York University; Center for Computational Neuroscience, Flatiron Institute
| | | | - Eero P Simoncelli
- Center for Computational Neuroscience, Flatiron Institute; Center for Neural Science, New York University
| | | |
Collapse
|
43
|
Jang H, Tong F. Improved modeling of human vision by incorporating robustness to blur in convolutional neural networks. Nat Commun 2024; 15:1989. [PMID: 38443349 PMCID: PMC10915141 DOI: 10.1038/s41467-024-45679-0] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/29/2023] [Accepted: 01/30/2024] [Indexed: 03/07/2024] Open
Abstract
Whenever a visual scene is cast onto the retina, much of it will appear degraded due to poor resolution in the periphery; moreover, optical defocus can cause blur in central vision. However, the pervasiveness of blurry or degraded input is typically overlooked in the training of convolutional neural networks (CNNs). We hypothesized that the absence of blurry training inputs may cause CNNs to rely excessively on high spatial frequency information for object recognition, thereby causing systematic deviations from biological vision. We evaluated this hypothesis by comparing standard CNNs with CNNs trained on a combination of clear and blurry images. We show that blur-trained CNNs outperform standard CNNs at predicting neural responses to objects across a variety of viewing conditions. Moreover, blur-trained CNNs acquire increased sensitivity to shape information and greater robustness to multiple forms of visual noise, leading to improved correspondence with human perception. Our results provide multi-faceted neurocomputational evidence that blurry visual experiences may be critical for conferring robustness to biological visual systems.
Collapse
Affiliation(s)
- Hojin Jang
- Department of Psychology, Vanderbilt Vision Research Center, Vanderbilt University, Nashville, TN, USA.
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA, USA.
- Department of Brain and Cognitive Engineering, Korea University, Seoul, South Korea.
| | - Frank Tong
- Department of Psychology, Vanderbilt Vision Research Center, Vanderbilt University, Nashville, TN, USA.
| |
Collapse
|
44
|
Tuckute G, Sathe A, Srikant S, Taliaferro M, Wang M, Schrimpf M, Kay K, Fedorenko E. Driving and suppressing the human language network using large language models. Nat Hum Behav 2024; 8:544-561. [PMID: 38172630 DOI: 10.1038/s41562-023-01783-7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/06/2023] [Accepted: 11/10/2023] [Indexed: 01/05/2024]
Abstract
Transformer models such as GPT generate human-like language and are predictive of human brain responses to language. Here, using functional-MRI-measured brain responses to 1,000 diverse sentences, we first show that a GPT-based encoding model can predict the magnitude of the brain response associated with each sentence. We then use the model to identify new sentences that are predicted to drive or suppress responses in the human language network. We show that these model-selected novel sentences indeed strongly drive and suppress the activity of human language areas in new individuals. A systematic analysis of the model-selected sentences reveals that surprisal and well-formedness of linguistic input are key determinants of response strength in the language network. These results establish the ability of neural network models to not only mimic human language but also non-invasively control neural activity in higher-level cortical areas, such as the language network.
Collapse
Affiliation(s)
- Greta Tuckute
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA, USA.
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA, USA.
| | - Aalok Sathe
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA, USA
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA, USA
| | - Shashank Srikant
- Computer Science & Artificial Intelligence Laboratory, Massachusetts Institute of Technology, Cambridge, MA, USA
- MIT-IBM Watson AI Lab, Cambridge, MA, USA
| | - Maya Taliaferro
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA, USA
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA, USA
| | - Mingye Wang
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA, USA
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA, USA
| | - Martin Schrimpf
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA, USA
- Quest for Intelligence, Massachusetts Institute of Technology, Cambridge, MA, USA
- Neuro-X Institute, École Polytechnique Fédérale de Lausanne, Lausanne, Switzerland
| | - Kendrick Kay
- Center for Magnetic Resonance Research, University of Minnesota, Minneapolis, MN, USA
| | - Evelina Fedorenko
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA, USA.
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA, USA.
- Program in Speech and Hearing Bioscience and Technology, Harvard University, Cambridge, MA, USA.
| |
Collapse
|
45
|
Matsuo Y, Kawakami A, Matsuo R. Visual afferents from an eye in the terrestrial slug Limax valentianus. J Comp Neurol 2024; 532:e25600. [PMID: 38433660 DOI: 10.1002/cne.25600] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/20/2023] [Revised: 02/10/2024] [Accepted: 02/19/2024] [Indexed: 03/05/2024]
Abstract
Terrestrial gastropods have a lens-bearing eye on the tip of their tentacles. There are two morphologically distinct photoreceptors, called Type-I and Type-II photoreceptors, in the retina. Type-I photoreceptors are equipped with highly developed photoreceptive microvilli in their outer rhabdomeric segment, whereas Type-II photoreceptors have short and fewer microvilli. Although both types of photoreceptors send afferent projections directly to the brain, their destinations in the brain, called optic neuropiles, have not been sufficiently investigated. Our recent studies revealed that there are commissural fibers in the cerebral ganglia that transmit photic information acquired by bilateral eyes. Moreover, some of the retinal photoreceptors are connected by gap junctions to the photosensitive brain neurons, suggesting the functional interaction of the photic information between the eye and brain photoreceptors, as well as between bilateral eyes. However, it has not been clarified which type of retinal photoreceptors send commissural projections to the contralateral hemiganglion nor interact with the brain photoreceptors. In the present study, we demonstrated by molecular histological analyses and tracer injections that (1) Type-I and Type-II photoreceptors send glutamatergic afferent projections to the medial and lateral lobes of the ipsilateral optic neuropile, respectively, (2) direct synaptic interaction between bilateral optic nerves occurs in the medial lobe of the optic neuropile, and (3) brain photosensory neurons form gap junctions with the medial lobe of the contralateral optic neuropile. These results reveal an ordered pattern of afferent projections from the retina and provide insight into the different functional roles of retinal photoreceptors.
Collapse
Affiliation(s)
- Yuko Matsuo
- Department of Environmental Sciences, International College of Arts and Sciences, Fukuoka Women's University, Fukuoka, Japan
| | - Airi Kawakami
- Department of Environmental Sciences, International College of Arts and Sciences, Fukuoka Women's University, Fukuoka, Japan
| | - Ryota Matsuo
- Department of Environmental Sciences, International College of Arts and Sciences, Fukuoka Women's University, Fukuoka, Japan
| |
Collapse
|
46
|
Bi Z, Li H, Tian L. Top-down generation of low-resolution representations improves visual perception and imagination. Neural Netw 2024; 171:440-456. [PMID: 38150870 DOI: 10.1016/j.neunet.2023.12.030] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/25/2023] [Revised: 11/30/2023] [Accepted: 12/18/2023] [Indexed: 12/29/2023]
Abstract
Perception or imagination requires top-down signals from high-level cortex to primary visual cortex (V1) to reconstruct or simulate the representations bottom-up stimulated by the seen images. Interestingly, top-down signals in V1 have lower spatial resolution than bottom-up representations. It is unclear why the brain uses low-resolution signals to reconstruct or simulate high-resolution representations. By modeling the top-down pathway of the visual system using the decoder of a variational auto-encoder (VAE), we reveal that low-resolution top-down signals can better reconstruct or simulate the information contained in the sparse activities of V1 simple cells, which facilitates perception and imagination. This advantage of low-resolution generation is related to facilitating high-level cortex to form geometry-respecting representations observed in experiments. Furthermore, we present two findings regarding this phenomenon in the context of AI-generated sketches, a style of drawings made of lines. First, we found that the quality of the generated sketches critically depends on the thickness of the lines in the sketches: thin-line sketches are harder to generate than thick-line sketches. Second, we propose a technique to generate high-quality thin-line sketches: instead of directly using original thin-line sketches, we use blurred sketches to train VAE or GAN (generative adversarial network), and then infer the thin-line sketches from the VAE- or GAN-generated blurred sketches. Collectively, our work suggests that low-resolution top-down generation is a strategy the brain uses to improve visual perception and imagination, which inspires new sketch-generation AI techniques.
Collapse
Affiliation(s)
- Zedong Bi
- Lingang Laboratory, Shanghai 200031, China.
| | - Haoran Li
- Department of Physics, Hong Kong Baptist University, Hong Kong, China
| | - Liang Tian
- Department of Physics, Hong Kong Baptist University, Hong Kong, China; Institute of Computational and Theoretical Studies, Hong Kong Baptist University, Hong Kong, China; Institute of Systems Medicine and Health Sciences, Hong Kong Baptist University, Hong Kong, China; State Key Laboratory of Environmental and Biological Analysis, Hong Kong Baptist University, Hong Kong, China.
| |
Collapse
|
47
|
Stoinski LM, Perkuhn J, Hebart MN. THINGSplus: New norms and metadata for the THINGS database of 1854 object concepts and 26,107 natural object images. Behav Res Methods 2024; 56:1583-1603. [PMID: 37095326 PMCID: PMC10991023 DOI: 10.3758/s13428-023-02110-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 03/13/2023] [Indexed: 04/26/2023]
Abstract
To study visual and semantic object representations, the need for well-curated object concepts and images has grown significantly over the past years. To address this, we have previously developed THINGS, a large-scale database of 1854 systematically sampled object concepts with 26,107 high-quality naturalistic images of these concepts. With THINGSplus, we significantly extend THINGS by adding concept- and image-specific norms and metadata for all 1854 concepts and one copyright-free image example per concept. Concept-specific norms were collected for the properties of real-world size, manmadeness, preciousness, liveliness, heaviness, naturalness, ability to move or be moved, graspability, holdability, pleasantness, and arousal. Further, we provide 53 superordinate categories as well as typicality ratings for all their members. Image-specific metadata includes a nameability measure, based on human-generated labels of the objects depicted in the 26,107 images. Finally, we identified one new public domain image per concept. Property (M = 0.97, SD = 0.03) and typicality ratings (M = 0.97, SD = 0.01) demonstrate excellent consistency, with the subsequently collected arousal ratings as the only exception (r = 0.69). Our property (M = 0.85, SD = 0.11) and typicality (r = 0.72, 0.74, 0.88) data correlated strongly with external norms, again with the lowest validity for arousal (M = 0.41, SD = 0.08). To summarize, THINGSplus provides a large-scale, externally validated extension to existing object norms and an important extension to THINGS, allowing detailed selection of stimuli and control variables for a wide range of research interested in visual object processing, language, and semantic memory.
Collapse
Affiliation(s)
- Laura M Stoinski
- Max Planck Institute for Human Cognitive & Brain Sciences, Leipzig, Germany.
| | - Jonas Perkuhn
- Max Planck Institute for Human Cognitive & Brain Sciences, Leipzig, Germany
| | - Martin N Hebart
- Max Planck Institute for Human Cognitive & Brain Sciences, Leipzig, Germany
- Justus Liebig University, Gießen, Germany
| |
Collapse
|
48
|
Rossion B, Jacques C, Jonas J. The anterior fusiform gyrus: The ghost in the cortical face machine. Neurosci Biobehav Rev 2024; 158:105535. [PMID: 38191080 DOI: 10.1016/j.neubiorev.2024.105535] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/21/2023] [Revised: 12/19/2023] [Accepted: 01/03/2024] [Indexed: 01/10/2024]
Abstract
Face-selective regions in the human ventral occipito-temporal cortex (VOTC) have been defined for decades mainly with functional magnetic resonance imaging. This face-selective VOTC network is traditionally divided in a posterior 'core' system thought to subtend face perception, and regions of the anterior temporal lobe as a semantic memory component of an extended general system. In between these two putative systems lies the anterior fusiform gyrus and surrounding sulci, affected by magnetic susceptibility artifacts. Here we suggest that this methodological gap overlaps with and contributes to a conceptual gap between (visual) perception and semantic memory for faces. Filling this gap with intracerebral recordings and direct electrical stimulation reveals robust face-selectivity in the anterior fusiform gyrus and a crucial role of this region, especially in the right hemisphere, in identity recognition for both familiar and unfamiliar faces. Based on these observations, we propose an integrated theoretical framework for human face (identity) recognition according to which face-selective regions in the anterior fusiform gyrus join the dots between posterior and anterior cortical face memories.
Collapse
Affiliation(s)
- Bruno Rossion
- Université de Lorraine, CNRS, IMoPA, F-54000 Nancy, France; Université de Lorraine, CHRU-Nancy, Service de Neurologie, F-54000 Nancy, France.
| | | | - Jacques Jonas
- Université de Lorraine, CNRS, IMoPA, F-54000 Nancy, France; Université de Lorraine, CHRU-Nancy, Service de Neurologie, F-54000 Nancy, France
| |
Collapse
|
49
|
Monosov IE. Curiosity: primate neural circuits for novelty and information seeking. Nat Rev Neurosci 2024; 25:195-208. [PMID: 38263217 DOI: 10.1038/s41583-023-00784-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 12/13/2023] [Indexed: 01/25/2024]
Abstract
For many years, neuroscientists have investigated the behavioural, computational and neurobiological mechanisms that support value-based decisions, revealing how humans and animals make choices to obtain rewards. However, many decisions are influenced by factors other than the value of physical rewards or second-order reinforcers (such as money). For instance, animals (including humans) frequently explore novel objects that have no intrinsic value solely because they are novel and they exhibit the desire to gain information to reduce their uncertainties about the future, even if this information cannot lead to reward or assist them in accomplishing upcoming tasks. In this Review, I discuss how circuits in the primate brain responsible for detecting, predicting and assessing novelty and uncertainty regulate behaviour and give rise to these behavioural components of curiosity. I also briefly discuss how curiosity-related behaviours arise during postnatal development and point out some important reasons for the persistence of curiosity across generations.
Collapse
Affiliation(s)
- Ilya E Monosov
- Department of Neuroscience, Washington University School of Medicine, St. Louis, MO, USA.
- Department of Electrical Engineering, Washington University, St. Louis, MO, USA.
- Department of Biomedical Engineering, Washington University, St. Louis, MO, USA.
- Department of Neurosurgery, Washington University, St. Louis, MO, USA.
- Pain Center, Washington University, St. Louis, MO, USA.
| |
Collapse
|
50
|
Esmailpour H, Vogels R. Location-specific deviant responses to object sequences in macaque inferior temporal cortex. Sci Rep 2024; 14:3757. [PMID: 38355712 PMCID: PMC10866936 DOI: 10.1038/s41598-024-54298-0] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/09/2023] [Accepted: 02/11/2024] [Indexed: 02/16/2024] Open
Abstract
Many species learn temporal regularities in their visual environment, demonstrating visual statistical learning. In this study, we explored the sensitivity of macaque inferior temporal (IT) cortical neurons to transition probabilities of sequentially presented visual images, presented at different locations in the visual field. We exposed monkeys to sequences of two images, where the first image was presented either foveally or peripherally, and the second image was consistently presented foveally. Following several weeks of exposure, we recorded IT responses to assess differences between the exposed (Fixed) and new, Deviant sequences, where the identity of the first image in a sequence differed from the exposure phase. While enhanced responses to Deviant sequences were observed when both images of a pair were foveally presented during exposure, no such deviant responses were present when the first image was presented peripherally. This finding challenges the notion that mere exposure to image sequences always leads to deviant responses in macaque IT. The results highlight the complexity of the mechanisms underlying statistical learning in primates, particularly in the context of peripheral image presentations, emphasizing the need for further investigation into the origins of these responses in the IT cortex.
Collapse
Affiliation(s)
- Hamideh Esmailpour
- Laboratorium Voor Neuro- en Psychofysiologie, Department of Neurosciences, KU Leuven, Leuven, Belgium
- Leuven Brain Institute, KU Leuven, Leuven, Belgium
| | - Rufin Vogels
- Laboratorium Voor Neuro- en Psychofysiologie, Department of Neurosciences, KU Leuven, Leuven, Belgium.
- Leuven Brain Institute, KU Leuven, Leuven, Belgium.
| |
Collapse
|