1
|
Kamps FS, Chen EM, Kanwisher N, Saxe R. Representation of navigational affordances and ego-motion in the occipital place area. bioRxiv 2024:2024.04.30.591964. [PMID: 38746251 PMCID: PMC11092631 DOI: 10.1101/2024.04.30.591964] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/16/2024]
Abstract
Humans effortlessly use vision to plan and guide navigation through the local environment, or "scene". A network of three cortical regions responds selectively to visual scene information, including the occipital (OPA), parahippocampal (PPA), and medial place areas (MPA) - but how this network supports visually-guided navigation is unclear. Recent evidence suggests that one region in particular, the OPA, supports visual representations for navigation, while PPA and MPA support other aspects of scene processing. However, most previous studies tested only static scene images, which lack the dynamic experience of navigating through scenes. We used dynamic movie stimuli to test whether OPA, PPA, and MPA represent two critical kinds of navigationally-relevant information: navigational affordances (e.g., can I walk to the left, right, or both?) and ego-motion (e.g., am I walking forward or backward? turning left or right?). We found that OPA is sensitive to both affordances and ego-motion, as well as the conflict between these cues - e.g., turning toward versus away from an open doorway. These effects were significantly weaker or absent in PPA and MPA. Responses in OPA were also dissociable from those in early visual cortex, consistent with the idea that OPA responses are not merely explained by lower-level visual features. OPA responses to affordances and ego-motion were stronger in the contralateral than ipsilateral visual field, suggesting that OPA encodes navigationally relevant information within an egocentric reference frame. Taken together, these results support the hypothesis that OPA contains visual representations that are useful for planning and guiding navigation through scenes.
Collapse
|
2
|
Tuckute G, Kanwisher N, Fedorenko E. Language in Brains, Minds, and Machines. Annu Rev Neurosci 2024. [PMID: 38669478 DOI: 10.1146/annurev-neuro-120623-101142] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 04/28/2024]
Abstract
It has long been argued that only humans could produce and understand language. But now, for the first time, artificial language models (LMs) achieve this feat. Here we survey the new purchase LMs are providing on the question of how language is implemented in the brain. We discuss why, a priori, LMs might be expected to share similarities with the human language system. We then summarize evidence that LMs represent linguistic information similarly enough to humans to enable relatively accurate brain encoding and decoding during language processing. Finally, we examine which LM properties-their architecture, task performance, or training-are critical for capturing human neural responses to language and review studies using LMs as in silico model organisms for testing hypotheses about language. These ongoing investigations bring us closer to understanding the representations and processes that underlie our ability to comprehend sentences and express thoughts in language.
Collapse
Affiliation(s)
- Greta Tuckute
- Department of Brain and Cognitive Sciences and McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, Massachusetts, USA;
| | - Nancy Kanwisher
- Department of Brain and Cognitive Sciences and McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, Massachusetts, USA;
| | - Evelina Fedorenko
- Department of Brain and Cognitive Sciences and McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, Massachusetts, USA;
| |
Collapse
|
3
|
Mahowald K, Ivanova AA, Blank IA, Kanwisher N, Tenenbaum JB, Fedorenko E. Dissociating language and thought in large language models. Trends Cogn Sci 2024:S1364-6613(24)00027-5. [PMID: 38508911 DOI: 10.1016/j.tics.2024.01.011] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/06/2023] [Revised: 01/31/2024] [Accepted: 01/31/2024] [Indexed: 03/22/2024]
Abstract
Large language models (LLMs) have come closest among all models to date to mastering human language, yet opinions about their linguistic and cognitive capabilities remain split. Here, we evaluate LLMs using a distinction between formal linguistic competence (knowledge of linguistic rules and patterns) and functional linguistic competence (understanding and using language in the world). We ground this distinction in human neuroscience, which has shown that formal and functional competence rely on different neural mechanisms. Although LLMs are surprisingly good at formal competence, their performance on functional competence tasks remains spotty and often requires specialized fine-tuning and/or coupling with external modules. We posit that models that use language in human-like ways would need to master both of these competence types, which, in turn, could require the emergence of separate mechanisms specialized for formal versus functional linguistic competence.
Collapse
|
4
|
Kosakowski HL, Norman-Haignere S, Mynick A, Takahashi A, Saxe R, Kanwisher N. Preliminary evidence for selective cortical responses to music in one-month-old infants. Dev Sci 2023; 26:e13387. [PMID: 36951215 DOI: 10.1111/desc.13387] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/13/2022] [Revised: 02/17/2023] [Accepted: 02/21/2023] [Indexed: 03/24/2023]
Abstract
Prior studies have observed selective neural responses in the adult human auditory cortex to music and speech that cannot be explained by the differing lower-level acoustic properties of these stimuli. Does infant cortex exhibit similarly selective responses to music and speech shortly after birth? To answer this question, we attempted to collect functional magnetic resonance imaging (fMRI) data from 45 sleeping infants (2.0- to 11.9-weeks-old) while they listened to monophonic instrumental lullabies and infant-directed speech produced by a mother. To match acoustic variation between music and speech sounds we (1) recorded music from instruments that had a similar spectral range as female infant-directed speech, (2) used a novel excitation-matching algorithm to match the cochleagrams of music and speech stimuli, and (3) synthesized "model-matched" stimuli that were matched in spectrotemporal modulation statistics to (yet perceptually distinct from) music or speech. Of the 36 infants we collected usable data from, 19 had significant activations to sounds overall compared to scanner noise. From these infants, we observed a set of voxels in non-primary auditory cortex (NPAC) but not in Heschl's Gyrus that responded significantly more to music than to each of the other three stimulus types (but not significantly more strongly than to the background scanner noise). In contrast, our planned analyses did not reveal voxels in NPAC that responded more to speech than to model-matched speech, although other unplanned analyses did. These preliminary findings suggest that music selectivity arises within the first month of life. A video abstract of this article can be viewed at https://youtu.be/c8IGFvzxudk. RESEARCH HIGHLIGHTS: Responses to music, speech, and control sounds matched for the spectrotemporal modulation-statistics of each sound were measured from 2- to 11-week-old sleeping infants using fMRI. Auditory cortex was significantly activated by these stimuli in 19 out of 36 sleeping infants. Selective responses to music compared to the three other stimulus classes were found in non-primary auditory cortex but not in nearby Heschl's Gyrus. Selective responses to speech were not observed in planned analyses but were observed in unplanned, exploratory analyses.
Collapse
Affiliation(s)
- Heather L Kosakowski
- Department of Brain and Cognitive Sciences, Massachusetts Institute, of Technology, Cambridge, Massachusetts, USA
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, Massachusetts, USA
- Center for Brains, Minds and Machines, Cambridge, Massachusetts, USA
| | | | - Anna Mynick
- Psychological and Brain Sciences, Dartmouth College, Hannover, New Hampshire, USA
| | - Atsushi Takahashi
- Department of Brain and Cognitive Sciences, Massachusetts Institute, of Technology, Cambridge, Massachusetts, USA
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, Massachusetts, USA
| | - Rebecca Saxe
- Department of Brain and Cognitive Sciences, Massachusetts Institute, of Technology, Cambridge, Massachusetts, USA
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, Massachusetts, USA
- Center for Brains, Minds and Machines, Cambridge, Massachusetts, USA
| | - Nancy Kanwisher
- Department of Brain and Cognitive Sciences, Massachusetts Institute, of Technology, Cambridge, Massachusetts, USA
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, Massachusetts, USA
- Center for Brains, Minds and Machines, Cambridge, Massachusetts, USA
| |
Collapse
|
5
|
Dobs K, Yuan J, Martinez J, Kanwisher N. Behavioral signatures of face perception emerge in deep neural networks optimized for face recognition. Proc Natl Acad Sci U S A 2023; 120:e2220642120. [PMID: 37523537 PMCID: PMC10410721 DOI: 10.1073/pnas.2220642120] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/06/2022] [Accepted: 06/08/2023] [Indexed: 08/02/2023] Open
Abstract
Human face recognition is highly accurate and exhibits a number of distinctive and well-documented behavioral "signatures" such as the use of a characteristic representational space, the disproportionate performance cost when stimuli are presented upside down, and the drop in accuracy for faces from races the participant is less familiar with. These and other phenomena have long been taken as evidence that face recognition is "special". But why does human face perception exhibit these properties in the first place? Here, we use deep convolutional neural networks (CNNs) to test the hypothesis that all of these signatures of human face perception result from optimization for the task of face recognition. Indeed, as predicted by this hypothesis, these phenomena are all found in CNNs trained on face recognition, but not in CNNs trained on object recognition, even when additionally trained to detect faces while matching the amount of face experience. To test whether these signatures are in principle specific to faces, we optimized a CNN on car discrimination and tested it on upright and inverted car images. As we found for face perception, the car-trained network showed a drop in performance for inverted vs. upright cars. Similarly, CNNs trained on inverted faces produced an inverted face inversion effect. These findings show that the behavioral signatures of human face perception reflect and are well explained as the result of optimization for the task of face recognition, and that the nature of the computations underlying this task may not be so special after all.
Collapse
Affiliation(s)
- Katharina Dobs
- Department of Psychology, Justus Liebig University Giessen, Giessen35394, Germany
- Center for Mind, Brain and Behavior (CMBB), University of Marburg and Justus Liebig University Giessen, Marburg35302, Germany
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA02139
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA02139
| | - Joanne Yuan
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA02139
| | - Julio Martinez
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA02139
- Department of Psychology, Stanford University, Stanford, CA94305
| | - Nancy Kanwisher
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA02139
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA02139
| |
Collapse
|
6
|
Kanwisher N, Khosla M, Dobs K. Using artificial neural networks to ask 'why' questions of minds and brains. Trends Neurosci 2023; 46:240-254. [PMID: 36658072 DOI: 10.1016/j.tins.2022.12.008] [Citation(s) in RCA: 12] [Impact Index Per Article: 12.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/19/2022] [Revised: 11/29/2022] [Accepted: 12/22/2022] [Indexed: 01/19/2023]
Abstract
Neuroscientists have long characterized the properties and functions of the nervous system, and are increasingly succeeding in answering how brains perform the tasks they do. But the question 'why' brains work the way they do is asked less often. The new ability to optimize artificial neural networks (ANNs) for performance on human-like tasks now enables us to approach these 'why' questions by asking when the properties of networks optimized for a given task mirror the behavioral and neural characteristics of humans performing the same task. Here we highlight the recent success of this strategy in explaining why the visual and auditory systems work the way they do, at both behavioral and neural levels.
Collapse
Affiliation(s)
- Nancy Kanwisher
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA, USA; McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA, USA
| | - Meenakshi Khosla
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA, USA; McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA, USA
| | - Katharina Dobs
- Department of Psychology, Justus Liebig University Giessen, Giessen, Germany; Center for Mind, Brain and Behavior (CMBB), University of Marburg and Justus Liebig University, Giessen, Germany.
| |
Collapse
|
7
|
Kanwisher N, Gupta P, Dobs K. CNNs reveal the computational implausibility of the expertise hypothesis. iScience 2023; 26:105976. [PMID: 36794151 PMCID: PMC9923184 DOI: 10.1016/j.isci.2023.105976] [Citation(s) in RCA: 5] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/13/2022] [Revised: 11/07/2022] [Accepted: 01/11/2023] [Indexed: 01/15/2023] Open
Abstract
Face perception has long served as a classic example of domain specificity of mind and brain. But an alternative "expertise" hypothesis holds that putatively face-specific mechanisms are actually domain-general, and can be recruited for the perception of other objects of expertise (e.g., cars for car experts). Here, we demonstrate the computational implausibility of this hypothesis: Neural network models optimized for generic object categorization provide a better foundation for expert fine-grained discrimination than do models optimized for face recognition.
Collapse
Affiliation(s)
- Nancy Kanwisher
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA 02139, USA,McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA 02139, USA
| | - Pranjul Gupta
- Department of Psychology, Justus-Liebig University Giessen, 35394 Giessen, Germany
| | - Katharina Dobs
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA 02139, USA,McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA 02139, USA,Department of Psychology, Justus-Liebig University Giessen, 35394 Giessen, Germany,Center for Mind, Brain and Behavior (CMBB), University of Marburg and Justus-Liebig University, 35032 Marburg, Germany,Corresponding author
| |
Collapse
|
8
|
abate A, Mieczkowski E, Khosla M, DiCarlo J, Kanwisher N, Murty NAR. Computational Models Recapitulate Key Signatures of Face, Body and Scene Processing in the FFA, EBA, and PPA. J Vis 2022. [DOI: 10.1167/jov.22.14.4337] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/23/2022] Open
Affiliation(s)
- Alex abate
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology
| | - Elizabeth Mieczkowski
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology
- McGovern Institute for Brain Research, Massachusetts Institute of Technology
| | - Meenakshi Khosla
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology
- McGovern Institute for Brain Research, Massachusetts Institute of Technology
| | - James DiCarlo
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology
- McGovern Institute for Brain Research, Massachusetts Institute of Technology
- The Center for Brains, Minds and Machines, Massachusetts Institute of Technology
| | - Nancy Kanwisher
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology
- McGovern Institute for Brain Research, Massachusetts Institute of Technology
- The Center for Brains, Minds and Machines, Massachusetts Institute of Technology
| | - N Apurva Ratan Murty
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology
- McGovern Institute for Brain Research, Massachusetts Institute of Technology
- The Center for Brains, Minds and Machines, Massachusetts Institute of Technology
| |
Collapse
|
9
|
Santiago-Reyes G, O'Connell T, Kanwisher N. Artificial neural networks predict human eye movement patterns as an emergent property of training for object classification. J Vis 2022. [DOI: 10.1167/jov.22.14.4194] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/23/2022] Open
|
10
|
Murty AR, Abate A, Kamps F, DiCarlo J, Kanwisher N. Functionally distinct sub-regions of the parahippocampal place area revealed by model-based neural control. J Vis 2022. [DOI: 10.1167/jov.22.14.4339] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/23/2022] Open
Affiliation(s)
- Apurva Ratan Murty
- McGovern Institute for Brain Research, Massachusetts Institute of Technology
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology
| | - Alex Abate
- McGovern Institute for Brain Research, Massachusetts Institute of Technology
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology
| | - Frederik Kamps
- McGovern Institute for Brain Research, Massachusetts Institute of Technology
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology
| | - James DiCarlo
- McGovern Institute for Brain Research, Massachusetts Institute of Technology
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology
| | - Nancy Kanwisher
- McGovern Institute for Brain Research, Massachusetts Institute of Technology
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology
| |
Collapse
|
11
|
Khosla M, Murty NAR, Kanwisher N. Data-driven component modeling reveals the functional organization of high-level visual cortex. J Vis 2022. [DOI: 10.1167/jov.22.14.4184] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/23/2022] Open
|
12
|
Khosla M, Ratan Murty NA, Kanwisher N. A highly selective response to food in human visual cortex revealed by hypothesis-free voxel decomposition. Curr Biol 2022; 32:4159-4171.e9. [PMID: 36027910 PMCID: PMC9561032 DOI: 10.1016/j.cub.2022.08.009] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/23/2022] [Revised: 08/03/2022] [Accepted: 08/05/2022] [Indexed: 12/14/2022]
Abstract
Prior work has identified cortical regions selectively responsive to specific categories of visual stimuli. However, this hypothesis-driven work cannot reveal how prominent these category selectivities are in the overall functional organization of the visual cortex, or what others might exist that scientists have not thought to look for. Furthermore, standard voxel-wise tests cannot detect distinct neural selectivities that coexist within voxels. To overcome these limitations, we used data-driven voxel decomposition methods to identify the main components underlying fMRI responses to thousands of complex photographic images. Our hypothesis-neutral analysis rediscovered components selective for faces, places, bodies, and words, validating our method and showing that these selectivities are dominant features of the ventral visual pathway. The analysis also revealed an unexpected component with a distinct anatomical distribution that responded highly selectively to images of food. Alternative accounts based on low- to mid-level visual features, such as color, shape, or texture, failed to account for the food selectivity of this component. High-throughput testing and control experiments with matched stimuli on a highly accurate computational model of this component confirm its selectivity for food. We registered our methods and hypotheses before replicating them on held-out participants and in a novel dataset. These findings demonstrate the power of data-driven methods and show that the dominant neural responses of the ventral visual pathway include not only selectivities for faces, scenes, bodies, and words but also the visually heterogeneous category of food, thus constraining accounts of when and why functional specialization arises in the cortex.
Collapse
Affiliation(s)
- Meenakshi Khosla
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA, USA.
| | - N Apurva Ratan Murty
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA, USA
| | - Nancy Kanwisher
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA, USA
| |
Collapse
|
13
|
Kamps FS, Richardson H, Murty NAR, Kanwisher N, Saxe R. Using child-friendly movie stimuli to study the development of face, place, and object regions from age 3 to 12 years. Hum Brain Mapp 2022; 43:2782-2800. [PMID: 35274789 PMCID: PMC9120553 DOI: 10.1002/hbm.25815] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/26/2021] [Revised: 02/11/2022] [Accepted: 02/13/2022] [Indexed: 01/21/2023] Open
Abstract
Scanning young children while they watch short, engaging, commercially-produced movies has emerged as a promising approach for increasing data retention and quality. Movie stimuli also evoke a richer variety of cognitive processes than traditional experiments, allowing the study of multiple aspects of brain development simultaneously. However, because these stimuli are uncontrolled, it is unclear how effectively distinct profiles of brain activity can be distinguished from the resulting data. Here we develop an approach for identifying multiple distinct subject-specific Regions of Interest (ssROIs) using fMRI data collected during movie-viewing. We focused on the test case of higher-level visual regions selective for faces, scenes, and objects. Adults (N = 13) were scanned while viewing a 5.6-min child-friendly movie, as well as a traditional localizer experiment with blocks of faces, scenes, and objects. We found that just 2.7 min of movie data could identify subject-specific face, scene, and object regions. While successful, movie-defined ssROIS still showed weaker domain selectivity than traditional ssROIs. Having validated our approach in adults, we then used the same methods on movie data collected from 3 to 12-year-old children (N = 122). Movie response timecourses in 3-year-old children's face, scene, and object regions were already significantly and specifically predicted by timecourses from the corresponding regions in adults. We also found evidence of continued developmental change, particularly in the face-selective posterior superior temporal sulcus. Taken together, our results reveal both early maturity and functional change in face, scene, and object regions, and more broadly highlight the promise of short, child-friendly movies for developmental cognitive neuroscience.
Collapse
Affiliation(s)
- Frederik S. Kamps
- Department of Brain and Cognitive SciencesMassachusetts Institute of TechnologyCambridgeMassachusettsUSA
| | - Hilary Richardson
- School of Philosophy, Psychology and Language SciencesUniversity of EdinburghEdinburghUK
| | - N. Apurva Ratan Murty
- Department of Brain and Cognitive SciencesMassachusetts Institute of TechnologyCambridgeMassachusettsUSA
| | - Nancy Kanwisher
- Department of Brain and Cognitive SciencesMassachusetts Institute of TechnologyCambridgeMassachusettsUSA
| | - Rebecca Saxe
- Department of Brain and Cognitive SciencesMassachusetts Institute of TechnologyCambridgeMassachusettsUSA
| |
Collapse
|
14
|
Abstract
Successful engagement with the world requires the ability to predict what will happen next. Here, we investigate how the brain makes a fundamental prediction about the physical world: whether the situation in front of us is stable, and hence likely to stay the same, or unstable, and hence likely to change in the immediate future. Specifically, we ask if judgments of stability can be supported by the kinds of representations that have proven to be highly effective at visual object recognition in both machines and brains, or instead if the ability to determine the physical stability of natural scenes may require generative algorithms that simulate the physics of the world. To find out, we measured responses in both convolutional neural networks (CNNs) and the brain (using fMRI) to natural images of physically stable versus unstable scenarios. We find no evidence for generalizable representations of physical stability in either standard CNNs trained on visual object and scene classification (ImageNet), or in the human ventral visual pathway, which has long been implicated in the same process. However, in frontoparietal regions previously implicated in intuitive physical reasoning we find both scenario-invariant representations of physical stability, and higher univariate responses to unstable than stable scenes. These results demonstrate abstract representations of physical stability in the dorsal but not ventral pathway, consistent with the hypothesis that the computations underlying stability entail not just pattern classification but forward physical simulation.
Collapse
Affiliation(s)
- RT Pramod
- Center for Brains, Minds and Machines, Massachusetts Institute of TechnologyCambridgeUnited States
- Department of Brain and Cognitive Sciences, Massachusetts Institute of TechnologyCambridgeUnited States
| | - Michael A Cohen
- Department of Brain and Cognitive Sciences, Massachusetts Institute of TechnologyCambridgeUnited States
- Amherst CollegeAmherstUnited States
| | - Joshua B Tenenbaum
- Center for Brains, Minds and Machines, Massachusetts Institute of TechnologyCambridgeUnited States
- Department of Brain and Cognitive Sciences, Massachusetts Institute of TechnologyCambridgeUnited States
| | - Nancy Kanwisher
- Center for Brains, Minds and Machines, Massachusetts Institute of TechnologyCambridgeUnited States
- Department of Brain and Cognitive Sciences, Massachusetts Institute of TechnologyCambridgeUnited States
| |
Collapse
|
15
|
Norman-Haignere SV, Feather J, Boebinger D, Brunner P, Ritaccio A, McDermott JH, Schalk G, Kanwisher N. A neural population selective for song in human auditory cortex. Curr Biol 2022; 32:1470-1484.e12. [PMID: 35196507 PMCID: PMC9092957 DOI: 10.1016/j.cub.2022.01.069] [Citation(s) in RCA: 23] [Impact Index Per Article: 11.5] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/23/2021] [Revised: 10/26/2021] [Accepted: 01/24/2022] [Indexed: 12/18/2022]
Abstract
How is music represented in the brain? While neuroimaging has revealed some spatial segregation between responses to music versus other sounds, little is known about the neural code for music itself. To address this question, we developed a method to infer canonical response components of human auditory cortex using intracranial responses to natural sounds, and further used the superior coverage of fMRI to map their spatial distribution. The inferred components replicated many prior findings, including distinct neural selectivity for speech and music, but also revealed a novel component that responded nearly exclusively to music with singing. Song selectivity was not explainable by standard acoustic features, was located near speech- and music-selective responses, and was also evident in individual electrodes. These results suggest that representations of music are fractionated into subpopulations selective for different types of music, one of which is specialized for the analysis of song.
Collapse
Affiliation(s)
- Sam V Norman-Haignere
- Zuckerman Institute, Columbia University, New York, NY, USA; HHMI Fellow of the Life Sciences Research Foundation, Chevy Chase, MD, USA; Laboratoire des Sytèmes Perceptifs, Département d'Études Cognitives, ENS, PSL University, CNRS, Paris, France; Department of Biostatistics & Computational Biology, University of Rochester Medical Center, Rochester, NY, USA; Department of Neuroscience, University of Rochester Medical Center, Rochester, NY, USA; Department of Biomedical Engineering, University of Rochester, Rochester, NY, USA; Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA, USA.
| | - Jenelle Feather
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA, USA; McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA, USA; Center for Brains, Minds and Machines, Cambridge, MA, USA
| | - Dana Boebinger
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA, USA; McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA, USA; Program in Speech and Hearing Biosciences and Technology, Harvard University, Cambridge, MA, USA
| | - Peter Brunner
- Department of Neurology, Albany Medical College, Albany, NY, USA; National Center for Adaptive Neurotechnologies, Albany, NY, USA; Department of Neurosurgery, Washington University School of Medicine, St. Louis, MO, USA
| | - Anthony Ritaccio
- Department of Neurology, Albany Medical College, Albany, NY, USA; Department of Neurology, Mayo Clinic, Jacksonville, FL, USA
| | - Josh H McDermott
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA, USA; McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA, USA; Center for Brains, Minds and Machines, Cambridge, MA, USA; Program in Speech and Hearing Biosciences and Technology, Harvard University, Cambridge, MA, USA
| | - Gerwin Schalk
- Department of Neurology, Albany Medical College, Albany, NY, USA
| | - Nancy Kanwisher
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA, USA; McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA, USA; Center for Brains, Minds and Machines, Cambridge, MA, USA
| |
Collapse
|
16
|
Norman-Haignere SV, Feather J, Boebinger D, Brunner P, Ritaccio A, McDermott JH, Schalk G, Kanwisher N. A neural population selective for song in human auditory cortex. Curr Biol 2022; 32:1454-1455. [PMID: 35349804 DOI: 10.1016/j.cub.2022.03.016] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/25/2022]
|
17
|
Dobs K, Martinez J, Kell AJE, Kanwisher N. Brain-like functional specialization emerges spontaneously in deep neural networks. Sci Adv 2022; 8:eabl8913. [PMID: 35294241 PMCID: PMC8926347 DOI: 10.1126/sciadv.abl8913] [Citation(s) in RCA: 30] [Impact Index Per Article: 15.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/11/2021] [Accepted: 01/21/2022] [Indexed: 05/10/2023]
Abstract
The human brain contains multiple regions with distinct, often highly specialized functions, from recognizing faces to understanding language to thinking about what others are thinking. However, it remains unclear why the cortex exhibits this high degree of functional specialization in the first place. Here, we consider the case of face perception using artificial neural networks to test the hypothesis that functional segregation of face recognition in the brain reflects a computational optimization for the broader problem of visual recognition of faces and other visual categories. We find that networks trained on object recognition perform poorly on face recognition and vice versa and that networks optimized for both tasks spontaneously segregate themselves into separate systems for faces and objects. We then show functional segregation to varying degrees for other visual categories, revealing a widespread tendency for optimization (without built-in task-specific inductive biases) to lead to functional specialization in machines and, we conjecture, also brains.
Collapse
Affiliation(s)
- Katharina Dobs
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA, USA
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA, USA
- Department of Psychology, Justus Liebig University Giessen, Giessen, Germany
| | - Julio Martinez
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA, USA
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA, USA
- Department of Psychology, Stanford University, Stanford, CA, USA
| | | | - Nancy Kanwisher
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA, USA
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA, USA
| |
Collapse
|
18
|
Kosakowski HL, Cohen MA, Takahashi A, Keil B, Kanwisher N, Saxe R. Selective responses to faces, scenes, and bodies in the ventral visual pathway of infants. Curr Biol 2022; 32:265-274.e5. [PMID: 34784506 PMCID: PMC8792213 DOI: 10.1016/j.cub.2021.10.064] [Citation(s) in RCA: 24] [Impact Index Per Article: 12.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/04/2021] [Revised: 09/27/2021] [Accepted: 10/28/2021] [Indexed: 01/26/2023]
Abstract
Three of the most robust functional landmarks in the human brain are the selective responses to faces in the fusiform face area (FFA), scenes in the parahippocampal place area (PPA), and bodies in the extrastriate body area (EBA). Are the selective responses of these regions present early in development or do they require many years to develop? Prior evidence leaves this question unresolved. We designed a new 32-channel infant magnetic resonance imaging (MRI) coil and collected high-quality functional MRI (fMRI) data from infants (2-9 months of age) while they viewed stimuli from four conditions-faces, bodies, objects, and scenes. We find that infants have face-, scene-, and body-selective responses in the location of the adult FFA, PPA, and EBA, respectively, powerfully constraining accounts of cortical development.
Collapse
Affiliation(s)
- Heather L Kosakowski
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, 77 Massachusetts Avenue, Cambridge, MA, USA.
| | - Michael A Cohen
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, 77 Massachusetts Avenue, Cambridge, MA, USA; Department of Psychology and Program in Neuroscience, Amherst College, 220 South Pleasant Street, Amherst, MA, USA
| | - Atsushi Takahashi
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, 77 Massachusetts Avenue, Cambridge, MA, USA
| | - Boris Keil
- Institute of Medical Physics and Radiation Protection, Department of Life Science Engineering, Mittelhessen University of Applied Science, Giessen, Germany
| | - Nancy Kanwisher
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, 77 Massachusetts Avenue, Cambridge, MA, USA
| | - Rebecca Saxe
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, 77 Massachusetts Avenue, Cambridge, MA, USA
| |
Collapse
|
19
|
Schrimpf M, Blank IA, Tuckute G, Kauf C, Hosseini EA, Kanwisher N, Tenenbaum JB, Fedorenko E. The neural architecture of language: Integrative modeling converges on predictive processing. Proc Natl Acad Sci U S A 2021; 118:e2105646118. [PMID: 34737231 PMCID: PMC8694052 DOI: 10.1073/pnas.2105646118] [Citation(s) in RCA: 103] [Impact Index Per Article: 34.3] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 09/03/2021] [Indexed: 01/30/2023] Open
Abstract
The neuroscience of perception has recently been revolutionized with an integrative modeling approach in which computation, brain function, and behavior are linked across many datasets and many computational models. By revealing trends across models, this approach yields novel insights into cognitive and neural mechanisms in the target domain. We here present a systematic study taking this approach to higher-level cognition: human language processing, our species' signature cognitive skill. We find that the most powerful "transformer" models predict nearly 100% of explainable variance in neural responses to sentences and generalize across different datasets and imaging modalities (functional MRI and electrocorticography). Models' neural fits ("brain score") and fits to behavioral responses are both strongly correlated with model accuracy on the next-word prediction task (but not other language tasks). Model architecture appears to substantially contribute to neural fit. These results provide computationally explicit evidence that predictive processing fundamentally shapes the language comprehension mechanisms in the human brain.
Collapse
Affiliation(s)
- Martin Schrimpf
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA 02139;
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA 02139
- Center for Brains, Minds and Machines, Massachusetts Institute of Technology, Cambridge, MA 02139
| | - Idan Asher Blank
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA 02139
- Department of Psychology, University of California, Los Angeles, CA 90095
| | - Greta Tuckute
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA 02139
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA 02139
| | - Carina Kauf
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA 02139
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA 02139
| | - Eghbal A Hosseini
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA 02139
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA 02139
| | - Nancy Kanwisher
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA 02139;
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA 02139
- Center for Brains, Minds and Machines, Massachusetts Institute of Technology, Cambridge, MA 02139
| | - Joshua B Tenenbaum
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA 02139
- Center for Brains, Minds and Machines, Massachusetts Institute of Technology, Cambridge, MA 02139
| | - Evelina Fedorenko
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA 02139;
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA 02139
| |
Collapse
|
20
|
Kosakowski HL, Cohen M, Kanwisher N, Saxe R. Object Responses in the Ventral and Dorsal Pathway of Human Infants. J Vis 2021. [DOI: 10.1167/jov.21.9.30] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/24/2022] Open
Affiliation(s)
| | - Michael Cohen
- Massachussetts Institute of Technology
- Amherst College
| | | | | |
Collapse
|
21
|
Lydic K, Pantazis D, Kanwisher N. Can MEG Source Localization Reveal the Time Course of Processing in the FFA, PPA, and EBA? J Vis 2021. [DOI: 10.1167/jov.21.9.2758] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/24/2022] Open
|
22
|
Dobs K, Yuan J, Martinez J, Kanwisher N. Using task-optimized neural networks to understand how experience might shape human face perception. J Vis 2021. [DOI: 10.1167/jov.21.9.2292] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/24/2022] Open
Affiliation(s)
- Katharina Dobs
- Justus-Liebig University Giessen
- Massachusetts Institute of Technology
| | | | | | | |
Collapse
|
23
|
Kamps F, Richardson H, Kanwisher N, Saxe R. Early emergence and later development of face, scene, and object regions revealed by natural vision. J Vis 2021. [DOI: 10.1167/jov.21.9.2587] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/24/2022] Open
|
24
|
Ratan Murty NA, Bashivan P, Abate A, DiCarlo JJ, Kanwisher N. Computational models of category-selective brain regions enable high-throughput tests of selectivity. Nat Commun 2021; 12:5540. [PMID: 34545079 PMCID: PMC8452636 DOI: 10.1038/s41467-021-25409-6] [Citation(s) in RCA: 30] [Impact Index Per Article: 10.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/28/2021] [Accepted: 08/04/2021] [Indexed: 02/08/2023] Open
Abstract
Cortical regions apparently selective to faces, places, and bodies have provided important evidence for domain-specific theories of human cognition, development, and evolution. But claims of category selectivity are not quantitatively precise and remain vulnerable to empirical refutation. Here we develop artificial neural network-based encoding models that accurately predict the response to novel images in the fusiform face area, parahippocampal place area, and extrastriate body area, outperforming descriptive models and experts. We use these models to subject claims of category selectivity to strong tests, by screening for and synthesizing images predicted to produce high responses. We find that these high-response-predicted images are all unambiguous members of the hypothesized preferred category for each region. These results provide accurate, image-computable encoding models of each category-selective region, strengthen evidence for domain specificity in the brain, and point the way for future research characterizing the functional organization of the brain with unprecedented computational precision.
Collapse
Affiliation(s)
- N Apurva Ratan Murty
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA, USA.
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA, USA.
- The Center for Brains, Minds and Machines, Massachusetts Institute of Technology, Cambridge, MA, USA.
| | - Pouya Bashivan
- Department of Physiology, McGill University, Montréal, QC, Canada
| | - Alex Abate
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA, USA
| | - James J DiCarlo
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA, USA
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA, USA
- The Center for Brains, Minds and Machines, Massachusetts Institute of Technology, Cambridge, MA, USA
| | - Nancy Kanwisher
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA, USA
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA, USA
- The Center for Brains, Minds and Machines, Massachusetts Institute of Technology, Cambridge, MA, USA
| |
Collapse
|
25
|
Ghotra A, Kosakowski HL, Takahashi A, Etzel R, May MW, Scholz A, Jansen A, Wald LL, Kanwisher N, Saxe R, Keil B. A size-adaptive 32-channel array coil for awake infant neuroimaging at 3 Tesla MRI. Magn Reson Med 2021; 86:1773-1785. [PMID: 33829546 DOI: 10.1002/mrm.28791] [Citation(s) in RCA: 7] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/06/2021] [Revised: 03/10/2021] [Accepted: 03/11/2021] [Indexed: 11/12/2022]
Abstract
PURPOSE Functional magnetic resonance imaging (fMRI) during infancy poses challenges due to practical, methodological, and analytical considerations. The aim of this study was to implement a hardware-related approach to increase subject compliance for fMRI involving awake infants. To accomplish this, we designed, constructed, and evaluated an adaptive 32-channel array coil. METHODS To allow imaging with a close-fitting head array coil for infants aged 1-18 months, an adjustable head coil concept was developed. The coil setup facilitates a half-seated scanning position to improve the infant's overall scan compliance. Earmuff compartments are integrated directly into the coil housing to enable the usage of sound protection without losing a snug fit of the coil around the infant's head. The constructed array coil was evaluated from phantom data using bench-level metrics, signal-to-noise ratio (SNR) performances, and accelerated imaging capabilities for both in-plane and simultaneous multislice (SMS) reconstruction methodologies. Furthermore, preliminary fMRI data were acquired to evaluate the in vivo coil performance. RESULTS Phantom data showed a 2.7-fold SNR increase on average when compared with a commercially available 32-channel head coil. At the center and periphery regions of the infant head phantom, the SNR gains were measured to be 1.25-fold and 3-fold, respectively. The infant coil further showed favorable encoding capabilities for undersampled k-space reconstruction methods and SMS techniques. CONCLUSIONS An infant-friendly head coil array was developed to improve sensitivity, spatial resolution, accelerated encoding, motion insensitivity, and subject tolerance in pediatric MRI. The adaptive 32-channel array coil is well-suited for fMRI acquisitions in awake infants.
Collapse
Affiliation(s)
- Anpreet Ghotra
- Institute of Medical Physics and Radiation Protection, Department of Life Science Engineering, TH Mittelhessen University of Applied Sciences, Giessen, Germany
| | - Heather L Kosakowski
- Department of Brain and Cognitive Sciences and McGovern Institute, Massachusetts Institute of Technology, Cambridge, MA, USA
| | - Atsushi Takahashi
- Department of Brain and Cognitive Sciences and McGovern Institute, Massachusetts Institute of Technology, Cambridge, MA, USA
| | - Robin Etzel
- Institute of Medical Physics and Radiation Protection, Department of Life Science Engineering, TH Mittelhessen University of Applied Sciences, Giessen, Germany
| | - Markus W May
- Institute of Medical Physics and Radiation Protection, Department of Life Science Engineering, TH Mittelhessen University of Applied Sciences, Giessen, Germany
| | - Alina Scholz
- Institute of Medical Physics and Radiation Protection, Department of Life Science Engineering, TH Mittelhessen University of Applied Sciences, Giessen, Germany
| | - Andreas Jansen
- Department of Psychiatry and Psychotherapy, University of Marburg, Marburg, Germany
- Center for Mind, Brain and Behavior (CMBB), Marburg, Germany
| | - Lawrence L Wald
- Athinoula A. Martinos Center for Biomedical Imaging, Department of Radiology, Harvard Medical School, Massachusetts General Hospital, Charlestown, MA, USA
- Harvard-MIT Division of Health Sciences and Technology, Cambridge, MA, USA
| | - Nancy Kanwisher
- Department of Brain and Cognitive Sciences and McGovern Institute, Massachusetts Institute of Technology, Cambridge, MA, USA
| | - Rebecca Saxe
- Department of Brain and Cognitive Sciences and McGovern Institute, Massachusetts Institute of Technology, Cambridge, MA, USA
| | - Boris Keil
- Institute of Medical Physics and Radiation Protection, Department of Life Science Engineering, TH Mittelhessen University of Applied Sciences, Giessen, Germany
- Center for Mind, Brain and Behavior (CMBB), Marburg, Germany
| |
Collapse
|
26
|
Boebinger D, Norman-Haignere SV, McDermott JH, Kanwisher N. Music-selective neural populations arise without musical training. J Neurophysiol 2021; 125:2237-2263. [PMID: 33596723 PMCID: PMC8285655 DOI: 10.1152/jn.00588.2020] [Citation(s) in RCA: 19] [Impact Index Per Article: 6.3] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/12/2020] [Revised: 02/12/2021] [Accepted: 02/12/2021] [Indexed: 11/22/2022] Open
Abstract
Recent work has shown that human auditory cortex contains neural populations anterior and posterior to primary auditory cortex that respond selectively to music. However, it is unknown how this selectivity for music arises. To test whether musical training is necessary, we measured fMRI responses to 192 natural sounds in 10 people with almost no musical training. When voxel responses were decomposed into underlying components, this group exhibited a music-selective component that was very similar in response profile and anatomical distribution to that previously seen in individuals with moderate musical training. We also found that musical genres that were less familiar to our participants (e.g., Balinese gamelan) produced strong responses within the music component, as did drum clips with rhythm but little melody, suggesting that these neural populations are broadly responsive to music as a whole. Our findings demonstrate that the signature properties of neural music selectivity do not require musical training to develop, showing that the music-selective neural populations are a fundamental and widespread property of the human brain.NEW & NOTEWORTHY We show that music-selective neural populations are clearly present in people without musical training, demonstrating that they are a fundamental and widespread property of the human brain. Additionally, we show music-selective neural populations respond strongly to music from unfamiliar genres as well as music with rhythm but little pitch information, suggesting that they are broadly responsive to music as a whole.
Collapse
Affiliation(s)
- Dana Boebinger
- Speech and Hearing Bioscience and Technology, Harvard University, Cambridge, Massachusetts
- Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, Massachusetts
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, Massachusetts
| | - Sam V Norman-Haignere
- Laboratoire des Sytèmes Perceptifs, Département d'Études Cognitives, École Normale Supérieure, PSL Research University, CNRS, Paris France
- Zuckerman Institute for Brain Research, Columbia University, New York, New York
| | - Josh H McDermott
- Speech and Hearing Bioscience and Technology, Harvard University, Cambridge, Massachusetts
- Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, Massachusetts
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, Massachusetts
- Center for Brains, Minds, and Machines, Massachusetts Institute of Technology, Cambridge, Massachusetts
| | - Nancy Kanwisher
- Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, Massachusetts
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, Massachusetts
- Center for Brains, Minds, and Machines, Massachusetts Institute of Technology, Cambridge, Massachusetts
| |
Collapse
|
27
|
Ivanova AA, Mineroff Z, Zimmerer V, Kanwisher N, Varley R, Fedorenko E. The Language Network Is Recruited but Not Required for Nonverbal Event Semantics. Neurobiol Lang (Camb) 2021; 2:176-201. [PMID: 37216147 PMCID: PMC10158592 DOI: 10.1162/nol_a_00030] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 09/04/2020] [Accepted: 01/07/2021] [Indexed: 05/24/2023]
Abstract
The ability to combine individual concepts of objects, properties, and actions into complex representations of the world is often associated with language. Yet combinatorial event-level representations can also be constructed from nonverbal input, such as visual scenes. Here, we test whether the language network in the human brain is involved in and necessary for semantic processing of events presented nonverbally. In Experiment 1, we scanned participants with fMRI while they performed a semantic plausibility judgment task versus a difficult perceptual control task on sentences and line drawings that describe/depict simple agent-patient interactions. We found that the language network responded robustly during the semantic task performed on both sentences and pictures (although its response to sentences was stronger). Thus, language regions in healthy adults are engaged during a semantic task performed on pictorial depictions of events. But is this engagement necessary? In Experiment 2, we tested two individuals with global aphasia, who have sustained massive damage to perisylvian language areas and display severe language difficulties, against a group of age-matched control participants. Individuals with aphasia were severely impaired on the task of matching sentences to pictures. However, they performed close to controls in assessing the plausibility of pictorial depictions of agent-patient interactions. Overall, our results indicate that the left frontotemporal language network is recruited but not necessary for semantic processing of nonverbally presented events.
Collapse
Affiliation(s)
- Anna A. Ivanova
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA, USA
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA, USA
| | - Zachary Mineroff
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA, USA
| | - Vitor Zimmerer
- Division of Psychology and Language Sciences, University College London, London, UK
| | - Nancy Kanwisher
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA, USA
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA, USA
| | - Rosemary Varley
- Division of Psychology and Language Sciences, University College London, London, UK
| | - Evelina Fedorenko
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA, USA
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA, USA
| |
Collapse
|
28
|
Dobs K, Kell AJE, Martinez J, Cohen M, Kanwisher N. Using task-optimized neural networks to understand why brains have specialized processing for faces. J Vis 2020. [DOI: 10.1167/jov.20.11.660] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/24/2022] Open
Affiliation(s)
| | | | | | - Michael Cohen
- Massachusetts Institute of Technology
- Amherst College
| | | |
Collapse
|
29
|
Kosakowski HL, Cohen MI, Keil B, Takahashi A, Nichoson I, Alves L, Kanwisher N, Saxe R. Face selectivity in human infant ventral temporal cortex. J Vis 2020. [DOI: 10.1167/jov.20.11.790] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/24/2022] Open
Affiliation(s)
| | | | - Boris Keil
- Mittelhessen University of Applied Science
| | | | | | | | | | | |
Collapse
|
30
|
Pramod RT, Cohen M, Lydic K, Tenenbaum J, Kanwisher N. Evidence that the Brain’s Physics Engine Runs Forward Simulations of What will Happen Next. J Vis 2020. [DOI: 10.1167/jov.20.11.1521] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/24/2022] Open
Affiliation(s)
- RT Pramod
- Massachusetts Institute of Technology
| | - Michael Cohen
- Massachusetts Institute of Technology
- Amherst College, Amherst, MA
| | | | | | | |
Collapse
|
31
|
Jouravlev O, Kell AJE, Mineroff Z, Haskins AJ, Ayyash D, Kanwisher N, Fedorenko E. Reduced Language Lateralization in Autism and the Broader Autism Phenotype as Assessed with Robust Individual-Subjects Analyses. Autism Res 2020; 13:1746-1761. [PMID: 32935455 DOI: 10.1002/aur.2393] [Citation(s) in RCA: 29] [Impact Index Per Article: 7.3] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/11/2020] [Revised: 07/28/2020] [Accepted: 08/25/2020] [Indexed: 12/13/2022]
Abstract
One of the few replicated functional brain differences between individuals with autism spectrum disorders (ASD) and neurotypical (NT) controls is reduced language lateralization. However, most prior reports relied on comparisons of group-level activation maps or functional markers that had not been validated at the individual-subject level, and/or used tasks that do not isolate language processing from other cognitive processes, complicating interpretation. Furthermore, few prior studies have examined functional responses in other brain networks, as needed to determine the spatial selectivity of the effect. Using functional magnetic resonance imaging (fMRI), we compared language lateralization between 28 adult ASD participants and carefully pairwise-matched controls, with the language regions defined individually using a well-validated language "localizer" task. Across two language comprehension paradigms, ASD participants showed less lateralized responses due to stronger right hemisphere activity. Furthermore, this effect did not stem from a ubiquitous reduction in lateralization of function across the brain: ASD participants did not differ from controls in the lateralization of two other large-scale networks-the Theory of Mind network and the Multiple Demand network. Finally, in an exploratory study, we tested whether reduced language lateralization may also be present in NT individuals with high autism-like traits. Indeed, autistic trait load in a large set of NT participants (n = 189) was associated with less lateralized language responses. These results suggest that reduced language lateralization is robustly associated with autism and, to some extent, with autism-like traits in the general population, and this lateralization reduction appears to be restricted to the language system. LAY SUMMARY: How do brains of individuals with autism spectrum disorders (ASD) differ from those of neurotypical (NT) controls? One of the most consistently reported differences is the reduction of lateralization during language processing in individuals with ASD. However, most prior studies have used methods that made this finding difficult to interpret, and perhaps even artifactual. Using robust individual-level markers of lateralization, we found that indeed, ASD individuals show reduced lateralization for language due to stronger right-hemisphere activity. We further show that this reduction is not due to a general reduction of lateralization of function across the brain. Finally, we show that greater autistic trait load is associated with less lateralized language responses in the NT population. These results suggest that reduced language lateralization is robustly associated with autism and, to some extent, with autism-like traits in the general population. Autism Res 2020, 13: 1746-1761. © 2020 International Society for Autism Research and Wiley Periodicals LLC.
Collapse
Affiliation(s)
- Olessia Jouravlev
- Brain and Cognitive Sciences Department, MIT, Cambridge, Massachusetts, USA.,Department of Cognitive Science, Carleton University, Ottawa, Ontario, Canada
| | - Alexander J E Kell
- Brain and Cognitive Sciences Department, MIT, Cambridge, Massachusetts, USA.,Zuckerman Institute, Columbia University, New York, New York, USA
| | - Zachary Mineroff
- Brain and Cognitive Sciences Department, MIT, Cambridge, Massachusetts, USA.,Eberly Center, Carnegie Mellon University, Pittsburgh, Pennsylvania, USA
| | - Amanda J Haskins
- Brain and Cognitive Sciences Department, MIT, Cambridge, Massachusetts, USA.,Department of Psychological and Brain Sciences, Dartmouth College, Hanover, New Hampshire, USA
| | - Dima Ayyash
- Brain and Cognitive Sciences Department, MIT, Cambridge, Massachusetts, USA.,McGovern Institute for Brain Research, MIT, Cambridge, Massachusetts, USA
| | - Nancy Kanwisher
- Brain and Cognitive Sciences Department, MIT, Cambridge, Massachusetts, USA.,McGovern Institute for Brain Research, MIT, Cambridge, Massachusetts, USA
| | - Evelina Fedorenko
- Brain and Cognitive Sciences Department, MIT, Cambridge, Massachusetts, USA.,McGovern Institute for Brain Research, MIT, Cambridge, Massachusetts, USA
| |
Collapse
|
32
|
Ratan Murty NA, Teng S, Beeler D, Mynick A, Oliva A, Kanwisher N. Visual experience is not necessary for the development of face-selectivity in the lateral fusiform gyrus. Proc Natl Acad Sci U S A 2020; 117:23011-23020. [PMID: 32839334 PMCID: PMC7502773 DOI: 10.1073/pnas.2004607117] [Citation(s) in RCA: 30] [Impact Index Per Article: 7.5] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/08/2023] Open
Abstract
The fusiform face area responds selectively to faces and is causally involved in face perception. How does face-selectivity in the fusiform arise in development, and why does it develop so systematically in the same location across individuals? Preferential cortical responses to faces develop early in infancy, yet evidence is conflicting on the central question of whether visual experience with faces is necessary. Here, we revisit this question by scanning congenitally blind individuals with fMRI while they haptically explored 3D-printed faces and other stimuli. We found robust face-selective responses in the lateral fusiform gyrus of individual blind participants during haptic exploration of stimuli, indicating that neither visual experience with faces nor fovea-biased inputs is necessary for face-selectivity to arise in the lateral fusiform gyrus. Our results instead suggest a role for long-range connectivity in specifying the location of face-selectivity in the human brain.
Collapse
Affiliation(s)
- N Apurva Ratan Murty
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA 02139
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA 02139
- The Center for Brains, Minds, and Machines, Massachusetts Institute of Technology, Cambridge, MA 02139
| | - Santani Teng
- The Smith-Kettlewell Eye Research Institute, San Francisco, CA 94115
- Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of Technology, Cambridge, MA 02139
| | - David Beeler
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA 02139
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA 02139
| | - Anna Mynick
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA 02139
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA 02139
| | - Aude Oliva
- Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of Technology, Cambridge, MA 02139
| | - Nancy Kanwisher
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA 02139;
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA 02139
- The Center for Brains, Minds, and Machines, Massachusetts Institute of Technology, Cambridge, MA 02139
| |
Collapse
|
33
|
Isik L, Mynick A, Pantazis D, Kanwisher N. The speed of human social interaction perception. Neuroimage 2020; 215:116844. [DOI: 10.1016/j.neuroimage.2020.116844] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [What about the content of this article? (0)] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/04/2019] [Revised: 02/27/2020] [Accepted: 04/08/2020] [Indexed: 11/17/2022] Open
|
34
|
Abstract
An intuitive understanding of physical objects and events is critical for successfully interacting with the world. Does the brain achieve this understanding by running simulations in a mental physics engine, which represents variables such as force and mass, or by analyzing patterns of motion without encoding underlying physical quantities? To investigate, we scanned participants with fMRI while they viewed videos of objects interacting in scenarios indicating their mass. Decoding analyses in brain regions previously implicated in intuitive physical inference revealed mass representations that generalized across variations in scenario, material, friction, and motion energy. These invariant representations were found during tasks without action planning, and tasks focusing on an orthogonal dimension (object color). Our results support an account of physical reasoning where abstract physical variables serve as inputs to a forward model of dynamics, akin to a physics engine, in parietal and frontal cortex.
Collapse
Affiliation(s)
- Sarah Schwettmann
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, United States.,Center for Brains, Minds, and Machines, Massachusetts Institute of Technology, Cambridge, United States.,McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, United States
| | - Joshua B Tenenbaum
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, United States.,Center for Brains, Minds, and Machines, Massachusetts Institute of Technology, Cambridge, United States.,McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, United States.,Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of Technology, Cambridge, United States
| | - Nancy Kanwisher
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, United States.,Center for Brains, Minds, and Machines, Massachusetts Institute of Technology, Cambridge, United States.,McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, United States
| |
Collapse
|
35
|
Ratan Murty NA, Teng S, Beeler D, Mynick A, Oliva A, Kanwisher N. Strong face selectivity in the fusiform can develop in the absence of visual experience. J Vis 2019. [DOI: 10.1167/19.10.54a] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/24/2022] Open
Affiliation(s)
- N Apurva Ratan Murty
- McGovern Institute for Brain Research, Massachusetts Institute of Technology
- Centre for Brains, Minds and Machines, Massachusetts Institute of Technology
| | - Santani Teng
- Computer Science and Artificial Intelligence Laboratory (CSAIL), MIT
- Smith-Ket-tlewell Eye Research Institute
| | - David Beeler
- McGovern Institute for Brain Research, Massachusetts Institute of Technology
| | - Anna Mynick
- McGovern Institute for Brain Research, Massachusetts Institute of Technology
| | - Aude Oliva
- Computer Science and Artificial Intelligence Laboratory (CSAIL), MIT
| | - Nancy Kanwisher
- McGovern Institute for Brain Research, Massachusetts Institute of Technology
- Centre for Brains, Minds and Machines, Massachusetts Institute of Technology
| |
Collapse
|
36
|
Abstract
Despite extensive investigation, the causes and nature of developmental prosopagnosia (DP)-a severe face identification impairment in the absence of acquired brain injury-remain poorly understood. Drawing on previous work showing that individuals identified as being neurotypical (NT) show robust individual differences in where they fixate on faces, and recognize faces best when the faces are presented at this location, we defined and tested four novel hypotheses for how atypical face-looking behavior and/or retinotopic face encoding could impair face recognition in DP: (a) fixating regions of poor information, (b) inconsistent saccadic targeting, (c) weak retinotopic tuning, and (d) fixating locations not matched to the individual's own face tuning. We found no support for the first three hypotheses, with NTs and DPs consistently fixating similar locations and showing similar retinotopic tuning of their face perception performance. However, in testing the fourth hypothesis, we found preliminary evidence for two distinct phenotypes of DP: (a) Subjects characterized by impaired face memory, typical face perception, and a preference to look high on the face, and (b) Subjects characterized by profound impairments to both face memory and perception and a preference to look very low on the face. Further, while all NTs and upper-looking DPs performed best when faces were presented near their preferred fixation location, this was not true for lower-looking DPs. These results suggest that face recognition deficits in a substantial proportion of people with DP may arise not from aberrant face gaze or compromised retinotopic tuning, but from the suboptimal matching of gaze to tuning.
Collapse
Affiliation(s)
- Matthew F Peterson
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA, USA
| | - Ian Zaun
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA, USA
| | - Harris Hoke
- Center for Brain Science, Harvard University, Cambridge, MA, USA
| | - Guo Jiahui
- Department of Psychological and Brain Sciences, Dartmouth College, Hanover, NH, USA
| | - Brad Duchaine
- Department of Psychological and Brain Sciences, Dartmouth College, Hanover, NH, USA
| | - Nancy Kanwisher
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA, USA
| |
Collapse
|
37
|
Norman-Haignere SV, Kanwisher N, McDermott JH, Conway BR. Divergence in the functional organization of human and macaque auditory cortex revealed by fMRI responses to harmonic tones. Nat Neurosci 2019; 22:1057-1060. [PMID: 31182868 PMCID: PMC6592717 DOI: 10.1038/s41593-019-0410-7] [Citation(s) in RCA: 28] [Impact Index Per Article: 5.6] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/10/2018] [Accepted: 04/19/2019] [Indexed: 12/02/2022]
Abstract
We report a difference between humans and macaque monkeys in the functional organization of cortical regions implicated in pitch perception: humans but not macaques showed regions with a strong preference for harmonic sounds compared to noise, measured with both synthetic tones and macaque vocalizations. In contrast, frequency-selective tonotopic maps were similar between the two species. This species difference may be driven by the unique demands of speech and music perception in humans.
Collapse
Affiliation(s)
- Sam V Norman-Haignere
- Zuckerman Institute for Mind, Brain and Behavior, Columbia University, New York, NY, USA. .,Department of Brain and Cognitive Sciences, MIT, Cambridge, MA, USA. .,HHMI Postdoctoral Fellow of the Life Sciences Research Institute, Chevy Chase, MD, USA. .,Laboratoire des Systèmes Perceptifs, Département d'Études Cognitives, École Normale Supérieure, PSL University, CNRS, Paris, France.
| | - Nancy Kanwisher
- Department of Brain and Cognitive Sciences, MIT, Cambridge, MA, USA.,McGovern Institute for Brain Research, Cambridge, MA, USA.,Center for Minds, Brains and Machines, Cambridge, MA, USA
| | - Josh H McDermott
- Department of Brain and Cognitive Sciences, MIT, Cambridge, MA, USA.,McGovern Institute for Brain Research, Cambridge, MA, USA.,Center for Minds, Brains and Machines, Cambridge, MA, USA.,Program in Speech and Hearing Biosciences and Technology, Harvard University, Cambridge, MA, USA
| | - Bevil R Conway
- Laboratory of Sensorimotor Research, NEI, NIH, Bethesda, MD, USA. .,National Institute of Mental Health, NIH, Bethesda, MD, USA. .,National Institute of Neurological Disease and Stroke, NIH, Bethesda, MD, USA.
| |
Collapse
|
38
|
Yildirim I, Wu J, Kanwisher N, Tenenbaum J. An integrative computational architecture for object-driven cortex. Curr Opin Neurobiol 2019; 55:73-81. [PMID: 30825704 PMCID: PMC6548583 DOI: 10.1016/j.conb.2019.01.010] [Citation(s) in RCA: 11] [Impact Index Per Article: 2.2] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/23/2018] [Revised: 12/24/2018] [Accepted: 01/13/2019] [Indexed: 01/09/2023]
Abstract
Computational architecture for object-driven cortex Objects in motion activate multiple cortical regions in every lobe of the human brain. Do these regions represent a collection of independent systems, or is there an overarching functional architecture spanning all of object-driven cortex? Inspired by recent work in artificial intelligence (AI), machine learning, and cognitive science, we consider the hypothesis that these regions can be understood as a coherent network implementing an integrative computational system that unifies the functions needed to perceive, predict, reason about, and plan with physical objects-as in the paradigmatic case of using or making tools. Our proposal draws on a modeling framework that combines multiple AI methods, including causal generative models, hybrid symbolic-continuous planning algorithms, and neural recognition networks, with object-centric, physics-based representations. We review evidence relating specific components of our proposal to the specific regions that comprise object-driven cortex, and lay out future research directions with the goal of building a complete functional and mechanistic account of this system.
Collapse
Affiliation(s)
- Ilker Yildirim
- Center for Brains, Minds, and Machines, MIT, Cambridge, MA 02138, United States; Department of Brain & Cognitive Science, MIT, Cambridge, MA 02138, United States.
| | - Jiajun Wu
- Center for Brains, Minds, and Machines, MIT, Cambridge, MA 02138, United States; Computer Science and Artificial Intelligence Laboratory, MIT, Cambridge, MA 02138, United States
| | - Nancy Kanwisher
- Center for Brains, Minds, and Machines, MIT, Cambridge, MA 02138, United States; McGovern Institute for Brain Research, MIT, Cambridge, MA 02138, United States; Department of Brain & Cognitive Science, MIT, Cambridge, MA 02138, United States
| | - Joshua Tenenbaum
- Center for Brains, Minds, and Machines, MIT, Cambridge, MA 02138, United States; McGovern Institute for Brain Research, MIT, Cambridge, MA 02138, United States; Computer Science and Artificial Intelligence Laboratory, MIT, Cambridge, MA 02138, United States; Department of Brain & Cognitive Science, MIT, Cambridge, MA 02138, United States
| |
Collapse
|
39
|
Abstract
Within a fraction of a second of viewing a face, we have already determined its gender, age and identity. A full understanding of this remarkable feat will require a characterization of the computational steps it entails, along with the representations extracted at each. Here, we used magnetoencephalography (MEG) to measure the time course of neural responses to faces, thereby addressing two fundamental questions about how face processing unfolds over time. First, using representational similarity analysis, we found that facial gender and age information emerged before identity information, suggesting a coarse-to-fine processing of face dimensions. Second, identity and gender representations of familiar faces were enhanced very early on, suggesting that the behavioral benefit for familiar faces results from tuning of early feed-forward processing mechanisms. These findings start to reveal the time course of face processing in humans, and provide powerful new constraints on computational theories of face perception.
Collapse
Affiliation(s)
- Katharina Dobs
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA, 02139, USA.
- McGovern Institute of Brain Research, Massachusetts Institute of Technology, Cambridge, MA, 02139, USA.
- The Center for Brains, Minds, and Machines, Massachusetts Institute of Technology, Cambridge, MA, 02139, USA.
| | - Leyla Isik
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA, 02139, USA
- McGovern Institute of Brain Research, Massachusetts Institute of Technology, Cambridge, MA, 02139, USA
- The Center for Brains, Minds, and Machines, Massachusetts Institute of Technology, Cambridge, MA, 02139, USA
| | - Dimitrios Pantazis
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA, 02139, USA
- McGovern Institute of Brain Research, Massachusetts Institute of Technology, Cambridge, MA, 02139, USA
| | - Nancy Kanwisher
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA, 02139, USA
- McGovern Institute of Brain Research, Massachusetts Institute of Technology, Cambridge, MA, 02139, USA
- The Center for Brains, Minds, and Machines, Massachusetts Institute of Technology, Cambridge, MA, 02139, USA
| |
Collapse
|
40
|
Dobs K, Isik L, Pantazis D, Kanwisher N. Rapid decoding of face identity, familiarity, gender and age. J Vis 2018. [DOI: 10.1167/18.10.1081] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/24/2022] Open
|
41
|
Isik L, Mynick A, Koldewyn K, Kanwisher N. Rapid detection of social interactions in the human brain. J Vis 2018. [DOI: 10.1167/18.10.1340] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/24/2022] Open
Affiliation(s)
- Leyla Isik
- Department of Brain and Cognitive Sciences, McGovern Institute for Brain Research, MIT
| | - Anna Mynick
- Department of Brain and Cognitive Sciences, McGovern Institute for Brain Research, MIT
| | | | - Nancy Kanwisher
- Department of Brain and Cognitive Sciences, McGovern Institute for Brain Research, MIT
| |
Collapse
|
42
|
Spiegel A, Lee J, Haskins AJ, Kanwisher N, Robertson C. Direct Neural Read-Out of Binocular Rivalry Dynamics in Autism using EEG. J Vis 2018. [DOI: 10.1167/18.10.37] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/24/2022] Open
Affiliation(s)
- Alina Spiegel
- McGovern Institute for Brain Research, MITSchool of Medicine, Johns Hopkins University
| | - Jackson Lee
- McGovern Institute for Brain Research, MITDuke University
| | - AJ Haskins
- McGovern Institute for Brain Research, MIT
| | | | - Caroline Robertson
- McGovern Institute for Brain Research, MITHarvard Society of Fellows, Harvard University
| |
Collapse
|
43
|
Peterson M, Haskins A, Zaun I, Kanwisher N. Mismatch of face fixation preference and retinotopic tuning of face perception in autism spectrum condition. J Vis 2018. [DOI: 10.1167/18.10.711] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/24/2022] Open
Affiliation(s)
- Matthew Peterson
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology
| | - Amanda Haskins
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology
| | - Ian Zaun
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology
| | - Nancy Kanwisher
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology
| |
Collapse
|
44
|
Mentch J, Spiegel A, Ricciardi C, Kanwisher N, Robertson C. Causal Push-and-Pull Modulation of Binocular Rivalry Dynamics using GABAergic Drugs. J Vis 2018. [DOI: 10.1167/18.10.956] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/24/2022] Open
Affiliation(s)
- Jeff Mentch
- McGovern Institute for Brain Research, MIT, Cambridge, MA
| | - Alina Spiegel
- McGovern Institute for Brain Research, MIT, Cambridge, MAJohns Hopkins University, Baltimore, MD
| | | | | | - Caroline Robertson
- McGovern Institute for Brain Research, MIT, Cambridge, MAHarvard Society of Fellows, Harvard, Cambridge, MA
| |
Collapse
|
45
|
Schwettmann S, Fischer J, Tenenbaum J, Kanwisher N. Neural representation of the intuitive physical dimension of mass. J Vis 2018. [DOI: 10.1167/18.10.731] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/24/2022] Open
Affiliation(s)
- Sarah Schwettmann
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology
| | - Jason Fischer
- Department of Psychological & Brain Sciences, Johns Hopkins
| | - Joshua Tenenbaum
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology
| | - Nancy Kanwisher
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology
| |
Collapse
|
46
|
Cohen MA, Dennett DC, Kanwisher N. What is the Bandwidth of Perceptual Experience? Trends Cogn Sci 2018; 20:324-335. [PMID: 27105668 DOI: 10.1016/j.tics.2016.03.006] [Citation(s) in RCA: 155] [Impact Index Per Article: 25.8] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/02/2016] [Revised: 03/06/2016] [Accepted: 03/09/2016] [Indexed: 12/01/2022]
Abstract
Although our subjective impression is of a richly detailed visual world, numerous empirical results suggest that the amount of visual information observers can perceive and remember at any given moment is limited. How can our subjective impressions be reconciled with these objective observations? Here, we answer this question by arguing that, although we see more than the handful of objects, claimed by prominent models of visual attention and working memory, we still see far less than we think we do. Taken together, we argue that these considerations resolve the apparent conflict between our subjective impressions and empirical data on visual capacity, while also illuminating the nature of the representations underlying perceptual experience.
Collapse
Affiliation(s)
- Michael A Cohen
- Department of Brain and Cognitive Sciences, McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA, USA.
| | - Daniel C Dennett
- Center for Cognitive Studies, Department of Philosophy, Tufts University, Medford, MA, USA
| | - Nancy Kanwisher
- Department of Brain and Cognitive Sciences, McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA, USA
| |
Collapse
|
47
|
Pereira F, Lou B, Pritchett B, Ritter S, Gershman SJ, Kanwisher N, Botvinick M, Fedorenko E. Toward a universal decoder of linguistic meaning from brain activation. Nat Commun 2018; 9:963. [PMID: 29511192 PMCID: PMC5840373 DOI: 10.1038/s41467-018-03068-4] [Citation(s) in RCA: 97] [Impact Index Per Article: 16.2] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/10/2017] [Accepted: 01/13/2018] [Indexed: 11/09/2022] Open
Abstract
Prior work decoding linguistic meaning from imaging data has been largely limited to concrete nouns, using similar stimuli for training and testing, from a relatively small number of semantic categories. Here we present a new approach for building a brain decoding system in which words and sentences are represented as vectors in a semantic space constructed from massive text corpora. By efficiently sampling this space to select training stimuli shown to subjects, we maximize the ability to generalize to new meanings from limited imaging data. To validate this approach, we train the system on imaging data of individual concepts, and show it can decode semantic vector representations from imaging data of sentences about a wide variety of both concrete and abstract topics from two separate datasets. These decoded representations are sufficiently detailed to distinguish even semantically similar sentences, and to capture the similarity structure of meaning relationships between sentences.
Collapse
Affiliation(s)
- Francisco Pereira
- Medical Imaging Technologies, Siemens Healthineers, Princeton, NJ 08540, USA.
| | - Bin Lou
- Medical Imaging Technologies, Siemens Healthineers, Princeton, NJ 08540, USA
| | - Brianna Pritchett
- Department of Brain and Cognitive Sciences, MIT, Cambridge, MA, 02139, USA
| | | | - Samuel J Gershman
- Department of Psychology and Center for Brain Science, Harvard University, Cambridge, MA, 02138, USA
| | - Nancy Kanwisher
- Department of Brain and Cognitive Sciences, MIT, Cambridge, MA, 02139, USA
- McGovern Institute for Brain Research, MIT, Cambridge, MA, 02139, USA
| | - Matthew Botvinick
- DeepMind, London, N1C 4AG, UK
- Gatsby Computational Neuroscience Unit, University College London, London, WC1E 6BT, UK
| | - Evelina Fedorenko
- McGovern Institute for Brain Research, MIT, Cambridge, MA, 02139, USA.
- Department of Psychiatry, Harvard Medical School, Boston, MA, 02115, USA.
- Department of Psychiatry, Massachusetts General Hospital, Boston, MA, 02114, USA.
| |
Collapse
|
48
|
Kamps FS, Julian JB, Battaglia P, Landau B, Kanwisher N, Dilks DD. Dissociating intuitive physics from intuitive psychology: Evidence from Williams syndrome. Cognition 2017; 168:146-153. [PMID: 28683351 PMCID: PMC5572752 DOI: 10.1016/j.cognition.2017.06.027] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/04/2016] [Revised: 06/16/2017] [Accepted: 06/27/2017] [Indexed: 12/01/2022]
Abstract
Prior work suggests that our understanding of how things work ("intuitive physics") and how people work ("intuitive psychology") are distinct domains of human cognition. Here we directly test the dissociability of these two domains by investigating knowledge of intuitive physics and intuitive psychology in adults with Williams syndrome (WS) - a genetic developmental disorder characterized by severely impaired spatial cognition, but relatively spared social cognition. WS adults and mental-age matched (MA) controls completed an intuitive physics task and an intuitive psychology task. If intuitive physics is a distinct domain (from intuitive psychology), then we should observe differential impairment on the physics task for individuals with WS compared to MA controls. Indeed, adults with WS performed significantly worse on the intuitive physics than the intuitive psychology task, relative to controls. These results support the hypothesis that knowledge of the physical world can be disrupted independently from knowledge of the social world.
Collapse
Affiliation(s)
- Frederik S Kamps
- Department of Psychology, Emory University, Atlanta, GA 30322, United States
| | - Joshua B Julian
- Department of Psychology, University of Pennsylvania, Philadelphia, PA 19104, United States
| | - Peter Battaglia
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA 02139, United States
| | - Barbara Landau
- Department of Cognitive Science, Johns Hopkins University, Baltimore, MD 21218, United States
| | - Nancy Kanwisher
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA 02139, United States
| | - Daniel D Dilks
- Department of Psychology, Emory University, Atlanta, GA 30322, United States.
| |
Collapse
|
49
|
Peterson M, Hoke H, Zaun I, Duchaine B, Kanwisher N. Retinotopic Specificity of Face Encoding in Neurotypicals and Developmental Prosopagnosics. J Vis 2017. [DOI: 10.1167/17.10.622] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/24/2022] Open
Affiliation(s)
| | | | - Ian Zaun
- Massachusetts Institute of Technology
| | | | | |
Collapse
|
50
|
Isik L, Lotter W, Crone N, Cox D, Kanwisher N, Andreson W, Kreiman G. Task dependent modulation before, during and after visually evoked responses in human intracranial recordings. J Vis 2017. [DOI: 10.1167/17.10.983] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/24/2022] Open
Affiliation(s)
- Leyla Isik
- Boston Children's Hospital, Harvard Medical SchoolMassachusetts Institute of Technology
| | - William Lotter
- Boston Children's Hospital, Harvard Medical SchoolHarvard University
| | | | - David Cox
- Massachusetts Institute of Technology
| | | | | | | |
Collapse
|