1
|
Golan T, Taylor J, Schütt H, Peters B, Sommers RP, Seeliger K, Doerig A, Linton P, Konkle T, van Gerven M, Kording K, Richards B, Kietzmann TC, Lindsay GW, Kriegeskorte N. Deep neural networks are not a single hypothesis but a language for expressing computational hypotheses. Behav Brain Sci 2023; 46:e392. [PMID: 38054329 DOI: 10.1017/s0140525x23001553] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/07/2023]
Abstract
An ideal vision model accounts for behavior and neurophysiology in both naturalistic conditions and designed lab experiments. Unlike psychological theories, artificial neural networks (ANNs) actually perform visual tasks and generate testable predictions for arbitrary inputs. These advantages enable ANNs to engage the entire spectrum of the evidence. Failures of particular models drive progress in a vibrant ANN research program of human vision.
Collapse
Affiliation(s)
- Tal Golan
- Department of Cognitive and Brain Sciences, Ben-Gurion University of the Negev, Be'er Sheva, Israel
| | - JohnMark Taylor
- Zuckerman Mind Brain Behavior Institute, Columbia University, New York, NY, USA ://linton.vision/
| | - Heiko Schütt
- Zuckerman Mind Brain Behavior Institute, Columbia University, New York, NY, USA ://linton.vision/
- Center for Neural Science, New York University, New York, NY, USA
| | - Benjamin Peters
- School of Psychology & Neuroscience, University of Glasgow, Glasgow, UK
| | - Rowan P Sommers
- Department of Neurobiology of Language, Max Planck Institute for Psycholinguistics, Nijmegen, The Netherlands
| | - Katja Seeliger
- Max Planck Institute for Human Cognitive and Brain Sciences, Leipzig, Germany
| | - Adrien Doerig
- Institute of Cognitive Science, University of Osnabrück, Osnabrück, Germany
| | - Paul Linton
- Zuckerman Mind Brain Behavior Institute, Columbia University, New York, NY, USA ://linton.vision/
- Presidential Scholars in Society and Neuroscience, Center for Science and Society, Columbia University, New York, NY, USA
- Italian Academy for Advanced Studies in America, Columbia University, New York, NY, USA
| | - Talia Konkle
- Department of Psychology and Center for Brain Sciences, Harvard University, Cambridge, MA, USA ://konklab.fas.harvard.edu/
| | - Marcel van Gerven
- Donders Institute for Brain, Cognition and Behaviour, Nijmegen, The Netherlandsartcogsys.com
| | - Konrad Kording
- Departments of Bioengineering and Neuroscience, University of Pennsylvania, Philadelphia, PA, USA
- Learning in Machines and Brains Program, CIFAR, Toronto, ON, Canada
| | - Blake Richards
- Learning in Machines and Brains Program, CIFAR, Toronto, ON, Canada
- Mila, Montreal, QC, Canada
- School of Computer Science, McGill University, Montreal, QC, Canada
- Department of Neurology & Neurosurgery, McGill University, Montreal, QC, Canada
- Montreal Neurological Institute, Montreal, QC, Canada
| | - Tim C Kietzmann
- Institute of Cognitive Science, University of Osnabrück, Osnabrück, Germany
| | - Grace W Lindsay
- Department of Psychology and Center for Data Science, New York University, New York, NY, USA
| | - Nikolaus Kriegeskorte
- Zuckerman Mind Brain Behavior Institute, Columbia University, New York, NY, USA ://linton.vision/
- Departments of Psychology, Neuroscience, and Electrical Engineering, Columbia University, New York, NY, USA
| |
Collapse
|
2
|
Karapetian A, Boyanova A, Pandaram M, Obermayer K, Kietzmann TC, Cichy RM. Empirically Identifying and Computationally Modeling the Brain-Behavior Relationship for Human Scene Categorization. J Cogn Neurosci 2023; 35:1879-1897. [PMID: 37590093 PMCID: PMC10586810 DOI: 10.1162/jocn_a_02043] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 08/19/2023]
Abstract
Humans effortlessly make quick and accurate perceptual decisions about the nature of their immediate visual environment, such as the category of the scene they face. Previous research has revealed a rich set of cortical representations potentially underlying this feat. However, it remains unknown which of these representations are suitably formatted for decision-making. Here, we approached this question empirically and computationally, using neuroimaging and computational modeling. For the empirical part, we collected EEG data and RTs from human participants during a scene categorization task (natural vs. man-made). We then related EEG data to behavior to behavior using a multivariate extension of signal detection theory. We observed a correlation between neural data and behavior specifically between ∼100 msec and ∼200 msec after stimulus onset, suggesting that the neural scene representations in this time period are suitably formatted for decision-making. For the computational part, we evaluated a recurrent convolutional neural network (RCNN) as a model of brain and behavior. Unifying our previous observations in an image-computable model, the RCNN predicted well the neural representations, the behavioral scene categorization data, as well as the relationship between them. Our results identify and computationally characterize the neural and behavioral correlates of scene categorization in humans.
Collapse
Affiliation(s)
- Agnessa Karapetian
- Freie Universität Berlin, Germany
- Charité - Universitätsmedizin Berlin, Einstein Center for Neurosciences Berlin, Germany
- Bernstein Center for Computational Neuroscience Berlin, Germany
| | | | | | - Klaus Obermayer
- Charité - Universitätsmedizin Berlin, Einstein Center for Neurosciences Berlin, Germany
- Bernstein Center for Computational Neuroscience Berlin, Germany
- Technische Universität Berlin, Germany
- Humboldt-Universität zu Berlin, Germany
| | | | - Radoslaw M Cichy
- Freie Universität Berlin, Germany
- Charité - Universitätsmedizin Berlin, Einstein Center for Neurosciences Berlin, Germany
- Bernstein Center for Computational Neuroscience Berlin, Germany
- Humboldt-Universität zu Berlin, Germany
| |
Collapse
|
3
|
Doerig A, Sommers RP, Seeliger K, Richards B, Ismael J, Lindsay GW, Kording KP, Konkle T, van Gerven MAJ, Kriegeskorte N, Kietzmann TC. The neuroconnectionist research programme. Nat Rev Neurosci 2023:10.1038/s41583-023-00705-w. [PMID: 37253949 DOI: 10.1038/s41583-023-00705-w] [Citation(s) in RCA: 14] [Impact Index Per Article: 14.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 04/21/2023] [Indexed: 06/01/2023]
Abstract
Artificial neural networks (ANNs) inspired by biology are beginning to be widely used to model behavioural and neural data, an approach we call 'neuroconnectionism'. ANNs have been not only lauded as the current best models of information processing in the brain but also criticized for failing to account for basic cognitive functions. In this Perspective article, we propose that arguing about the successes and failures of a restricted set of current ANNs is the wrong approach to assess the promise of neuroconnectionism for brain science. Instead, we take inspiration from the philosophy of science, and in particular from Lakatos, who showed that the core of a scientific research programme is often not directly falsifiable but should be assessed by its capacity to generate novel insights. Following this view, we present neuroconnectionism as a general research programme centred around ANNs as a computational language for expressing falsifiable theories about brain computation. We describe the core of the programme, the underlying computational framework and its tools for testing specific neuroscientific hypotheses and deriving novel understanding. Taking a longitudinal view, we review past and present neuroconnectionist projects and their responses to challenges and argue that the research programme is highly progressive, generating new and otherwise unreachable insights into the workings of the brain.
Collapse
Affiliation(s)
- Adrien Doerig
- Institute of Cognitive Science, University of Osnabrück, Osnabrück, Germany.
- Donders Institute for Brain, Cognition and Behaviour, Nijmegen, The Netherlands.
| | - Rowan P Sommers
- Department of Neurobiology of Language, Max Planck Institute for Psycholinguistics, Nijmegen, The Netherlands
| | - Katja Seeliger
- Max Planck Institute for Human Cognitive and Brain Sciences, Leipzig, Germany
| | - Blake Richards
- Department of Neurology and Neurosurgery, McGill University, Montréal, QC, Canada
- School of Computer Science, McGill University, Montréal, QC, Canada
- Mila, Montréal, QC, Canada
- Montréal Neurological Institute, Montréal, QC, Canada
- Learning in Machines and Brains Program, CIFAR, Toronto, ON, Canada
| | | | | | - Konrad P Kording
- Learning in Machines and Brains Program, CIFAR, Toronto, ON, Canada
- Bioengineering, Neuroscience, University of Pennsylvania, Pennsylvania, PA, USA
| | | | | | | | - Tim C Kietzmann
- Institute of Cognitive Science, University of Osnabrück, Osnabrück, Germany
| |
Collapse
|
4
|
Jozwik KM, Kietzmann TC, Cichy RM, Kriegeskorte N, Mur M. Deep Neural Networks and Visuo-Semantic Models Explain Complementary Components of Human Ventral-Stream Representational Dynamics. J Neurosci 2023; 43:1731-1741. [PMID: 36759190 PMCID: PMC10010451 DOI: 10.1523/jneurosci.1424-22.2022] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/20/2022] [Revised: 11/08/2022] [Accepted: 12/20/2022] [Indexed: 02/11/2023] Open
Abstract
Deep neural networks (DNNs) are promising models of the cortical computations supporting human object recognition. However, despite their ability to explain a significant portion of variance in neural data, the agreement between models and brain representational dynamics is far from perfect. We address this issue by asking which representational features are currently unaccounted for in neural time series data, estimated for multiple areas of the ventral stream via source-reconstructed magnetoencephalography data acquired in human participants (nine females, six males) during object viewing. We focus on the ability of visuo-semantic models, consisting of human-generated labels of object features and categories, to explain variance beyond the explanatory power of DNNs alone. We report a gradual reversal in the relative importance of DNN versus visuo-semantic features as ventral-stream object representations unfold over space and time. Although lower-level visual areas are better explained by DNN features starting early in time (at 66 ms after stimulus onset), higher-level cortical dynamics are best accounted for by visuo-semantic features starting later in time (at 146 ms after stimulus onset). Among the visuo-semantic features, object parts and basic categories drive the advantage over DNNs. These results show that a significant component of the variance unexplained by DNNs in higher-level cortical dynamics is structured and can be explained by readily nameable aspects of the objects. We conclude that current DNNs fail to fully capture dynamic representations in higher-level human visual cortex and suggest a path toward more accurate models of ventral-stream computations.SIGNIFICANCE STATEMENT When we view objects such as faces and cars in our visual environment, their neural representations dynamically unfold over time at a millisecond scale. These dynamics reflect the cortical computations that support fast and robust object recognition. DNNs have emerged as a promising framework for modeling these computations but cannot yet fully account for the neural dynamics. Using magnetoencephalography data acquired in human observers during object viewing, we show that readily nameable aspects of objects, such as 'eye', 'wheel', and 'face', can account for variance in the neural dynamics over and above DNNs. These findings suggest that DNNs and humans may in part rely on different object features for visual recognition and provide guidelines for model improvement.
Collapse
Affiliation(s)
- Kamila M Jozwik
- Department of Psychology, University of Cambridge, Cambridge CB2 3EB, United Kingdom
| | - Tim C Kietzmann
- Institute of Cognitive Science, University of Osnabrück, 49069 Osnabrück, Germany
| | - Radoslaw M Cichy
- Department of Education and Psychology, Freie Universität Berlin, 14195 Berlin, Germany
| | - Nikolaus Kriegeskorte
- Zuckerman Mind Brain Behavior Institute, Columbia University, New York, New York 10027
| | - Marieke Mur
- Department of Psychology, Western University, London, Ontario N6A 3K7, Canada
- Department of Computer Science, Western University, London, Ontario N6A 3K7, Canada
| |
Collapse
|
5
|
Ólafsdóttir IM, Albertsdóttir SL, Ásgeirsdóttir UA, Kietzmann TC, Sigurdardottir HM. Visual and semantic factors in object recognition. J Vis 2022. [DOI: 10.1167/jov.22.14.3928] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/23/2022] Open
|
6
|
Gert AL, Ehinger BV, Timm S, Kietzmann TC, König P. WildLab: A naturalistic free viewing experiment reveals previously unknown electroencephalography signatures of face processing. Eur J Neurosci 2022; 56:6022-6038. [PMID: 36113866 DOI: 10.1111/ejn.15824] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/09/2022] [Revised: 08/26/2022] [Accepted: 08/30/2022] [Indexed: 12/29/2022]
Abstract
Neural mechanisms of face perception are predominantly studied in well-controlled experimental settings that involve random stimulus sequences and fixed eye positions. Although powerful, the employed paradigms are far from what constitutes natural vision. Here, we demonstrate the feasibility of ecologically more valid experimental paradigms using natural viewing behaviour, by combining a free viewing paradigm on natural scenes, free of photographer bias, with advanced data processing techniques that correct for overlap effects and co-varying non-linear dependencies of multiple eye movement parameters. We validate this approach by replicating classic N170 effects in neural responses, triggered by fixation onsets (fixation event-related potentials [fERPs]). Importantly, besides finding a strong correlation between both experiments, our more natural stimulus paradigm yielded smaller variability between subjects than the classic setup. Moving beyond classic temporal and spatial effect locations, our experiment furthermore revealed previously unknown signatures of face processing: This includes category-specific modulation of the event-related potential (ERP)'s amplitude even before fixation onset, as well as adaptation effects across subsequent fixations depending on their history.
Collapse
Affiliation(s)
- Anna L Gert
- Institute of Cognitive Science, Osnabrück University, Osnabrück, Germany
| | - Benedikt V Ehinger
- Donders Institute for Brain, Cognition and Behaviour, Radboud University, Nijmegen, The Netherlands.,Stuttgart Center for Simulation Science, University of Stuttgart, Stuttgart, Germany
| | - Silja Timm
- Institute of Cognitive Science, Osnabrück University, Osnabrück, Germany
| | - Tim C Kietzmann
- Donders Institute for Brain, Cognition and Behaviour, Radboud University, Nijmegen, The Netherlands.,MRC Cognition and Brain Sciences Unit, Cambridge University, Cambridge, UK
| | - Peter König
- Institute of Cognitive Science, Osnabrück University, Osnabrück, Germany.,Department of Neurophysiology and Pathophysiology, University Medical Center Hamburg-Eppendorf, Hamburg, Germany
| |
Collapse
|
7
|
Singer JJD, Seeliger K, Kietzmann TC, Hebart MN. From photos to sketches - how humans and deep neural networks process objects across different levels of visual abstraction. J Vis 2022; 22:4. [PMID: 35129578 PMCID: PMC8822363 DOI: 10.1167/jov.22.2.4] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/24/2022] Open
Abstract
Line drawings convey meaning with just a few strokes. Despite strong simplifications, humans can recognize objects depicted in such abstracted images without effort. To what degree do deep convolutional neural networks (CNNs) mirror this human ability to generalize to abstracted object images? While CNNs trained on natural images have been shown to exhibit poor classification performance on drawings, other work has demonstrated highly similar latent representations in the networks for abstracted and natural images. Here, we address these seemingly conflicting findings by analyzing the activation patterns of a CNN trained on natural images across a set of photographs, drawings, and sketches of the same objects and comparing them to human behavior. We find a highly similar representational structure across levels of visual abstraction in early and intermediate layers of the network. This similarity, however, does not translate to later stages in the network, resulting in low classification performance for drawings and sketches. We identified that texture bias in CNNs contributes to the dissimilar representational structure in late layers and the poor performance on drawings. Finally, by fine-tuning late network layers with object drawings, we show that performance can be largely restored, demonstrating the general utility of features learned on natural images in early and intermediate layers for the recognition of drawings. In conclusion, generalization to abstracted images, such as drawings, seems to be an emergent property of CNNs trained on natural images, which is, however, suppressed by domain-related biases that arise during later processing stages in the network.
Collapse
Affiliation(s)
- Johannes J D Singer
- Vision and Computational Cognition Group, Max Planck Institute for Human Cognitive and Brain Sciences, Leipzig, Germany.,Department of Psychology, Ludwig Maximilian University, Munich, Germany.,
| | - Katja Seeliger
- Vision and Computational Cognition Group, Max Planck Institute for Human Cognitive and Brain Sciences, Leipzig, Germany.,
| | - Tim C Kietzmann
- Donders Institute for Brain, Cognition and Behavior, Nijmegen, The Netherlands.,
| | - Martin N Hebart
- Vision and Computational Cognition Group, Max Planck Institute for Human Cognitive and Brain Sciences, Leipzig, Germany.,
| |
Collapse
|
8
|
Storrs KR, Kietzmann TC, Walther A, Mehrer J, Kriegeskorte N. Diverse Deep Neural Networks All Predict Human Inferior Temporal Cortex Well, After Training and Fitting. J Cogn Neurosci 2021; 33:2044-2064. [PMID: 34272948 DOI: 10.1101/2020.05.07.082743] [Citation(s) in RCA: 10] [Impact Index Per Article: 3.3] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 05/28/2023]
Abstract
Deep neural networks (DNNs) trained on object recognition provide the best current models of high-level visual cortex. What remains unclear is how strongly experimental choices, such as network architecture, training, and fitting to brain data, contribute to the observed similarities. Here, we compare a diverse set of nine DNN architectures on their ability to explain the representational geometry of 62 object images in human inferior temporal cortex (hIT), as measured with fMRI. We compare untrained networks to their task-trained counterparts and assess the effect of cross-validated fitting to hIT, by taking a weighted combination of the principal components of features within each layer and, subsequently, a weighted combination of layers. For each combination of training and fitting, we test all models for their correlation with the hIT representational dissimilarity matrix, using independent images and subjects. Trained models outperform untrained models (accounting for 57% more of the explainable variance), suggesting that structured visual features are important for explaining hIT. Model fitting further improves the alignment of DNN and hIT representations (by 124%), suggesting that the relative prevalence of different features in hIT does not readily emerge from the Imagenet object-recognition task used to train the networks. The same models can also explain the disparate representations in primary visual cortex (V1), where stronger weights are given to earlier layers. In each region, all architectures achieved equivalently high performance once trained and fitted. The models' shared properties-deep feedforward hierarchies of spatially restricted nonlinear filters-seem more important than their differences, when modeling human visual representations.
Collapse
Affiliation(s)
- Katherine R Storrs
- Justus Liebig University Giessen, Germany
- Centre for Mind, Brain and Behaviour (CMBB), Research Campus Central Hessen
| | - Tim C Kietzmann
- Donders Institute for Brain, Cognition and Behaviour, Nijmegen, The Netherlands
- MRC Cognition and Brain Sciences Unit, Cambridge, United Kingdom
| | | | - Johannes Mehrer
- MRC Cognition and Brain Sciences Unit, Cambridge, United Kingdom
| | | |
Collapse
|
9
|
Storrs KR, Kietzmann TC, Walther A, Mehrer J, Kriegeskorte N. Diverse Deep Neural Networks All Predict Human Inferior Temporal Cortex Well, After Training and Fitting. J Cogn Neurosci 2021; 33:2044-2064. [PMID: 34272948 DOI: 10.1162/jocn_a_01755] [Citation(s) in RCA: 16] [Impact Index Per Article: 5.3] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/04/2022]
Abstract
Deep neural networks (DNNs) trained on object recognition provide the best current models of high-level visual cortex. What remains unclear is how strongly experimental choices, such as network architecture, training, and fitting to brain data, contribute to the observed similarities. Here, we compare a diverse set of nine DNN architectures on their ability to explain the representational geometry of 62 object images in human inferior temporal cortex (hIT), as measured with fMRI. We compare untrained networks to their task-trained counterparts and assess the effect of cross-validated fitting to hIT, by taking a weighted combination of the principal components of features within each layer and, subsequently, a weighted combination of layers. For each combination of training and fitting, we test all models for their correlation with the hIT representational dissimilarity matrix, using independent images and subjects. Trained models outperform untrained models (accounting for 57% more of the explainable variance), suggesting that structured visual features are important for explaining hIT. Model fitting further improves the alignment of DNN and hIT representations (by 124%), suggesting that the relative prevalence of different features in hIT does not readily emerge from the Imagenet object-recognition task used to train the networks. The same models can also explain the disparate representations in primary visual cortex (V1), where stronger weights are given to earlier layers. In each region, all architectures achieved equivalently high performance once trained and fitted. The models' shared properties-deep feedforward hierarchies of spatially restricted nonlinear filters-seem more important than their differences, when modeling human visual representations.
Collapse
Affiliation(s)
- Katherine R Storrs
- Justus Liebig University Giessen, Germany.,Centre for Mind, Brain and Behaviour (CMBB), Research Campus Central Hessen
| | - Tim C Kietzmann
- Donders Institute for Brain, Cognition and Behaviour, Nijmegen, The Netherlands.,MRC Cognition and Brain Sciences Unit, Cambridge, United Kingdom
| | | | - Johannes Mehrer
- MRC Cognition and Brain Sciences Unit, Cambridge, United Kingdom
| | | |
Collapse
|
10
|
Fjell AM, Grydeland H, Wang Y, Amlien IK, Bartres-Faz D, Brandmaier AM, Düzel S, Elman J, Franz CE, Håberg AK, Kietzmann TC, Kievit RA, Kremen WS, Krogsrud SK, Kühn S, Lindenberger U, Macía D, Mowinckel AM, Nyberg L, Panizzon MS, Solé-Padullés C, Sørensen Ø, Westerhausen R, Walhovd KB. The genetic organization of longitudinal subcortical volumetric change is stable throughout the lifespan. eLife 2021; 10:66466. [PMID: 34180395 PMCID: PMC8260220 DOI: 10.7554/elife.66466] [Citation(s) in RCA: 6] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/12/2021] [Accepted: 06/26/2021] [Indexed: 11/13/2022] Open
Abstract
Development and aging of the cerebral cortex show similar topographic organization and are governed by the same genes. It is unclear whether the same is true for subcortical regions, which follow fundamentally different ontogenetic and phylogenetic principles. We tested the hypothesis that genetically governed neurodevelopmental processes can be traced throughout life by assessing to which degree brain regions that develop together continue to change together through life. Analyzing over 6000 longitudinal MRIs of the brain, we used graph theory to identify five clusters of coordinated development, indexed as patterns of correlated volumetric change in brain structures. The clusters tended to follow placement along the cranial axis in embryonic brain development, suggesting continuity from prenatal stages, and correlated with cognition. Across independent longitudinal datasets, we demonstrated that developmental clusters were conserved through life. Twin-based genetic correlations revealed distinct sets of genes governing change in each cluster. Single-nucleotide polymorphisms-based analyses of 38,127 cross-sectional MRIs showed a similar pattern of genetic volume–volume correlations. In conclusion, coordination of subcortical change adheres to fundamental principles of lifespan continuity and genetic organization.
Collapse
Affiliation(s)
- Anders Martin Fjell
- Center for Lifespan Changes in Brain and Cognition, Department of Psychology, University of Oslo, Oslo, Norway.,Department of Radiology and Nuclear Medicine, Oslo University Hospital, Oslo, Norway
| | - Hakon Grydeland
- Center for Lifespan Changes in Brain and Cognition, Department of Psychology, University of Oslo, Oslo, Norway
| | - Yunpeng Wang
- Center for Lifespan Changes in Brain and Cognition, Department of Psychology, University of Oslo, Oslo, Norway
| | - Inge K Amlien
- Center for Lifespan Changes in Brain and Cognition, Department of Psychology, University of Oslo, Oslo, Norway
| | - David Bartres-Faz
- Departament de Medicina, Facultat de Medicina i Ciències de la Salut, Universitat de Barcelona, and Institut de Neurociències, Universitat de Barcelona, Barcelona, Spain
| | - Andreas M Brandmaier
- Center for Lifespan Psychology, Max Planck Institute for Human Development, Berlin, Germany.,Max Planck UCL Centre for Computational Psychiatry and Ageing Research, Berlin, Germany
| | - Sandra Düzel
- Center for Lifespan Psychology, Max Planck Institute for Human Development, Berlin, Germany
| | - Jeremy Elman
- Center for Behavioral Genomics Twin Research Laboratory, University of California, San Diego, La Jolla, United States
| | - Carol E Franz
- Center for Behavioral Genomics Twin Research Laboratory, University of California, San Diego, La Jolla, United States
| | - Asta K Håberg
- Department of Radiology and Nuclear Medicine, St. Olavs Hospital, Trondheim University Hospital, Trondheim, Norway.,Department of Neuromedicine and Movement Science, Faculty of Medicine and Health Sciences, Norwegian University of Science and Technology (NTNU), Trondheim, Norway
| | - Tim C Kietzmann
- MRC Cognition and Brain Sciences Unit, University of Cambridge, Cambridge, United Kingdom.,Donders Institute for Brain, Cognition and Behaviour, Radboud University, Nijmegen, Netherlands
| | - Rogier Andrew Kievit
- MRC Cognition and Brain Sciences Unit, University of Cambridge, Cambridge, United Kingdom
| | - William S Kremen
- Center for Behavioral Genomics Twin Research Laboratory, University of California, San Diego, La Jolla, United States.,Center of Excellence for Stress and Mental Health, VA San Diego Healthcare System, La Jolla, United States
| | - Stine K Krogsrud
- Center for Lifespan Changes in Brain and Cognition, Department of Psychology, University of Oslo, Oslo, Norway
| | - Simone Kühn
- Center for Lifespan Psychology, Max Planck Institute for Human Development, Berlin, Germany.,Lise Meitner Group for Environmental Neuroscience, Max Planck Institute for Human Development, Berlin, Germany
| | - Ulman Lindenberger
- Center for Lifespan Psychology, Max Planck Institute for Human Development, Berlin, Germany.,Max Planck UCL Centre for Computational Psychiatry and Ageing Research, Berlin, Germany
| | - Didac Macía
- Departament de Medicina, Facultat de Medicina i Ciències de la Salut, Universitat de Barcelona, and Institut de Neurociències, Universitat de Barcelona, Barcelona, Spain
| | - Athanasia Monika Mowinckel
- Center for Lifespan Changes in Brain and Cognition, Department of Psychology, University of Oslo, Oslo, Norway
| | - Lars Nyberg
- Center for Lifespan Changes in Brain and Cognition, Department of Psychology, University of Oslo, Oslo, Norway.,Department of Radiation Sciences, Umeå Center for Functional Brain Imaging, Umeå University, Umeå, Sweden
| | - Matthew S Panizzon
- Center for Behavioral Genomics Twin Research Laboratory, University of California, San Diego, La Jolla, United States.,Department of Psychiatry, University of California, San Diego, La Jolla, United States
| | - Cristina Solé-Padullés
- Departament de Medicina, Facultat de Medicina i Ciències de la Salut, Universitat de Barcelona, and Institut de Neurociències, Universitat de Barcelona, Barcelona, Spain
| | - Øystein Sørensen
- Center for Lifespan Changes in Brain and Cognition, Department of Psychology, University of Oslo, Oslo, Norway
| | - Rene Westerhausen
- Center for Lifespan Changes in Brain and Cognition, Department of Psychology, University of Oslo, Oslo, Norway
| | - Kristine Beate Walhovd
- Center for Lifespan Changes in Brain and Cognition, Department of Psychology, University of Oslo, Oslo, Norway.,Department of Radiology and Nuclear Medicine, Oslo University Hospital, Oslo, Norway
| |
Collapse
|
11
|
Fjell AM, Sørensen Ø, Amlien IK, Bartrés-Faz D, Bros DM, Buchmann N, Demuth I, Drevon CA, Düzel S, Ebmeier KP, Idland AV, Kietzmann TC, Kievit R, Kühn S, Lindenberger U, Mowinckel AM, Nyberg L, Price D, Sexton CE, Solé-Padullés C, Pudas S, Sederevicius D, Suri S, Wagner G, Watne LO, Westerhausen R, Zsoldos E, Walhovd KB. Self-reported sleep relates to hippocampal atrophy across the adult lifespan: results from the Lifebrain consortium. Sleep 2021; 43:5628807. [PMID: 31738420 PMCID: PMC7215271 DOI: 10.1093/sleep/zsz280] [Citation(s) in RCA: 41] [Impact Index Per Article: 13.7] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/16/2019] [Revised: 10/25/2019] [Indexed: 12/17/2022] Open
Abstract
Objectives Poor sleep is associated with multiple age-related neurodegenerative and neuropsychiatric conditions. The hippocampus plays a special role in sleep and sleep-dependent cognition, and accelerated hippocampal atrophy is typically seen with higher age. Hence, it is critical to establish how the relationship between sleep and hippocampal volume loss unfolds across the adult lifespan. Methods Self-reported sleep measures and MRI-derived hippocampal volumes were obtained from 3105 cognitively normal participants (18–90 years) from major European brain studies in the Lifebrain consortium. Hippocampal volume change was estimated from 5116 MRIs from 1299 participants for whom longitudinal MRIs were available, followed up to 11 years with a mean interval of 3.3 years. Cross-sectional analyses were repeated in a sample of 21,390 participants from the UK Biobank. Results No cross-sectional sleep—hippocampal volume relationships were found. However, worse sleep quality, efficiency, problems, and daytime tiredness were related to greater hippocampal volume loss over time, with high scorers showing 0.22% greater annual loss than low scorers. The relationship between sleep and hippocampal atrophy did not vary across age. Simulations showed that the observed longitudinal effects were too small to be detected as age-interactions in the cross-sectional analyses. Conclusions Worse self-reported sleep is associated with higher rates of hippocampal volume decline across the adult lifespan. This suggests that sleep is relevant to understand individual differences in hippocampal atrophy, but limited effect sizes call for cautious interpretation.
Collapse
Affiliation(s)
- Anders M Fjell
- Center for Lifespan Changes in Brain and Cognition, University of Oslo, Norway.,Department of Radiology and Nuclear Medicine, Oslo University Hospital, Norway
| | - Øystein Sørensen
- Center for Lifespan Changes in Brain and Cognition, University of Oslo, Norway
| | - Inge K Amlien
- Center for Lifespan Changes in Brain and Cognition, University of Oslo, Norway
| | - David Bartrés-Faz
- Departament de Medicina, Facultat de Medicina i Ciències de la Salut, Universitat de Barcelona, and Institut de Neurociències, Universitat de Barcelona, Spain
| | - Didac Maciá Bros
- Departament de Medicina, Facultat de Medicina i Ciències de la Salut, Universitat de Barcelona, and Institut de Neurociències, Universitat de Barcelona, Spain
| | - Nikolaus Buchmann
- Department of Cardiology, Charité - University Medicine Berlin Campus Benjamin Franklin, Berlin, Germany
| | - Ilja Demuth
- Charité - Universitätsmedizin Berlin, corporate member of Freie Universität Berlin, Humboldt-Universität zu Berlin, and Berlin Institute of Health, Lipid Clinic at the Interdisciplinary Metabolism Center, Germany
| | - Christian A Drevon
- Vitas AS, Research Park, Gaustadalleen 21, 0349, Oslo and 6 University of Oslo, Department of Nutrition, Institute of Basic Medical Sciences, Faculty of Medicine, Medicine/University of Oslo, Norway
| | - Sandra Düzel
- Max Planck Institute for Human Development, Germany
| | | | - Ane-Victoria Idland
- Center for Lifespan Changes in Brain and Cognition, University of Oslo, Norway.,Oslo Delirium Research Group, Department of Geriatric Medicine, University of Oslo, Norway.,Institute of Basic Medical Sciences, University of Oslo, Norway
| | - Tim C Kietzmann
- MRC Cognition and Brain Sciences Unit, University of Cambridge, UK
| | - Rogier Kievit
- MRC Cognition and Brain Sciences Unit, University of Cambridge, UK
| | - Simone Kühn
- Max Planck Institute for Human Development, Germany.,Department of Psychiatry and Psychotherapy, University Medical Center Hamburg-Eppendorf, Germany
| | | | | | - Lars Nyberg
- Umeå Center for Functional Brain Imaging, Umeå University, Umeå, Sweden
| | - Darren Price
- MRC Cognition and Brain Sciences Unit, University of Cambridge, UK
| | - Claire E Sexton
- Department of Psychiatry, University of Oxford, UK.,Global Brain Health Institute, Department of Neurology, University of California San Francisco, CA.,Wellcome Centre for Integrative Neuroimaging, University of Oxford, UK
| | - Cristina Solé-Padullés
- Departament de Medicina, Facultat de Medicina i Ciències de la Salut, Universitat de Barcelona, and Institut de Neurociències, Universitat de Barcelona, Spain
| | - Sara Pudas
- Umeå Center for Functional Brain Imaging, Umeå University, Umeå, Sweden
| | | | - Sana Suri
- Department of Psychiatry, University of Oxford, UK.,Wellcome Centre for Integrative Neuroimaging, University of Oxford, UK
| | - Gerd Wagner
- Psychiatric Brain and Body Research Group, Department of Psychiatry and Psychotherapy, Jena University Hospital, Jena, Germany
| | - Leiv Otto Watne
- Oslo Delirium Research Group, Department of Geriatric Medicine, University of Oslo, Norway
| | - René Westerhausen
- Center for Lifespan Changes in Brain and Cognition, University of Oslo, Norway
| | - Enikő Zsoldos
- Department of Psychiatry, University of Oxford, UK.,Wellcome Centre for Integrative Neuroimaging, University of Oxford, UK
| | - Kristine B Walhovd
- Center for Lifespan Changes in Brain and Cognition, University of Oslo, Norway.,Department of Radiology and Nuclear Medicine, Oslo University Hospital, Norway
| |
Collapse
|
12
|
Mehrer J, Spoerer CJ, Jones EC, Kriegeskorte N, Kietzmann TC. An ecologically motivated image dataset for deep learning yields better models of human vision. Proc Natl Acad Sci U S A 2021; 118:e2011417118. [PMID: 33593900 PMCID: PMC7923360 DOI: 10.1073/pnas.2011417118] [Citation(s) in RCA: 36] [Impact Index Per Article: 12.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/24/2022] Open
Abstract
Deep neural networks provide the current best models of visual information processing in the primate brain. Drawing on work from computer vision, the most commonly used networks are pretrained on data from the ImageNet Large Scale Visual Recognition Challenge. This dataset comprises images from 1,000 categories, selected to provide a challenging testbed for automated visual object recognition systems. Moving beyond this common practice, we here introduce ecoset, a collection of >1.5 million images from 565 basic-level categories selected to better capture the distribution of objects relevant to humans. Ecoset categories were chosen to be both frequent in linguistic usage and concrete, thereby mirroring important physical objects in the world. We test the effects of training on this ecologically more valid dataset using multiple instances of two neural network architectures: AlexNet and vNet, a novel architecture designed to mimic the progressive increase in receptive field sizes along the human ventral stream. We show that training on ecoset leads to significant improvements in predicting representations in human higher-level visual cortex and perceptual judgments, surpassing the previous state of the art. Significant and highly consistent benefits are demonstrated for both architectures on two separate functional magnetic resonance imaging (fMRI) datasets and behavioral data, jointly covering responses to 1,292 visual stimuli from a wide variety of object categories. These results suggest that computational visual neuroscience may take better advantage of the deep learning framework by using image sets that reflect the human perceptual and cognitive experience. Ecoset and trained network models are openly available to the research community.
Collapse
Affiliation(s)
- Johannes Mehrer
- MRC Cognition and Brain Sciences Unit, University of Cambridge, CB2 7EF Cambridge, United Kingdom
| | - Courtney J Spoerer
- MRC Cognition and Brain Sciences Unit, University of Cambridge, CB2 7EF Cambridge, United Kingdom
| | - Emer C Jones
- MRC Cognition and Brain Sciences Unit, University of Cambridge, CB2 7EF Cambridge, United Kingdom
| | - Nikolaus Kriegeskorte
- Department of Psychology, Zuckerman Institute, Columbia University, New York, NY 10027
| | - Tim C Kietzmann
- MRC Cognition and Brain Sciences Unit, University of Cambridge, CB2 7EF Cambridge, United Kingdom;
- Donders Institute for Brain, Cognition and Behaviour, Radboud University, 6525 XZ Nijmegen, Netherlands
| |
Collapse
|
13
|
Fjell AM, Sørensen Ø, Amlien IK, Bartrés-Faz D, Brandmaier AM, Buchmann N, Demuth I, Drevon CA, Düzel S, Ebmeier KP, Ghisletta P, Idland AV, Kietzmann TC, Kievit RA, Kühn S, Lindenberger U, Magnussen F, Macià D, Mowinckel AM, Nyberg L, Sexton CE, Solé-Padullés C, Pudas S, Roe JM, Sederevicius D, Suri S, Vidal-Piñeiro D, Wagner G, Watne LO, Westerhausen R, Zsoldos E, Walhovd KB. Poor Self-Reported Sleep is Related to Regional Cortical Thinning in Aging but not Memory Decline-Results From the Lifebrain Consortium. Cereb Cortex 2020; 31:1953-1969. [PMID: 33236064 PMCID: PMC7945023 DOI: 10.1093/cercor/bhaa332] [Citation(s) in RCA: 17] [Impact Index Per Article: 4.3] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/20/2020] [Revised: 09/17/2020] [Accepted: 10/12/2020] [Indexed: 12/16/2022] Open
Abstract
We examined whether sleep quality and quantity are associated with cortical and memory changes in cognitively healthy participants across the adult lifespan. Associations between self-reported sleep parameters (Pittsburgh Sleep Quality Index, PSQI) and longitudinal cortical change were tested using five samples from the Lifebrain consortium (n = 2205, 4363 MRIs, 18–92 years). In additional analyses, we tested coherence with cell-specific gene expression maps from the Allen Human Brain Atlas, and relations to changes in memory performance. “PSQI # 1 Subjective sleep quality” and “PSQI #5 Sleep disturbances” were related to thinning of the right lateral temporal cortex, with lower quality and more disturbances being associated with faster thinning. The association with “PSQI #5 Sleep disturbances” emerged after 60 years, especially in regions with high expression of genes related to oligodendrocytes and S1 pyramidal neurons. None of the sleep scales were related to a longitudinal change in episodic memory function, suggesting that sleep-related cortical changes were independent of cognitive decline. The relationship to cortical brain change suggests that self-reported sleep parameters are relevant in lifespan studies, but small effect sizes indicate that self-reported sleep is not a good biomarker of general cortical degeneration in healthy older adults.
Collapse
Affiliation(s)
- Anders M Fjell
- Center for Lifespan Changes in Brain and Cognition, University of Oslo, 0315 Oslo, Norway.,Department of Radiology and Nuclear Medicine, Oslo University Hospital, 0188 Oslo, Norway
| | - Øystein Sørensen
- Center for Lifespan Changes in Brain and Cognition, University of Oslo, 0315 Oslo, Norway
| | - Inge K Amlien
- Center for Lifespan Changes in Brain and Cognition, University of Oslo, 0315 Oslo, Norway
| | - David Bartrés-Faz
- Departament de Medicina, Facultat de Medicina i Ciències de la Salut, Institut de Neurociències, Universitat de Barcelona, 08007 Barcelona, Spain
| | - Andreas M Brandmaier
- Center for Lifespan Psychology, Max Planck Institute for Human Development, 14195 Berlin, Germany.,Max Planck UCL Centre for Computational Psychiatry and Ageing Research, Berlin, Germany, and London, UK
| | - Nikolaus Buchmann
- Department of Cardiology, Charité - University Medicine Berlin Campus Benjamin Franklin, 12203 Berlin, Germany
| | - Ilja Demuth
- Charité - Universitätsmedizin Berlin, corporate member of Freie Universität Berlin, Humboldt-Universität zu Berlin, and Berlin Institute of Health, Lipid Clinic at the Interdisciplinary Metabolism Center, Charité - Universitätsmedizin Berlin, BCRT - Berlin Institute of Health Center for Regenerative Therapies, 10117 Berlin, Germany
| | - Christian A Drevon
- Vitas AS, Research Park, Gaustadalleen 21, 0349 Oslo, Norway.,Department of Nutrition, Institute of Basic Medical Sciences, Faculty of Medicine, University of Oslo, 0315 Oslo, Norway
| | - Sandra Düzel
- Center for Lifespan Psychology, Max Planck Institute for Human Development, 14195 Berlin, Germany
| | - Klaus P Ebmeier
- Department of Psychiatry, University of Oxford, Oxford OX1 2JD UK
| | - Paolo Ghisletta
- Faculty of Psychology and Educational Sciences, Swiss Distance University Institute, Swiss National Centre of Competence in Research LIVES, University of Geneva, 1205 Geneva, Switzerland
| | - Ane-Victoria Idland
- Center for Lifespan Changes in Brain and Cognition, University of Oslo, 0315 Oslo, Norway.,Oslo Delirium Research Group, Department of Geriatric Medicine, University of Oslo, 0315 Oslo, Norway.,Institute of Basic Medical Sciences, University of Oslo, 0315 Oslo, Norway
| | - Tim C Kietzmann
- MRC Cognition and Brain Sciences Unit, University of Cambridge, Cambridge CB2 1TN, UK.,Donders Institute for Brain, Cognition and Behaviour, Radboud University, 6525 XZ Nijmegen, The Netherlands
| | - Rogier A Kievit
- MRC Cognition and Brain Sciences Unit, University of Cambridge, Cambridge CB2 1TN, UK
| | - Simone Kühn
- Center for Lifespan Psychology, Max Planck Institute for Human Development, 14195 Berlin, Germany.,Department of Psychiatry and Psychotherapy, University Medical Center Hamburg-Eppendorf, 20251 Hamburg, Germany
| | - Ulman Lindenberger
- Center for Lifespan Psychology, Max Planck Institute for Human Development, 14195 Berlin, Germany.,Max Planck UCL Centre for Computational Psychiatry and Ageing Research, Berlin, Germany, and London, UK
| | - Fredrik Magnussen
- Center for Lifespan Changes in Brain and Cognition, University of Oslo, 0315 Oslo, Norway
| | - Didac Macià
- Departament de Medicina, Facultat de Medicina i Ciències de la Salut, Institut de Neurociències, Universitat de Barcelona, 08007 Barcelona, Spain
| | - Athanasia M Mowinckel
- Center for Lifespan Changes in Brain and Cognition, University of Oslo, 0315 Oslo, Norway
| | - Lars Nyberg
- Umeå Center for Functional Brain Imaging, Umeå University, 901 87 Umeå, Sweden
| | - Claire E Sexton
- Department of Psychiatry, University of Oxford, Oxford OX1 2JD UK.,Global Brain Health Institute, Department of Neurology, University of California, San Francisco, San Francisco, CA 94143, USA.,Wellcome Centre for Integrative Neuroimaging, University of Oxford, Oxford OX1 2JD, UK
| | - Cristina Solé-Padullés
- Departament de Medicina, Facultat de Medicina i Ciències de la Salut, Institut de Neurociències, Universitat de Barcelona, 08007 Barcelona, Spain
| | - Sara Pudas
- Umeå Center for Functional Brain Imaging, Umeå University, 901 87 Umeå, Sweden
| | - James M Roe
- Center for Lifespan Changes in Brain and Cognition, University of Oslo, 0315 Oslo, Norway
| | - Donatas Sederevicius
- Center for Lifespan Changes in Brain and Cognition, University of Oslo, 0315 Oslo, Norway
| | - Sana Suri
- Department of Psychiatry, University of Oxford, Oxford OX1 2JD UK.,Wellcome Centre for Integrative Neuroimaging, University of Oxford, Oxford OX1 2JD, UK
| | - Didac Vidal-Piñeiro
- Center for Lifespan Changes in Brain and Cognition, University of Oslo, 0315 Oslo, Norway
| | - Gerd Wagner
- Psychiatric Brain and Body Research Group, Department of Psychiatry and Psychotherapy, Jena University Hospital, 07743 Jena, Germany
| | - Leiv Otto Watne
- Oslo Delirium Research Group, Department of Geriatric Medicine, University of Oslo, 0315 Oslo, Norway
| | - René Westerhausen
- Center for Lifespan Changes in Brain and Cognition, University of Oslo, 0315 Oslo, Norway
| | - Enikő Zsoldos
- Department of Psychiatry, University of Oxford, Oxford OX1 2JD UK.,Wellcome Centre for Integrative Neuroimaging, University of Oxford, Oxford OX1 2JD, UK
| | - Kristine B Walhovd
- Center for Lifespan Changes in Brain and Cognition, University of Oslo, 0315 Oslo, Norway.,Department of Radiology and Nuclear Medicine, Oslo University Hospital, 0188 Oslo, Norway
| |
Collapse
|
14
|
Abstract
Deep neural networks (DNNs) excel at visual recognition tasks and are increasingly used as a modeling framework for neural computations in the primate brain. Just like individual brains, each DNN has a unique connectivity and representational profile. Here, we investigate individual differences among DNN instances that arise from varying only the random initialization of the network weights. Using tools typically employed in systems neuroscience, we show that this minimal change in initial conditions prior to training leads to substantial differences in intermediate and higher-level network representations despite similar network-level classification performance. We locate the origins of the effects in an under-constrained alignment of category exemplars, rather than misaligned category centroids. These results call into question the common practice of using single networks to derive insights into neural information processing and rather suggest that computational neuroscientists working with DNNs may need to base their inferences on groups of multiple network instances.
Collapse
Affiliation(s)
- Johannes Mehrer
- MRC Cognition and Brain Sciences Unit, University of Cambridge, 15 Chaucer Road, Cambridge, CB2 7EF, UK.
| | - Courtney J Spoerer
- MRC Cognition and Brain Sciences Unit, University of Cambridge, 15 Chaucer Road, Cambridge, CB2 7EF, UK
| | | | - Tim C Kietzmann
- MRC Cognition and Brain Sciences Unit, University of Cambridge, 15 Chaucer Road, Cambridge, CB2 7EF, UK.
- Donders Institute for Brain, Cognition and Behaviour, Radboud University, Montessorilaan 3, 6525, HR, Nijmegen, Netherlands.
| |
Collapse
|
15
|
Spoerer CJ, Kietzmann TC, Mehrer J, Charest I, Kriegeskorte N. Recurrent neural networks can explain flexible trading of speed and accuracy in biological vision. PLoS Comput Biol 2020; 16:e1008215. [PMID: 33006992 PMCID: PMC7556458 DOI: 10.1371/journal.pcbi.1008215] [Citation(s) in RCA: 36] [Impact Index Per Article: 9.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/20/2019] [Revised: 10/14/2020] [Accepted: 08/03/2020] [Indexed: 11/18/2022] Open
Abstract
Deep feedforward neural network models of vision dominate in both computational neuroscience and engineering. The primate visual system, by contrast, contains abundant recurrent connections. Recurrent signal flow enables recycling of limited computational resources over time, and so might boost the performance of a physically finite brain or model. Here we show: (1) Recurrent convolutional neural network models outperform feedforward convolutional models matched in their number of parameters in large-scale visual recognition tasks on natural images. (2) Setting a confidence threshold, at which recurrent computations terminate and a decision is made, enables flexible trading of speed for accuracy. At a given confidence threshold, the model expends more time and energy on images that are harder to recognise, without requiring additional parameters for deeper computations. (3) The recurrent model's reaction time for an image predicts the human reaction time for the same image better than several parameter-matched and state-of-the-art feedforward models. (4) Across confidence thresholds, the recurrent model emulates the behaviour of feedforward control models in that it achieves the same accuracy at approximately the same computational cost (mean number of floating-point operations). However, the recurrent model can be run longer (higher confidence threshold) and then outperforms parameter-matched feedforward comparison models. These results suggest that recurrent connectivity, a hallmark of biological visual systems, may be essential for understanding the accuracy, flexibility, and dynamics of human visual recognition.
Collapse
Affiliation(s)
- Courtney J. Spoerer
- Medical Research Council Cognition and Brain Sciences Unit, University of Cambridge, Cambridge, United Kingdom
| | - Tim C. Kietzmann
- Medical Research Council Cognition and Brain Sciences Unit, University of Cambridge, Cambridge, United Kingdom
- Donders Institute for Brain, Cognition and Behaviour, Radboud University, Nijmegen, Netherlands
| | - Johannes Mehrer
- Medical Research Council Cognition and Brain Sciences Unit, University of Cambridge, Cambridge, United Kingdom
| | - Ian Charest
- School of Psychology and Centre for Human Brain Health, University of Birmingham, United Kingdom
| | - Nikolaus Kriegeskorte
- Department of Psychology, Department of Neuroscience, Department of Electrical Engineering, Zuckerman Mind Brain Behavior Institute, Columbia University, New York, NY, USA
| |
Collapse
|
16
|
Wilming N, Onat S, Ossandón JP, Açık A, Kietzmann TC, Kaspar K, Gameiro RR, Vormberg A, König P. An extensive dataset of eye movements during viewing of complex images. Sci Data 2017; 4:160126. [PMID: 28140391 PMCID: PMC5283059 DOI: 10.1038/sdata.2016.126] [Citation(s) in RCA: 22] [Impact Index Per Article: 3.1] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/03/2016] [Accepted: 11/15/2016] [Indexed: 11/18/2022] Open
Abstract
We present a dataset of free-viewing eye-movement recordings that contains more than 2.7 million fixation locations from 949 observers on more than 1000 images from different categories. This dataset aggregates and harmonizes data from 23 different studies conducted at the Institute of Cognitive Science at Osnabrück University and the University Medical Center in Hamburg-Eppendorf. Trained personnel recorded all studies under standard conditions with homogeneous equipment and parameter settings. All studies allowed for free eye-movements, and differed in the age range of participants (~7-80 years), stimulus sizes, stimulus modifications (phase scrambled, spatial filtering, mirrored), and stimuli categories (natural and urban scenes, web sites, fractal, pink-noise, and ambiguous artistic figures). The size and variability of viewing behavior within this dataset presents a strong opportunity for evaluating and comparing computational models of overt attention, and furthermore, for thoroughly quantifying strategies of viewing behavior. This also makes the dataset a good starting point for investigating whether viewing strategies change in patient groups.
Collapse
Affiliation(s)
- Niklas Wilming
- Institute of Cognitive Science, University of Osnabrück, 49069 Osnabrück, Germany
- Department of Neurophysiology and Pathophysiology, University Medical Center Hamburg-Eppendorf, 20246 Hamburg, Germany
| | - Selim Onat
- Institute of Cognitive Science, University of Osnabrück, 49069 Osnabrück, Germany
- Department of Systems Neuroscience, University Medical Center Hamburg-Eppendorf, 20246 Hamburg, Germany
| | - José P. Ossandón
- Institute of Cognitive Science, University of Osnabrück, 49069 Osnabrück, Germany
- Biological Psychology & Neuropsychology, University of Hamburg, 20146 Hamburg, Germany
| | - Alper Açık
- Institute of Cognitive Science, University of Osnabrück, 49069 Osnabrück, Germany
- Department of Psychology, Ozyegin University, 34716 Istanbul, Turkey
| | - Tim C. Kietzmann
- Institute of Cognitive Science, University of Osnabrück, 49069 Osnabrück, Germany
- Medical Research Council, Cognition and Brain Sciences Unit, Cambridge CB2 7EF, UK
| | - Kai Kaspar
- Institute of Cognitive Science, University of Osnabrück, 49069 Osnabrück, Germany
- Department of Psychology, University of Cologne, 50931 Cologne, Germany
| | - Ricardo R. Gameiro
- Institute of Cognitive Science, University of Osnabrück, 49069 Osnabrück, Germany
| | - Alexandra Vormberg
- Institute of Cognitive Science, University of Osnabrück, 49069 Osnabrück, Germany
- Frankfurt Institute for Advanced Studies (FIAS), 60438 Frankfurt/Main, Germany
- Ernst Strüngmann Institute (ESI) for Neuroscience in Cooperation with Max Planck Society, 60528 Frankfurt/Main, Germany
| | - Peter König
- Institute of Cognitive Science, University of Osnabrück, 49069 Osnabrück, Germany
- Department of Neurophysiology and Pathophysiology, University Medical Center Hamburg-Eppendorf, 20246 Hamburg, Germany
| |
Collapse
|
17
|
Wilming N, Kietzmann TC, Jutras M, Xue C, Treue S, Buffalo EA, König P. Differential Contribution of Low- and High-level Image Content to Eye Movements in Monkeys and Humans. Cereb Cortex 2017; 27:279-293. [PMID: 28077512 PMCID: PMC5942390 DOI: 10.1093/cercor/bhw399] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/08/2016] [Accepted: 12/13/2016] [Indexed: 11/25/2022] Open
Abstract
Oculomotor selection exerts a fundamental impact on our experience of the environment. To better understand the underlying principles, researchers typically rely on behavioral data from humans, and electrophysiological recordings in macaque monkeys. This approach rests on the assumption that the same selection processes are at play in both species. To test this assumption, we compared the viewing behavior of 106 humans and 11 macaques in an unconstrained free-viewing task. Our data-driven clustering analyses revealed distinct human and macaque clusters, indicating species-specific selection strategies. Yet, cross-species predictions were found to be above chance, indicating some level of shared behavior. Analyses relying on computational models of visual saliency indicate that such cross-species commonalities in free viewing are largely due to similar low-level selection mechanisms, with only a small contribution by shared higher level selection mechanisms and with consistent viewing behavior of monkeys being a subset of the consistent viewing behavior of humans.
Collapse
Affiliation(s)
- Niklas Wilming
- Institute of Cognitive Science, University of Osnabrück, Osnabrück, Germany.,Department of Physiology and Biophysics, University of Washington, Seattle, WA 98195, USA.,Yerkes National Primate Research Center, Atlanta, GA 30329, USA.,Department of Neurophysiology and Pathophysiology, University Medical Center Hamburg-Eppendorf, Hamburg, Germany.,Washington National Primate Research Center, Seattle, WA 09195, USA
| | - Tim C Kietzmann
- Institute of Cognitive Science, University of Osnabrück, Osnabrück, Germany.,Medical Research Council, Cognition and Brain Sciences Unit, Cambridge CB2 7EF, UK
| | - Megan Jutras
- Department of Physiology and Biophysics, University of Washington, Seattle, WA 98195, USA.,Yerkes National Primate Research Center, Atlanta, GA 30329, USA.,Washington National Primate Research Center, Seattle, WA 09195, USA
| | - Cheng Xue
- Cognitive Neuroscience Laboratory, German Primate Center - Leibniz-Institute for Primate Research, Goettingen, Germany
| | - Stefan Treue
- Cognitive Neuroscience Laboratory, German Primate Center - Leibniz-Institute for Primate Research, Goettingen, Germany.,Faculty of Biology and Psychology, Goettingen University, Goettingen, Germany.,Leibniz-ScienceCampus Primate Cognition, Goettingen, Germany
| | - Elizabeth A Buffalo
- Department of Physiology and Biophysics, University of Washington, Seattle, WA 98195, USA.,Yerkes National Primate Research Center, Atlanta, GA 30329, USA.,Washington National Primate Research Center, Seattle, WA 09195, USA
| | - Peter König
- Institute of Cognitive Science, University of Osnabrück, Osnabrück, Germany.,Department of Neurophysiology and Pathophysiology, University Medical Center Hamburg-Eppendorf, Hamburg, Germany
| |
Collapse
|
18
|
König P, Wilming N, Kietzmann TC, Ossandón JP, Onat S, Ehinger BV, Gameiro RR, Kaspar K. Eye movements as a window to cognitive processes. J Eye Mov Res 2016. [DOI: 10.16910/jemr.9.5.3] [Citation(s) in RCA: 17] [Impact Index Per Article: 2.1] [Reference Citation Analysis] [What about the content of this article? (0)] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/18/2022] Open
Abstract
Eye movement research is a highly active and productive research field. Here we focus on how the embodied nature of eye movements can act as a window to the brain and the mind. In particular, we discuss how conscious perception depends on the trajectory of fixated locations and consequently address how fixation locations are selected. Specifically, we argue that the selection of fixation points during visual exploration can be understood to a large degree based on retinotopically structured models. Yet, these models largely ignore spatiotemporal structure in eye-movement sequences. Explaining spatiotemporal structure in eye-movement trajectories requires an understanding of spatiotemporal properties of the visual sampling process. With this in mind, we discuss the availability of external information to internal inference about causes in the world. We demonstrate that visual foraging is a dynamic process that can be systematically modulated either towards exploration or exploitation. For an analysis at high temporal resolution, we suggest a new method: The renewal density allows the investigation of precise temporal relation of eye movements and other actions like a button press. We conclude with an outlook and propose that eye movement research has reached an appropriate stage and can easily be combined with other research methods to utilize this window to the brain and mind to its fullest.
Collapse
|
19
|
Abstract
Faces provide a wealth of information, including the identity of the seen person and social cues, such as the direction of gaze. Crucially, different aspects of face processing require distinct forms of information encoding. Another person's attentional focus can be derived based on a view-dependent code. In contrast, identification benefits from invariance across all viewpoints. Different cortical areas have been suggested to subserve these distinct functions. However, little is known about the temporal aspects of differential viewpoint encoding in the human brain. Here, we combine EEG with multivariate data analyses to resolve the dynamics of face processing with high temporal resolution. This revealed a distinct sequence of viewpoint encoding. Head orientations were encoded first, starting after around 60 msec of processing. Shortly afterward, peaking around 115 msec after stimulus onset, a different encoding scheme emerged. At this latency, mirror-symmetric viewing angles elicited highly similar cortical responses. Finally, about 280 msec after visual onset, EEG response patterns demonstrated a considerable degree of viewpoint invariance across all viewpoints tested, with the noteworthy exception of the front-facing view. Taken together, our results indicate that the processing of facial viewpoints follows a temporal sequence of encoding schemes, potentially mirroring different levels of computational complexity.
Collapse
Affiliation(s)
- Tim C Kietzmann
- Medical Research Council, Cambridge, UK.,University of Osnabrück, Germany
| | | | | | - Peter König
- University of Osnabrück, Germany.,University Medical Center Hamburg-Eppendorf, Germany
| |
Collapse
|
20
|
Kietzmann TC, König P. Effects of contextual information and stimulus ambiguity on overt visual sampling behavior. Vision Res 2015; 110:76-86. [PMID: 25805148 DOI: 10.1016/j.visres.2015.02.023] [Citation(s) in RCA: 9] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/15/2014] [Revised: 02/25/2015] [Accepted: 02/27/2015] [Indexed: 11/26/2022]
Abstract
The sampling of our visual environment through saccadic eye movements is an essential function of the brain, allowing us to overcome the limits of peripheral vision. Understanding which parts of a scene attract overt visual attention is subject to intense research, and considerable progress has been made in unraveling the underlying cortical mechanisms. In contrast to spatial aspects, however, relatively little is understood about temporal aspects of overt visual sampling. At every fixation, the oculomotor system faces the decision whether to keep exploring different aspects of an object or scene or whether to remain fixated to allow for in-depth cortical processing - a situation that can be understood in terms of an exploration-exploitation dilemma. To improve our understanding of the factors involved in these decisions, we here investigate how the level of visual information, experimentally manipulated by scene context and stimulus ambiguity, changes the sampling behavior preceding the recognition of centrally presented ambiguous and disambiguated objects. Behaviorally, we find that context, although only presented until the first voluntary saccade, biases the perceptual outcome and significantly reduces reaction times. Importantly, we find that increased information about an object significantly alters its visual exploration, as evident through increased fixation durations and reduced saccade amplitudes. These results demonstrate that the initial sampling of an object, preceding its recognition, is subject to change based on the amount of information available in the system: increased evidence for its identity biases the exploration-exploitation strategy towards in-depth analyses.
Collapse
Affiliation(s)
- T C Kietzmann
- Institute of Cognitive Science, University of Osnabrück, Osnabrück, Germany.
| | - P König
- Institute of Cognitive Science, University of Osnabrück, Osnabrück, Germany; Department of Neurophysiology and Pathophysiology, University Medical Center Hamburg-Eppendorf, Hamburg, Germany.
| |
Collapse
|
21
|
Abstract
Models of fixation selection are a central tool in the quest to understand how the human mind selects relevant information. Using this tool in the evaluation of competing claims often requires comparing different models' relative performance in predicting eye movements. However, studies use a wide variety of performance measures with markedly different properties, which makes a comparison difficult. We make three main contributions to this line of research: First we argue for a set of desirable properties, review commonly used measures, and conclude that no single measure unites all desirable properties. However the area under the ROC curve (a classification measure) and the KL-divergence (a distance measure of probability distributions) combine many desirable properties and allow a meaningful comparison of critical model performance. We give an analytical proof of the linearity of the ROC measure with respect to averaging over subjects and demonstrate an appropriate correction of entropy-based measures like KL-divergence for small sample sizes in the context of eye-tracking data. Second, we provide a lower bound and an upper bound of these measures, based on image-independent properties of fixation data and between subject consistency respectively. Based on these bounds it is possible to give a reference frame to judge the predictive power of a model of fixation selection . We provide open-source python code to compute the reference frame. Third, we show that the upper, between subject consistency bound holds only for models that predict averages of subject populations. Departing from this we show that incorporating subject-specific viewing behavior can generate predictions which surpass that upper bound. Taken together, these findings lay out the required information that allow a well-founded judgment of the quality of any model of fixation selection and should therefore be reported when a new model is introduced.
Collapse
Affiliation(s)
- Niklas Wilming
- Institute of Cognitive Science, University of Osnabrück, Osnabrück, Germany.
| | | | | | | |
Collapse
|
22
|
Abstract
Our everyday conscious experience of the visual world is fundamentally shaped by the interaction of overt visual attention and object awareness. Although the principal impact of both components is undisputed, it is still unclear how they interact. Here we recorded eye-movements preceding and following conscious object recognition, collected during the free inspection of ambiguous and corresponding unambiguous stimuli. Using this paradigm, we demonstrate that fixations recorded prior to object awareness predict the later recognized object identity, and that subjects accumulate more evidence that is consistent with their later percept than for the alternative. The timing of reached awareness was verified by a reaction-time based correction method and also based on changes in pupil dilation. Control experiments, in which we manipulated the initial locus of visual attention, confirm a causal influence of overt attention on the subsequent result of object perception. The current study thus demonstrates that distinct patterns of overt attentional selection precede object awareness and thereby directly builds on recent electrophysiological findings suggesting two distinct neuronal mechanisms underlying the two phenomena. Our results emphasize the crucial importance of overt visual attention in the formation of our conscious experience of the visual world.
Collapse
Affiliation(s)
- Tim C Kietzmann
- Institute of Cognitive Science, University of Osnabrück, Osnabrück, Germany.
| | | | | |
Collapse
|
23
|
Kietzmann TC, König P. Perceptual learning of parametric face categories leads to the integration of high-level class-based information but not to high-level pop-out. J Vis 2010; 10:20. [PMID: 21106685 DOI: 10.1167/10.13.20] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/24/2022] Open
Abstract
To date, the relative contribution of the different levels of the visual hierarchy during perceptual decisions remains unclear. Typical models of visual processing, with the reverse hierarchy theory (RHT) as a prominent example, strongly emphasize the role of higher levels and interpret lower levels as sequence of simple feature detectors. Here, we investigate this issue based on two analyses. Using a novel combination of perceptual learning based on two classes of parametric faces and a subsequent odd-one-out paradigm, we first test a vital prediction of RHT: high-level pop-out. With this experimental approach, we overcome the low-level confounds of previous studies while still introducing distinct high-level representations. Contrary to previous findings, our analyses show that there is no high-level pop-out, despite very early, near-perfect classification accuracy and extensive training of our subjects. Second, we explore the underlying form of category representation during subsequent stages of perceptual training. This is accomplished by including class-external and class-internal target-distractor combinations. Whereas the subjects' responses during the first sessions are best explained instance-based and dependent on low-level metric differences, later patterns exhibit the inclusion of high-level, class-based information that is independent of target-stimulus similarity. Finally, we show that the utilized level of information is highly task-dependent.
Collapse
Affiliation(s)
- Tim C Kietzmann
- Neurobiopsychology Laboratory, Institute of Cognitive Science, University of Osnabrück, Albrechstrasse 28, Osnabrück, Germany.
| | | |
Collapse
|
24
|
Betz T, Kietzmann TC, Wilming N, König P. Investigating task-dependent top-down effects on overt visual attention. J Vis 2010; 10:15.1-14. [PMID: 20377292 DOI: 10.1167/10.3.15] [Citation(s) in RCA: 44] [Impact Index Per Article: 3.1] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/02/2009] [Accepted: 02/09/2010] [Indexed: 11/24/2022] Open
Abstract
Different tasks can induce different viewing behavior, yet it is still an open question how or whether at all high-level task information interacts with the bottom-up processing of stimulus-related information. Two possible causal routes are considered in this paper. Firstly, the weak top-down hypothesis, according to which top-down effects are mediated by changes of feature weights in the bottom-up system. Secondly, the strong top-down hypothesis, which proposes that top-down information acts independently of the bottom-up process. To clarify the influences of these different routes, viewing behavior was recorded on web pages for three different tasks: free viewing, content awareness, and information search. The data reveal significant task-dependent differences in viewing behavior that are accompanied by minor changes in feature-fixation correlations. Extensive computational modeling shows that these small but significant changes are insufficient to explain the observed differences in viewing behavior. Collectively, the results show that task-dependent differences in the current setting are not mediated by a reweighting of features in the bottom-up hierarchy, ruling out the weak top-down hypothesis. Consequently, the strong top-down hypothesis is the most viable explanation for the observed data.
Collapse
Affiliation(s)
- Torsten Betz
- Institute of Cognitive Science, University of Osnabrück, Osnabrück, Germany.
| | | | | | | |
Collapse
|
25
|
Kietzmann TC, Lange S, Riedmiller M. Computational object recognition: a biologically motivated approach. Biol Cybern 2009; 100:59-79. [PMID: 19089445 DOI: 10.1007/s00422-008-0281-6] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/28/2008] [Accepted: 11/14/2008] [Indexed: 05/27/2023]
Abstract
We propose a conceptual framework for artificial object recognition systems based on findings from neurophysiological and neuropsychological research on the visual system in primate cortex. We identify some essential questions, which have to be addressed in the course of designing object recognition systems. As answers, we review some major aspects of biological object recognition, which are then translated into the technical field of computer vision. The key suggestions are the use of incremental and view-based approaches together with the ability of online feature selection and the interconnection of object-views to form an overall object representation. The effectiveness of the computational approach is estimated by testing a possible realization in various tasks and conditions explicitly designed to allow for a direct comparison with the biological counterpart. The results exhibit excellent performance with regard to recognition accuracy, the creation of sparse models and the selection of appropriate features.
Collapse
Affiliation(s)
- Tim C Kietzmann
- Institute of Cognitive Science, University of Osnabrück, Osnabrück, Germany.
| | | | | |
Collapse
|