1
|
Li W, Li J, Chu C, Cao D, Shi W, Zhang Y, Jiang T. Common Sequential Organization of Face Processing in the Human Brain and Convolutional Neural Networks. Neuroscience 2024; 541:1-13. [PMID: 38266906 DOI: 10.1016/j.neuroscience.2024.01.015] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/09/2023] [Revised: 01/11/2024] [Accepted: 01/16/2024] [Indexed: 01/26/2024]
Abstract
Face processing includes two crucial processing levels - face detection and face recognition. However, it remains unclear how human brains organize the two processing levels sequentially. While some studies found that faces are recognized as fast as they are detected, others have reported that faces are detected first, followed by recognition. We discriminated the two processing levels on a fine time scale by combining human intracranial EEG (two females, three males, and three subjects without reported sex information) and representation similarity analysis. Our results demonstrate that the human brain exhibits a "detection-first, recognition-later" pattern during face processing. In addition, we used convolutional neural networks to test the hypothesis that the sequential organization of the two face processing levels in the brain reflects computational optimization. Our findings showed that the networks trained on face recognition also exhibited the "detection-first, recognition-later" pattern. Moreover, this sequential organization mechanism developed gradually during the training of the networks and was observed only for correctly predicted images. These findings collectively support the computational account as to why the brain organizes them in this way.
Collapse
Affiliation(s)
- Wenlu Li
- Brainnetome Center, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China; School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, China
| | - Jin Li
- School of Psychology, Capital Normal University, Beijing 100048, China
| | - Congying Chu
- Brainnetome Center, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China
| | - Dan Cao
- Brainnetome Center, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China
| | - Weiyang Shi
- Brainnetome Center, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China
| | - Yu Zhang
- Research Center for Augmented Intelligence, Zhejiang Lab, Hangzhou 311100, China
| | - Tianzi Jiang
- Brainnetome Center, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China; School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, China; Research Center for Augmented Intelligence, Zhejiang Lab, Hangzhou 311100, China; Xiaoxiang Institute for Brain Health and Yongzhou Central Hospital, Yongzhou 425000, Hunan Province, China.
| |
Collapse
|
2
|
Tarigopula P, Fairhall SL, Bavaresco A, Truong N, Hasson U. Improved prediction of behavioral and neural similarity spaces using pruned DNNs. Neural Netw 2023; 168:89-104. [PMID: 37748394 DOI: 10.1016/j.neunet.2023.08.049] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/23/2023] [Revised: 08/18/2023] [Accepted: 08/28/2023] [Indexed: 09/27/2023]
Abstract
Deep Neural Networks (DNNs) have become an important tool for modeling brain and behavior. One key area of interest has been to apply these networks to model human similarity judgements. Several previous works have used the embeddings from the penultimate layer of vision DNNs and showed that a reweighting of these features improves the fit between human similarity judgments and DNNs. These studies underline the idea that these embeddings form a good basis set but lack the correct level of salience. Here we re-examined the grounds for this idea and on the contrary, we hypothesized that these embeddings, beyond forming a good basis set, also have the correct level of salience to account for similarity judgments. It is just that the huge dimensional embedding needs to be pruned to select those features relevant for the considered domain for which a similarity space is modeled. In Study 1 we supervised DNN pruning based on a subset of human similarity judgments. We found that pruning: i) improved out-of-sample prediction of human similarity judgments from DNN embeddings, ii) produced better alignment with WordNet hierarchy, and iii) retained much higher classification accuracy than reweighting. Study 2 showed that pruning by neurobiological data is highly effective in improving out-of-sample prediction of brain-derived representational dissimilarity matrices from DNN embeddings, at times fleshing out isomorphisms not otherwise observable. Using pruned DNNs, image-level heatmaps can be produced to identify image sections whose features load on dimensions coded by a brain area. Pruning supervised by human brain/behavior therefore effectively identifies alignable dimensions of knowledge between DNNs and humans and constitutes an effective method for understanding the organization of knowledge in neural networks.
Collapse
Affiliation(s)
- Priya Tarigopula
- Center for Mind/Brain Sciences - CIMeC, University of Trento, Italy.
| | | | | | - Nhut Truong
- Center for Mind/Brain Sciences - CIMeC, University of Trento, Italy.
| | - Uri Hasson
- Center for Mind/Brain Sciences - CIMeC, University of Trento, Italy.
| |
Collapse
|
3
|
Wencheng W, Ge Y, Zuo Z, Chen L, Qin X, Zuxiang L. Visual number sense for real-world scenes shared by deep neural networks and humans. Heliyon 2023; 9:e18517. [PMID: 37560656 PMCID: PMC10407052 DOI: 10.1016/j.heliyon.2023.e18517] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/26/2023] [Revised: 07/17/2023] [Accepted: 07/19/2023] [Indexed: 08/11/2023] Open
Abstract
Recently, visual number sense has been identified from deep neural networks (DNNs). However, whether DNNs have the same capacity for real-world scenes, rather than the simple geometric figures that are often tested, is unclear. In this study, we explore the number perception of scenes using AlexNet and find that numerosity can be represented by the pattern of group activation of the category layer units. The global activation of these units increases with the number of objects in the scene, and the variations in their activation decrease accordingly. By decoding the numerosity from this pattern, we reveal that the embedding coefficient of a scene determines the likelihood of potential objects to contribute to numerical perception. This was demonstrated by the more optimized performance for pictures with relatively high embedding coefficients in both DNNs and humans. This study for the first time shows that a distinct feature in visual environments, revealed by DNNs, can modulate human perception, supported by a group-coding mechanism.
Collapse
Affiliation(s)
- Wu Wencheng
- AHU-IAI AI Joint Laboratory, Anhui University, Hefei, 230601, China
- Institute of Artificial Intelligence, Hefei Comprehensive National Science Center, Hefei, 230088, China
| | - Yingxi Ge
- State Key Laboratory of Brain and Cognitive Science, Institute of Biophysics, Chinese Academy of Sciences, 15 Datun Road, Beijing, 100101, China
- CAS Center for Excellence in Brain Science and Intelligence Technology, China
- University of Chinese Academy of Sciences, 19A Yuquan Road, Beijing, 100049, China
| | - Zhentao Zuo
- Institute of Artificial Intelligence, Hefei Comprehensive National Science Center, Hefei, 230088, China
- State Key Laboratory of Brain and Cognitive Science, Institute of Biophysics, Chinese Academy of Sciences, 15 Datun Road, Beijing, 100101, China
- CAS Center for Excellence in Brain Science and Intelligence Technology, China
- University of Chinese Academy of Sciences, 19A Yuquan Road, Beijing, 100049, China
| | - Lin Chen
- Institute of Artificial Intelligence, Hefei Comprehensive National Science Center, Hefei, 230088, China
- State Key Laboratory of Brain and Cognitive Science, Institute of Biophysics, Chinese Academy of Sciences, 15 Datun Road, Beijing, 100101, China
- CAS Center for Excellence in Brain Science and Intelligence Technology, China
- University of Chinese Academy of Sciences, 19A Yuquan Road, Beijing, 100049, China
| | - Xu Qin
- Key Laboratory of Intelligent Computing and Signal Processing of Ministry of Education, Hefei, 230601, China
- Anhui Provincial Key Laboratory of Multimodal Cognitive Computation, Anhui University, Hefei, 230601, China
- School of Computer Science and Technology, Anhui University, Hefei 230601, China
| | - Liu Zuxiang
- Institute of Artificial Intelligence, Hefei Comprehensive National Science Center, Hefei, 230088, China
- State Key Laboratory of Brain and Cognitive Science, Institute of Biophysics, Chinese Academy of Sciences, 15 Datun Road, Beijing, 100101, China
- CAS Center for Excellence in Brain Science and Intelligence Technology, China
- University of Chinese Academy of Sciences, 19A Yuquan Road, Beijing, 100049, China
| |
Collapse
|
4
|
Semantic Relatedness in DBpedia: A Comparative and Experimental Assessment. Inf Sci (N Y) 2022. [DOI: 10.1016/j.ins.2022.11.025] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/23/2022]
|
5
|
Huang T, Song Y, Liu J. Real-world size of objects serves as an axis of object space. Commun Biol 2022; 5:749. [PMID: 35896715 PMCID: PMC9329427 DOI: 10.1038/s42003-022-03711-3] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/06/2021] [Accepted: 07/13/2022] [Indexed: 12/02/2022] Open
Abstract
Our mind can represent various objects from physical world in an abstract and complex high-dimensional object space, with axes encoding critical features to quickly and accurately recognize objects. Among object features identified in previous neurophysiological and fMRI studies that may serve as the axes, objects’ real-world size is of particular interest because it provides not only visual information for broad conceptual distinctions between objects but also ecological information for objects’ affordance. Here we use deep convolutional neural networks (DCNNs), which enable direct manipulation of visual experience and units’ activation, to explore how objects’ real-world size is extracted to construct the axis of object space. Like the human brain, the DCNNs pre-trained for object recognition also encode objects’ size as an independent axis of the object space. Further, we find that the shape of objects, rather than retinal size, context, task demands or texture features, is critical to inferring objects’ size for both DCNNs and humans. In short, with DCNNs as a brain-like model, our study devises a paradigm supplemental to conventional approaches to explore the structure of object space, which provides computational support for empirical observations on human perceptual and neural representations of objects. Combined fMRI and deep convolutional neural network analysis suggest that an axis of object space specifically encodes objects’ real-world size based on objects’ shape.
Collapse
Affiliation(s)
- Taicheng Huang
- Department of Psychology and Tsinghua Laboratory of Brain & Intelligence, Tsinghua University, Beijing, China
| | - Yiying Song
- Beijing Key Laboratory of Applied Experimental Psychology, Faculty of Psychology, Beijing Normal University, Beijing, China.
| | - Jia Liu
- Department of Psychology and Tsinghua Laboratory of Brain & Intelligence, Tsinghua University, Beijing, China.
| |
Collapse
|
6
|
Tian F, Xie H, Song Y, Hu S, Liu J. The Face Inversion Effect in Deep Convolutional Neural Networks. Front Comput Neurosci 2022; 16:854218. [PMID: 35615057 PMCID: PMC9124772 DOI: 10.3389/fncom.2022.854218] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/13/2022] [Accepted: 03/22/2022] [Indexed: 11/13/2022] Open
Abstract
The face inversion effect (FIE) is a behavioral marker of face-specific processing that the recognition of inverted faces is disproportionately disrupted than that of inverted non-face objects. One hypothesis is that while upright faces are represented by face-specific mechanism, inverted faces are processed as objects. However, evidence from neuroimaging studies is inconclusive, possibly because the face system, such as the fusiform face area, is interacted with the object system, and therefore the observation from the face system may indirectly reflect influences from the object system. Here we examined the FIE in an artificial face system, visual geometry group network-face (VGG-Face), a deep convolutional neural network (DCNN) specialized for identifying faces. In line with neuroimaging studies on humans, a stronger FIE was found in VGG-Face than that in DCNN pretrained for processing objects. Critically, further classification error analysis revealed that in VGG-Face, inverted faces were miscategorized as objects behaviorally, and the analysis on internal representations revealed that VGG-Face represented inverted faces in a similar fashion as objects. In short, our study supported the hypothesis that inverted faces are represented as objects in a pure face system.
Collapse
Affiliation(s)
- Fang Tian
- State Key Laboratory of Cognitive Neuroscience and Learning and IDG/McGovern Institute for Brain Research, Beijing Normal University, Beijing, China
| | - Hailun Xie
- Beijing Key Laboratory of Applied Experimental Psychology, Faculty of Psychology, Beijing Normal University, Beijing, China
| | - Yiying Song
- Beijing Key Laboratory of Applied Experimental Psychology, Faculty of Psychology, Beijing Normal University, Beijing, China
- *Correspondence: Yiying Song
| | - Siyuan Hu
- Beijing Key Laboratory of Applied Experimental Psychology, Faculty of Psychology, Beijing Normal University, Beijing, China
| | - Jia Liu
- Department of Psychology & Tsinghua Laboratory of Brain and Intelligence, Tsinghua University, Beijing, China
| |
Collapse
|