1
|
Sp A. Trailblazers in Neuroscience: Using compositionality to understand how parts combine in whole objects. Eur J Neurosci 2022; 56:4378-4392. [PMID: 35760552 PMCID: PMC10084036 DOI: 10.1111/ejn.15746] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/06/2022] [Revised: 06/09/2022] [Accepted: 06/16/2022] [Indexed: 11/27/2022]
Abstract
A fundamental question for any visual system is whether its image representation can be understood in terms of its components. Decomposing any image into components is challenging because there are many possible decompositions with no common dictionary, and enumerating them leads to a combinatorial explosion. Even in perception, many objects are readily seen as containing parts, but there are many exceptions. These exceptions include objects that are not perceived as containing parts, properties like symmetry that cannot be localized to any single part, and also special categories like words and faces whose perception is widely believed to be holistic. Here, I describe a novel approach we have used to address these issues and evaluate compositionality at the behavioral and neural levels. The key design principle is to create a large number of objects by combining a small number of pre-defined components in all possible ways. This allows for building component-based models that explain whole objects using a combination of these components. Importantly, any systematic error in model fits can be used to detect the presence of emergent or holistic properties. Using this approach, we have found that whole object representations are surprisingly predictable from their components, that some components are preferred to others in perception, and that emergent properties can be discovered or explained using compositional models. Thus, compositionality is a powerful approach for understanding how whole objects relate to their parts.
Collapse
Affiliation(s)
- Arun Sp
- Centre for Neuroscience, Indian Institute of Science Bangalore
| |
Collapse
|
2
|
Katti H, Arun SP. A separable neural code in monkey IT enables perfect CAPTCHA decoding. J Neurophysiol 2022; 127:869-884. [PMID: 35196158 PMCID: PMC8957334 DOI: 10.1152/jn.00160.2021] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/22/2022] Open
Abstract
Reading distorted letters is easy for us but so challenging for the machine vision that it is used on websites as CAPTCHA (Completely Automated Public Turing Test to tell Computers and Humans Apart). How does our brain solve this problem? One solution is to have neurons selective for letter combinations but invariant to distortions. Another is for neurons to encode letter distortions and longer strings to enable separable decoding. Here, we provide evidence for the latter possibility using neural recordings in the monkey inferior temporal (IT) cortex. Neural responses to distorted strings were explained better as a product (but not sum) of shape and distortion tuning, whereas by contrast, responses to letter combinations were explained better as a sum (but not product) of letters. These two rules were sufficient for perfect CAPTCHA decoding and were also emergent in neural networks trained for word recognition. Thus, a separable neural code enables efficient letter recognition. NEW & NOTEWORTHY Many websites ask us to recognize distorted letters to deny access to malicious computer programs. Why is this task easy for our brains but hard for the computers? Here, we show that, in the monkey inferior temporal cortex, an area critical for recognition, single neurons encode distorted letter strings according to highly systematic rules that enable perfect distorted letter decoding. Remarkably, the same rules were present in neural networks trained for text recognition.
Collapse
Affiliation(s)
- Harish Katti
- Centre for Neuroscience, Indian Institute of Science, Bangalore, India
| | | |
Collapse
|
3
|
Pramod RT, Arun SP. Improving Machine Vision Using Human Perceptual Representations: The Case of Planar Reflection Symmetry for Object Classification. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 2022; 44:228-241. [PMID: 32750809 PMCID: PMC7611439 DOI: 10.1109/tpami.2020.3008107] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/15/2023]
Abstract
Achieving human-like visual abilities is a holy grail for machine vision, yet precisely how insights from human vision can improve machines has remained unclear. Here, we demonstrate two key conceptual advances: First, we show that most machine vision models are systematically different from human object perception. To do so, we collected a large dataset of perceptual distances between isolated objects in humans and asked whether these perceptual data can be predicted by many common machine vision algorithms. We found that while the best algorithms explain ∼ 70 percent of the variance in the perceptual data, all the algorithms we tested make systematic errors on several types of objects. In particular, machine algorithms underestimated distances between symmetric objects compared to human perception. Second, we show that fixing these systematic biases can lead to substantial gains in classification performance. In particular, augmenting a state-of-the-art convolutional neural network with planar/reflection symmetry scores along multiple axes produced significant improvements in classification accuracy (1-10 percent) across categories. These results show that machine vision can be improved by discovering and fixing systematic differences from human vision.
Collapse
|
4
|
Jacob G, Pramod RT, Katti H, Arun SP. Qualitative similarities and differences in visual object representations between brains and deep networks. Nat Commun 2021; 12:1872. [PMID: 33767141 PMCID: PMC7994307 DOI: 10.1038/s41467-021-22078-3] [Citation(s) in RCA: 34] [Impact Index Per Article: 11.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/20/2019] [Accepted: 02/22/2021] [Indexed: 02/06/2023] Open
Abstract
Deep neural networks have revolutionized computer vision, and their object representations across layers match coarsely with visual cortical areas in the brain. However, whether these representations exhibit qualitative patterns seen in human perception or brain representations remains unresolved. Here, we recast well-known perceptual and neural phenomena in terms of distance comparisons, and ask whether they are present in feedforward deep neural networks trained for object recognition. Some phenomena were present in randomly initialized networks, such as the global advantage effect, sparseness, and relative size. Many others were present after object recognition training, such as the Thatcher effect, mirror confusion, Weber’s law, relative size, multiple object normalization and correlated sparseness. Yet other phenomena were absent in trained networks, such as 3D shape processing, surface invariance, occlusion, natural parts and the global advantage. These findings indicate sufficient conditions for the emergence of these phenomena in brains and deep networks, and offer clues to the properties that could be incorporated to improve deep networks. Deep neural networks are widely considered as good models for biological vision. Here, we describe several qualitative similarities and differences in object representations between brains and deep networks that elucidate when deep networks can be considered good models for biological vision and how they can be improved.
Collapse
Affiliation(s)
- Georgin Jacob
- Centre for Neuroscience, Indian Institute of Science, Bangalore, India.,Department of Electrical Communication Engineering, Indian Institute of Science, Bangalore, India
| | - R T Pramod
- Centre for Neuroscience, Indian Institute of Science, Bangalore, India.,Department of Electrical Communication Engineering, Indian Institute of Science, Bangalore, India
| | - Harish Katti
- Centre for Neuroscience, Indian Institute of Science, Bangalore, India
| | - S P Arun
- Centre for Neuroscience, Indian Institute of Science, Bangalore, India.
| |
Collapse
|
5
|
Xu ZJ, Lleras A, Buetti S. Predicting how surface texture and shape combine in the human visual system to direct attention. Sci Rep 2021; 11:6170. [PMID: 33731840 PMCID: PMC7971056 DOI: 10.1038/s41598-021-85605-8] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/11/2020] [Accepted: 02/23/2021] [Indexed: 11/12/2022] Open
Abstract
Objects differ from one another along a multitude of visual features. The more distinct an object is from other objects in its surroundings, the easier it is to find it. However, it is still unknown how this distinctiveness advantage emerges in human vision. Here, we studied how visual distinctiveness signals along two feature dimensions—shape and surface texture—combine to determine the overall distinctiveness of an object in the scene. Distinctiveness scores between a target object and distractors were measured separately for shape and texture using a search task. These scores were then used to predict search times when a target differed from distractors along both shape and texture. Model comparison showed that the overall object distinctiveness was best predicted when shape and texture combined using a Euclidian metric, confirming the brain is computing independent distinctiveness scores for shape and texture and combining them to direct attention.
Collapse
Affiliation(s)
- Zoe Jing Xu
- University of Illinois, 603 E. Daniel St., Champaign, IL, 61820, USA.
| | - Alejandro Lleras
- University of Illinois, 603 E. Daniel St., Champaign, IL, 61820, USA
| | - Simona Buetti
- University of Illinois, 603 E. Daniel St., Champaign, IL, 61820, USA
| |
Collapse
|
6
|
Jacob G, Arun SP. How the forest interacts with the trees: Multiscale shape integration explains global and local processing. J Vis 2020; 20:20. [PMID: 33107916 PMCID: PMC7594584 DOI: 10.1167/jov.20.10.20] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/18/2019] [Accepted: 09/29/2020] [Indexed: 11/24/2022] Open
Abstract
Hierarchical stimuli have been widely used to study global and local processing. Two classic phenomena have been observed using these stimuli: the global advantage effect (we identify the global shape faster) and an interference effect (we identify shape slower when the global and local shapes are different). Because these phenomena have been observed during shape categorization tasks, it is unclear whether they reflect the categorical judgment or the underlying shape representation. Understanding the underlying shape representation is also critical because both global and local processing are modulated by stimulus properties. We performed two experiments to investigate these issues. In Experiment 1, we show that these phenomena can be observed in a same-different task, and that participants show systematic variation in response times across image pairs. We show that the response times to any pair of images can be accurately predicted using two factors: their dissimilarity and their distinctiveness relative to other images. In Experiment 2, we show that these phenomena can also be observed in a visual search task where participant did not have to make any categorical shape judgments. Here too, participants showed highly systematic variations in response time that could be explained as a linear sum of shape comparisons across global and local scales. Finally, the dissimilarity and distinctiveness factors estimated from the same-different task were systematically related to the search dissimilarities observed during visual search. In sum, our results show that global and local processing phenomena are properties of a systematic shape representation governed by simple rules.
Collapse
Affiliation(s)
- Georgin Jacob
- Department of Electrical Communication Engineering, Indian Institute of Science, Bangalore
| | - S P Arun
- Centre for Neuroscience, Indian Institute of Science, Bangalore
- https://sites.google.com/site/visionlabiisc/
| |
Collapse
|
7
|
Agrawal A, Hari KVS, Arun SP. A compositional neural code in high-level visual cortex can explain jumbled word reading. eLife 2020; 9:e54846. [PMID: 32369017 PMCID: PMC7272193 DOI: 10.7554/elife.54846] [Citation(s) in RCA: 11] [Impact Index Per Article: 2.8] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/02/2020] [Accepted: 05/04/2020] [Indexed: 11/13/2022] Open
Abstract
We read jubmled wrods effortlessly, but the neural correlates of this remarkable ability remain poorly understood. We hypothesized that viewing a jumbled word activates a visual representation that is compared to known words. To test this hypothesis, we devised a purely visual model in which neurons tuned to letter shape respond to longer strings in a compositional manner by linearly summing letter responses. We found that dissimilarities between letter strings in this model can explain human performance on visual search, and responses to jumbled words in word reading tasks. Brain imaging revealed that viewing a string activates this letter-based code in the lateral occipital (LO) region and that subsequent comparisons to stored words are consistent with activations of the visual word form area (VWFA). Thus, a compositional neural code potentially contributes to efficient reading.
Collapse
Affiliation(s)
- Aakash Agrawal
- Centre for BioSystems Science & Engineering, Indian Institute of ScienceBangaloreIndia
| | - KVS Hari
- Department of Electrical Communication Engineering, Indian Institute of ScienceBangaloreIndia
| | - SP Arun
- Centre for Neuroscience, Indian Institute of ScienceBangaloreIndia
| |
Collapse
|
8
|
Abstract
Reading causes widespread changes in the brain, but its effect on visual word representations is unknown. Learning to read may facilitate visual processing by forming specialized detectors for longer strings or by making word responses more predictable from single letters—that is, by increasing compositionality. We provided evidence for the latter hypothesis using experiments that compared nonoverlapping groups of readers of two Indian languages (Telugu and Malayalam). Readers showed increased single-letter discrimination and decreased letter interactions for bigrams during visual search. Importantly, these interactions predicted subjects’ overall reading fluency. In a separate brain-imaging experiment, we observed increased compositionality in readers, whereby responses to bigrams were more predictable from single letters. This effect was specific to the anterior lateral occipital region, where activations best matched behavior. Thus, learning to read facilitates visual processing by increasing the compositionality of visual word representations.
Collapse
Affiliation(s)
- Aakash Agrawal
- Centre for BioSystems Science and Engineering, Indian Institute of Science
| | - K V S Hari
- Department of Electrical Communication Engineering, Indian Institute of Science
| | - S P Arun
- Centre for Neuroscience, Indian Institute of Science
| |
Collapse
|
9
|
Pramod RT, Arun SP. Symmetric Objects Become Special in Perception Because of Generic Computations in Neurons. Psychol Sci 2017; 29:95-109. [PMID: 29219748 PMCID: PMC5772447 DOI: 10.1177/0956797617729808] [Citation(s) in RCA: 13] [Impact Index Per Article: 1.9] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/21/2022] Open
Abstract
Symmetry is a salient visual property: It is easy to detect and influences perceptual
phenomena from segmentation to recognition. Yet researchers know little about its neural
basis. Using recordings from single neurons in monkey IT cortex, we asked whether
symmetry—being an emergent property—induces nonlinear interactions between object parts.
Remarkably, we found no such deviation: Whole-object responses were always the sum of
responses to the object’s parts, regardless of symmetry. The only defining characteristic
of symmetric objects was that they were more distinctive compared with asymmetric objects.
This was a consequence of neurons preferring the same part across locations within an
object. Just as mixing diverse paints produces a homogeneous overall color, adding
heterogeneous parts within an asymmetric object renders it indistinct. In contrast, adding
identical parts within a symmetric object renders it distinct. This distinctiveness
systematically predicted human symmetry judgments, and it explains many previous
observations about symmetry perception. Thus, symmetry becomes special in perception
despite being driven by generic computations at the level of single neurons.
Collapse
Affiliation(s)
- R T Pramod
- Centre for Neuroscience and Department of Electrical Communication Engineering, Indian Institute of Science, Bangalore, India
| | - S P Arun
- Centre for Neuroscience and Department of Electrical Communication Engineering, Indian Institute of Science, Bangalore, India
| |
Collapse
|
10
|
Sunder S, Arun SP. Look before you seek: Preview adds a fixed benefit to all searches. J Vis 2016; 16:3. [PMID: 27919099 PMCID: PMC5142796 DOI: 10.1167/16.15.3] [Citation(s) in RCA: 16] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/06/2016] [Accepted: 10/11/2016] [Indexed: 11/24/2022] Open
Abstract
Knowing in advance what to look for speeds up search, but how this knowledge guides search is poorly understood. The little available evidence suggests that previewing the target produces larger reductions in search times for harder searches. To investigate this issue further, we performed seven experiments in which subjects searched for an oddball target after previewing the target, distracter, or an unrelated square. Consistent with previous studies, harder searches showed bigger reductions in search time for an informative preview. However, the same data replotted using the reciprocal of search time showed a remarkably different result: The informative preview showed a fixed additive increase in reciprocal search time across all searches regardless of difficulty. This is a nontrivial outcome because it cannot be explained using a simple relationship between search times in the informative and uninformative preview conditions. We interpret our findings by proposing that the reciprocal of search time reflects the strength of an underlying accumulating signal related to the distinctiveness or salience of the target over the distracters and that preview additively increases this signal for all searches. This in turn implies that the top-down signals related to target preview and bottom-up signals related to target-distracter salience sum linearly.
Collapse
Affiliation(s)
- Sricharan Sunder
- Centre for Neuroscience, Indian Institute of Science, Bangalore,
| | - S P Arun
- Centre for Neuroscience, Indian Institute of Science, Bangalore, ://www.cns.iisc.ac.in/~sparun
| |
Collapse
|