Task-context-dependent Linear Representation of Multiple Visual Objects in Human Parietal Cortex

Integrative processing in artificial and biological vision predicts the perceived beauty of natural images

Previous research shows that the beauty of natural images is already determined during perceptual analysis. However, it is unclear which perceptual computations give rise to the perception of beauty. Here, we tested whether perceived beauty is predicted by spatial integration across an image, a perceptual computation that reduces processing demands by aggregating image parts into more efficient representations of the whole. We quantified integrative processing in an artificial deep neural network model, where the degree of integration was determined by the amount of deviation between activations for the whole image and its constituent parts. This quantification of integration predicted beauty ratings for natural images across four studies with different stimuli and designs. In a complementary functional magnetic resonance imaging study, we show that integrative processing in human visual cortex similarly predicts perceived beauty. Together, our results establish integration as a computational principle that facilitates perceptual analysis and thereby mediates the perception of beauty.

Parietal-driven visual working memory representation in occipito-temporal cortex

Human fMRI studies have documented extensively that the content of visual working memory (VWM) can be reliably decoded from fMRI voxel response patterns during the delay period in both the occipito-temporal cortex (OTC), including early visual areas (EVC), and the posterior parietal cortex (PPC).1,2,3,4 Further work has revealed that VWM signal in OTC is largely sustained by feedback from associative areas such as prefrontal cortex (PFC) and PPC.4,5,6,7,8,9 It is unclear, however, if feedback during VWM simply restores sensory representations initially formed in OTC or if it can reshape the representational content of OTC during VWM delay. Taking advantage of a recent finding showing that object representational geometry differs between OTC and PPC in perception,10 here we find that, during VWM delay, the object representational geometry in OTC becomes more aligned with that of PPC during perception than with itself during perception. This finding supports the role of feedback in shaping the content of VWM in OTC, with the VWM content of OTC more determined by information retained in PPC than by the sensory information initially encoded in OTC.

Multiple visual objects are represented differently in the human brain and convolutional neural networks

Objects in the real world usually appear with other objects. To form object representations independent of whether or not other objects are encoded concurrently, in the primate brain, responses to an object pair are well approximated by the average responses to each constituent object shown alone. This is found at the single unit level in the slope of response amplitudes of macaque IT neurons to paired and single objects, and at the population level in fMRI voxel response patterns in human ventral object processing regions (e.g., LO). Here, we compare how the human brain and convolutional neural networks (CNNs) represent paired objects. In human LO, we show that averaging exists in both single fMRI voxels and voxel population responses. However, in the higher layers of five CNNs pretrained for object classification varying in architecture, depth and recurrent processing, slope distribution across units and, consequently, averaging at the population level both deviated significantly from the brain data. Object representations thus interact with each other in CNNs when objects are shown together and differ from when objects are shown individually. Such distortions could significantly limit CNNs' ability to generalize object representations formed in different contexts.

Comparing the Dominance of Color and Form Information across the Human Ventral Visual Pathway and Convolutional Neural Networks

Color and form information can be decoded in every region of the human ventral visual hierarchy, and at every layer of many convolutional neural networks (CNNs) trained to recognize objects, but how does the coding strength of these features vary over processing? Here, we characterize for these features both their absolute coding strength-how strongly each feature is represented independent of the other feature-and their relative coding strength-how strongly each feature is encoded relative to the other, which could constrain how well a feature can be read out by downstream regions across variation in the other feature. To quantify relative coding strength, we define a measure called the form dominance index that compares the relative influence of color and form on the representational geometry at each processing stage. We analyze brain and CNN responses to stimuli varying based on color and either a simple form feature, orientation, or a more complex form feature, curvature. We find that while the brain and CNNs largely differ in how the absolute coding strength of color and form vary over processing, comparing them in terms of their relative emphasis of these features reveals a striking similarity: For both the brain and for CNNs trained for object recognition (but not for untrained CNNs), orientation information is increasingly de-emphasized, and curvature information is increasingly emphasized, relative to color information over processing, with corresponding processing stages showing largely similar values of the form dominance index.

Category-specific and category-general neural codes of recognition memory in the ventral visual pathway

Researchers have identified category-specific brain regions, such as the fusiform face area (FFA) and parahippocampal place area (PPA) in the ventral visual pathway, which respond preferentially to one particular category of visual objects. In addition to their category-specific role in visual object identification and categorization, regions in the ventral visual pathway play critical roles in recognition memory. Nevertheless, it is not clear whether the contributions of those brain regions to recognition memory are category-specific or category-general. To address this question, the present study adopted a subsequent memory paradigm and multivariate pattern analysis (MVPA) to explore category-specific and category-general neural codes of recognition memory in the visual pathway. The results revealed that the right FFA and the bilateral PPA showed category-specific neural patterns supporting recognition memory of faces and scenes, respectively. In contrast, the lateral occipital cortex seemed to carry category-general neural codes of recognition memory. These results provide neuroimaging evidence for category-specific and category-general neural mechanisms of recognition memory in the ventral visual pathway.

Representing Multiple Visual Objects in the Human Brain and Convolutional Neural Networks

Objects in the real world often appear with other objects. To recover the identity of an object whether or not other objects are encoded concurrently, in primate object-processing regions, neural responses to an object pair have been shown to be well approximated by the average responses to each constituent object shown alone, indicating the whole is equal to the average of its parts. This is present at the single unit level in the slope of response amplitudes of macaque IT neurons to paired and single objects, and at the population level in response patterns of fMRI voxels in human ventral object processing regions (e.g., LO). Here we show that averaging exists in both single fMRI voxels and voxel population responses in human LO, with better averaging in single voxels leading to better averaging in fMRI response patterns, demonstrating a close correspondence of averaging at the fMRI unit and population levels. To understand if a similar averaging mechanism exists in convolutional neural networks (CNNs) pretrained for object classification, we examined five CNNs with varying architecture, depth and the presence/absence of recurrent processing. We observed averaging at the CNN unit level but rarely at the population level, with CNN unit response distribution in most cases did not resemble human LO or macaque IT responses. The whole is thus not equal to the average of its parts in CNNs, potentially rendering the individual objects in a pair less accessible in CNNs during visual processing than they are in the human brain.

Revisit once more the sensory storage account of visual working memory

Recent work has highlighted the role of early visual areas in visual working memory (VWM) storage and put forward a sensory storage account of VWM. Using a distractor interference paradigm, however, we previolsy showed that the contribution of early visual areas to VWM storage may not be essential. Instead, higher cortical regions such as the posterior parietal cortex may play a more significant role in VWM storage. This is consistent with reviews of other available behavioral, neuroimaging and neurophysiology results. Recently, a number of studies brought forward new evidence regarding this debate. Here I review these new pieces of evidence in detail and show that there is still no strong and definitive evidence supporting an essential role of the early visual areas in VWM storage. Instead, converging evidence suggests that early visual areas may contribute to the decision stage of a VWM task by facilitating target and probe comparison. Aside from further clarifying this debate, it is also important to note that whether or not VWM storage uses a sensory code depends on how it is defined, and that behavioral interactions between VWM and perception tasks do not necessarily support the involvement of sensory regions in VWM storage.

The Cortical Neuroanatomy Related to Specific Neuropsychological Deficits in Alzheimer's Continuum

Background and Purpose

In Alzheimer's continuum (a comprehensive of preclinical Alzheimer's disease [AD], mild cognitive impairment [MCI] due to AD, and AD dementia), cognitive dysfunctions are often related to cortical atrophy in specific brain regions. The purpose of this study was to investigate the association between anatomical pattern of cortical atrophy and specific neuropsychological deficits.

Methods

A total of 249 participants with Alzheimer's continuum (125 AD dementia, 103 MCI due to AD, and 21 preclinical AD) who were confirmed to be positive for amyloid deposits were collected from the memory disorder clinic in the department of neurology at Samsung Medical Center in Korea between September 2013 and March 2018. To analyze neuropsychological test-specific neural correlates representing the relationship between cortical atrophy measured by cortical thickness and performance in specific neuropsychological tests, a linear regression analysis was performed. Two neural correlates acquired by 2 different standardized scores in neuropsychological tests were also compared.

Results

Cortical atrophy in several specific brain regions was associated with most neuropsychological deficits, including digit span backward, naming, drawing-copying, verbal and visual recall, semantic fluency, phonemic fluency, and response inhibition. There were a few differences between 2 neural correlates obtained by different z-scores.

Conclusions

The poor performance of most neuropsychological tests is closely related to cortical thinning in specific brain areas in Alzheimer's continuum. Therefore, the brain atrophy pattern in patients with Alzheimer's continuum can be predict by an accurate analysis of neuropsychological tests in clinical practice.

Task modulation of the 2-pathway characterization of occipitotemporal and posterior parietal visual object representations

Recent studies have reported the existence of rich non-spatial visual object representations in both human and monkey posterior parietal cortex (PPC), similar to those found in occipito-temporal cortex (OTC). Despite this similarity, we recently showed that visual object representation still differ between OTC and PPC in two aspects. In one study, by manipulating whether object shape or color was task relevant, we showed that visual object representations were under greater top-down attention and task control in PPC than in OTC (Vaziri-Pashkam & Xu, 2017, J Neurosci). In another study, using a bottom-up data driven approach, we showed that there exists a large separation between PPC and OTC regions in the representational space, with OTC regions lining up hierarchically along an OTC pathway and PPC regions lining up hierarchically along an orthogonal PPC pathway (Vaziri-Pashkam & Xu, 2019, Cereb Cortex). To understand the interaction of goal-driven visual processing and the two-pathway structure in the representational space, here we performed a set of new analyses of the data from the three experiments of Vaziri-Pashkam and Xu (2017) and directly compared the two-pathway separation of OTC and PPC regions when object shapes were attended and task relevant and when they were not. We found that in all three experiments the correlation of visual object representational structure between superior IPS (a key PPC visual region) and lateral and ventral occipito-temporal regions (higher OTC visual regions) became greater when object shapes were attended than when they were not. This modified the two-pathway structure, with PPC regions moving closer to higher OTC regions and a compression of the PPC pathway towards the OTC pathway in the representational space when shapes were attended. Consistent with this observation, the correlation between neural and behavioral measures of visual representational structure was also higher in superior IPS when shapes were attended than when they were not. By comparing representational structures across experiments and tasks, we further showed that attention to object shape resulted in the formation of more similar object representations in superior IPS across experiments than between the two tasks within the same experiment despite noise and stimulus differences across the experiments. Overall, these results demonstrated that, despite the separation of the OTC and PPC pathways in the representational space, the visual representational structure of PPC is flexible and can be modulated by the task demand. This reaffirms the adaptive nature of visual processing in PPC and further distinguishes it from the more invariant nature of visual processing in OTC.

An Information-Driven 2-Pathway Characterization of Occipitotemporal and Posterior Parietal Visual Object Representations

Recent studies have demonstrated the existence of rich visual representations in both occipitotemporal cortex (OTC) and posterior parietal cortex (PPC). Using fMRI decoding and a bottom-up data-driven approach, we showed that although robust object category representations exist in both OTC and PPC, there is an information-driven 2-pathway separation among these regions in the representational space, with occipitotemporal regions arranging hierarchically along 1 pathway and posterior parietal regions along another pathway. We obtained 10 independent replications of this 2-pathway distinction, accounting for 58-81% of the total variance of the region-wise differences in visual representation. The separation of the PPC regions from higher occipitotemporal regions was not driven by a difference in tolerance to changes in low-level visual features, did not rely on the presence of special object categories, and was present whether or not object category was task relevant. Our information-driven 2-pathway structure differs from the well-known ventral-what and dorsal-where/how characterization of posterior brain regions. Here both pathways contain rich nonspatial visual representations. The separation we see likely reflects a difference in neural coding scheme used by PPC to represent visual information compared with that of OTC.

The Posterior Parietal Cortex in Adaptive Visual Processing

Although the primate posterior parietal cortex (PPC) has been largely associated with space, attention, and action-related processing, a growing number of studies have reported the direct representation of a diverse array of action-independent nonspatial visual information in the PPC during both perception and visual working memory. By describing the distinctions and the close interactions of visual representation with space, attention, and action-related processing in the PPC, here I propose that we may understand these diverse PPC functions together through the unique contribution of the PPC to adaptive visual processing and form a more integrated and structured view of the role of the PPC in vision, cognition, and action.

A Tale of Two Visual Systems: Invariant and Adaptive Visual Information Representations in the Primate Brain

Visual information processing contains two opposite needs. There is both a need to comprehend the richness of the visual world and a need to extract only pertinent visual information to guide thoughts and behavior at a given moment. I argue that these two aspects of visual processing are mediated by two complementary visual systems in the primate brain-specifically, the occipitotemporal cortex (OTC) and the posterior parietal cortex (PPC). The role of OTC in visual processing has been documented extensively by decades of neuroscience research. I review here recent evidence from human imaging and monkey neurophysiology studies to highlight the role of PPC in adaptive visual processing. I first document the diverse array of visual representations found in PPC. I then describe the adaptive nature of visual representation in PPC by contrasting visual processing in OTC and PPC and by showing that visual representations in PPC largely originate from OTC.