Scene Perception in the Human Brain

A Texture Statistics Encoding Model Reveals Hierarchical Feature Selectivity across Human Visual Cortex

Midlevel features, such as contour and texture, provide a computational link between low- and high-level visual representations. Although the nature of midlevel representations in the brain is not fully understood, past work has suggested a texture statistics model, called the P-S model (Portilla and Simoncelli, 2000), is a candidate for predicting neural responses in areas V1-V4 as well as human behavioral data. However, it is not currently known how well this model accounts for the responses of higher visual cortex to natural scene images. To examine this, we constructed single-voxel encoding models based on P-S statistics and fit the models to fMRI data from human subjects (both sexes) from the Natural Scenes Dataset (Allen et al., 2022). We demonstrate that the texture statistics encoding model can predict the held-out responses of individual voxels in early retinotopic areas and higher-level category-selective areas. The ability of the model to reliably predict signal in higher visual cortex suggests that the representation of texture statistics features is widespread throughout the brain. Furthermore, using variance partitioning analyses, we identify which features are most uniquely predictive of brain responses and show that the contributions of higher-order texture features increase from early areas to higher areas on the ventral and lateral surfaces. We also demonstrate that patterns of sensitivity to texture statistics can be used to recover broad organizational axes within visual cortex, including dimensions that capture semantic image content. These results provide a key step forward in characterizing how midlevel feature representations emerge hierarchically across the visual system.SIGNIFICANCE STATEMENT Intermediate visual features, like texture, play an important role in cortical computations and may contribute to tasks like object and scene recognition. Here, we used a texture model proposed in past work to construct encoding models that predict the responses of neural populations in human visual cortex (measured with fMRI) to natural scene stimuli. We show that responses of neural populations at multiple levels of the visual system can be predicted by this model, and that the model is able to reveal an increase in the complexity of feature representations from early retinotopic cortex to higher areas of ventral and lateral visual cortex. These results support the idea that texture-like representations may play a broad underlying role in visual processing.

Hippocampal spatial view cells for memory and navigation, and their underlying connectivity in humans

Hippocampal and parahippocampal gyrus spatial view neurons in primates respond to the spatial location being looked at. The representation is allocentric, in that the responses are to locations "out there" in the world, and are relatively invariant with respect to retinal position, eye position, head direction, and the place where the individual is located. The underlying connectivity in humans is from ventromedial visual cortical regions to the parahippocampal scene area, leading to the theory that spatial view cells are formed by combinations of overlapping feature inputs self-organized based on their closeness in space. Thus, although spatial view cells represent "where" for episodic memory and navigation, they are formed by ventral visual stream feature inputs in the parahippocampal gyrus in what is the parahippocampal scene area. A second "where" driver of spatial view cells are parietal inputs, which it is proposed provide the idiothetic update for spatial view cells, used for memory recall and navigation when the spatial view details are obscured. Inferior temporal object "what" inputs and orbitofrontal cortex reward inputs connect to the human hippocampal system, and in macaques can be associated in the hippocampus with spatial view cell "where" representations to implement episodic memory. Hippocampal spatial view cells also provide a basis for navigation to a series of viewed landmarks, with the orbitofrontal cortex reward inputs to the hippocampus providing the goals for navigation, which can then be implemented by hippocampal connectivity in humans to parietal cortex regions involved in visuomotor actions in space. The presence of foveate vision and the highly developed temporal lobe for object and scene processing in primates including humans provide a basis for hippocampal spatial view cells to be key to understanding episodic memory in the primate and human hippocampus, and the roles of this system in primate including human navigation.

The representational hierarchy in human and artificial visual systems in the presence of object-scene regularities

Human vision is still largely unexplained. Computer vision made impressive progress on this front, but it is still unclear to which extent artificial neural networks approximate human object vision at the behavioral and neural levels. Here, we investigated whether machine object vision mimics the representational hierarchy of human object vision with an experimental design that allows testing within-domain representations for animals and scenes, as well as across-domain representations reflecting their real-world contextual regularities such as animal-scene pairs that often co-occur in the visual environment. We found that DCNNs trained in object recognition acquire representations, in their late processing stage, that closely capture human conceptual judgements about the co-occurrence of animals and their typical scenes. Likewise, the DCNNs representational hierarchy shows surprising similarities with the representational transformations emerging in domain-specific ventrotemporal areas up to domain-general frontoparietal areas. Despite these remarkable similarities, the underlying information processing differs. The ability of neural networks to learn a human-like high-level conceptual representation of object-scene co-occurrence depends upon the amount of object-scene co-occurrence present in the image set thus highlighting the fundamental role of training history. Further, although mid/high-level DCNN layers represent the category division for animals and scenes as observed in VTC, its information content shows reduced domain-specific representational richness. To conclude, by testing within- and between-domain selectivity while manipulating contextual regularities we reveal unknown similarities and differences in the information processing strategies employed by human and artificial visual systems.

Angular Gyrus Responses Show Joint Statistical Dependence with Brain Regions Selective for Different Categories

Category selectivity is a fundamental principle of organization of perceptual brain regions. Human occipitotemporal cortex is subdivided into areas that respond preferentially to faces, bodies, artifacts, and scenes. However, observers need to combine information about objects from different categories to form a coherent understanding of the world. How is this multicategory information encoded in the brain? Studying the multivariate interactions between brain regions of male and female human subjects with fMRI and artificial neural networks, we found that the angular gyrus shows joint statistical dependence with multiple category-selective regions. Adjacent regions show effects for the combination of scenes and each other category, suggesting that scenes provide a context to combine information about the world. Additional analyses revealed a cortical map of areas that encode information across different subsets of categories, indicating that multicategory information is not encoded in a single centralized location, but in multiple distinct brain regions.SIGNIFICANCE STATEMENT Many cognitive tasks require combining information about entities from different categories. However, visual information about different categorical objects is processed by separate, specialized brain regions. How is the joint representation from multiple category-selective regions implemented in the brain? Using fMRI movie data and state-of-the-art multivariate statistical dependence based on artificial neural networks, we identified the angular gyrus encoding responses across face-, body-, artifact-, and scene-selective regions. Further, we showed a cortical map of areas that encode information across different subsets of categories. These findings suggest that multicategory information is not encoded in a single centralized location, but at multiple cortical sites which might contribute to distinct cognitive functions, offering insights to understand integration in a variety of domains.

Distinct neural mechanisms for heading retrieval and context recognition in the hippocampus during spatial reorientation

Reorientation, the process of regaining one's bearings after becoming lost, requires identification of a spatial context (context recognition) and recovery of heading direction within that context (heading retrieval). We previously showed that these processes rely on the use of features and geometry, respectively. Here, we examine reorientation behavior in a task that creates contextual ambiguity over a long timescale to demonstrate that mice learn to combine both featural and geometric cues to recover heading with experience. At the neural level, most CA1 neurons persistently align to geometry, and this alignment predicts heading behavior. However, a small subset of cells shows feature-sensitive place field remapping, which serves to predict context. Efficient heading retrieval and context recognition require integration of featural and geometric information in the active network through rate changes. These data illustrate how context recognition and heading retrieval are coded in CA1 and how these processes change with experience.

The human posterior parietal cortex: effective connectome, and its relation to function

The effective connectivity between 21 regions in the human posterior parietal cortex, and 360 cortical regions was measured in 171 Human Connectome Project (HCP) participants using the HCP atlas, and complemented with functional connectivity and diffusion tractography. Intraparietal areas LIP, VIP, MIP, and AIP have connectivity from early cortical visual regions, and to visuomotor regions such as the frontal eye fields, consistent with functions in eye saccades and tracking. Five superior parietal area 7 regions receive from similar areas and from the intraparietal areas, but also receive somatosensory inputs and connect with premotor areas including area 6, consistent with functions in performing actions to reach for, grasp, and manipulate objects. In the anterior inferior parietal cortex, PFop, PFt, and PFcm are mainly somatosensory, and PF in addition receives visuo-motor and visual object information, and is implicated in multimodal shape and body image representations. In the posterior inferior parietal cortex, PFm and PGs combine visuo-motor, visual object, and reward input and connect with the hippocampal system. PGi in addition provides a route to motion-related superior temporal sulcus regions involved in social interactions. PGp has connectivity with intraparietal regions involved in coordinate transforms and may be involved in idiothetic update of hippocampal visual scene representations.

A tripartite view of the posterior cingulate cortex

The posterior cingulate cortex (PCC) is one of the least understood regions of the cerebral cortex. By contrast, the anterior cingulate cortex has been the subject of intensive investigation in humans and model animal systems, leading to detailed behavioural and computational theoretical accounts of its function. The time is right for similar progress to be made in the PCC given its unique anatomical and physiological properties and demonstrably important contributions to higher cognitive functions and brain diseases. Here, we describe recent progress in understanding the PCC, with a focus on convergent findings across species and techniques that lay a foundation for establishing a formal theoretical account of its functions. Based on this converging evidence, we propose that the broader PCC region contains three major subregions - the dorsal PCC, ventral PCC and retrosplenial cortex - that respectively support the integration of executive, mnemonic and spatial processing systems. This tripartite subregional view reconciles inconsistencies in prior unitary theories of PCC function and offers promising new avenues for progress.

The human posterior cingulate, retrosplenial, and medial parietal cortex effective connectome, and implications for memory and navigation

The human posterior cingulate, retrosplenial, and medial parietal cortex are involved in memory and navigation. The functional anatomy underlying these cognitive functions was investigated by measuring the effective connectivity of these Posterior Cingulate Division (PCD) regions in the Human Connectome Project-MMP1 atlas in 171 HCP participants, and complemented with functional connectivity and diffusion tractography. First, the postero-ventral parts of the PCD (31pd, 31pv, 7m, d23ab, and v23ab) have effective connectivity with the temporal pole, inferior temporal visual cortex, cortex in the superior temporal sulcus implicated in auditory and semantic processing, with the reward-related vmPFC and pregenual anterior cingulate cortex, with the inferior parietal cortex, and with the hippocampal system. This connectivity implicates it in hippocampal episodic memory, providing routes for "what," reward and semantic schema-related information to access the hippocampus. Second, the antero-dorsal parts of the PCD (especially 31a and 23d, PCV, and also RSC) have connectivity with early visual cortical areas including those that represent spatial scenes, with the superior parietal cortex, with the pregenual anterior cingulate cortex, and with the hippocampal system. This connectivity implicates it in the "where" component for hippocampal episodic memory and for spatial navigation. The dorsal-transitional-visual (DVT) and ProStriate regions where the retrosplenial scene area is located have connectivity from early visual cortical areas to the parahippocampal scene area, providing a ventromedial route for spatial scene information to reach the hippocampus. These connectivities provide important routes for "what," reward, and "where" scene-related information for human hippocampal episodic memory and navigation. The midcingulate cortex provides a route from the anterior dorsal parts of the PCD and the supracallosal part of the anterior cingulate cortex to premotor regions.

Peripheral visual field loss and activities of daily living

PURPOSE OF REVIEW

Peripheral visual field (VF) loss affects 13% of the population over 65. Its effect on activities of daily living and higher order visual processing is as important as it is inadequately understood. The purpose of this review is to summarize available literature on the impact of peripheral vision loss on driving, reading, face recognition, scene recognition and scene navigation.

RECENT FINDINGS

In this review, glaucoma and retrochiasmal cortical damage are utilized as examples of peripheral field loss which typically spare central vision and have patterns respecting the horizontal and vertical meridians, respectively. In both glaucoma and retrochiasmal damage, peripheral field loss causes driving difficulty - especially with lane maintenance - leading to driving cessation, loss of independence, and depression. Likewise, peripheral field loss can lead to slower reading speeds and decreased enjoyment from reading, and anxiety. In glaucoma and retrochiasmal field loss, face processing is impaired which impacts social functioning. Finally, scene recognition and navigation are also adversely affected, impacting wayfinding and hazard detection leading to decreased independence as well as more frequent injury.

SUMMARY

Peripheral VF loss is an under-recognized cause of patient distress and disability. All peripheral field loss is not the same, differential patterns of loss affect parameters of activities of daily living (ADL) and visual processing in particular ways. Future research should aim to further characterize patterns of deranged ADL and visual processing, their correlation with types of field loss, and associated mechanisms.

Rethinking retrosplenial cortex: Perspectives and predictions

The last decade has produced exciting new ideas about retrosplenial cortex (RSC) and its role in integrating diverse inputs. Here, we review the diversity in forms of spatial and directional tuning of RSC activity, temporal organization of RSC activity, and features of RSC interconnectivity with other brain structures. We find that RSC anatomy and dynamics are more consistent with roles in multiple sensorimotor and cognitive processes than with any isolated function. However, two more generalized categories of function may best characterize roles for RSC in complex cognitive processes: (1) shifting and relating perspectives for spatial cognition and (2) prediction and error correction for current sensory states with internal representations of the environment. Both functions likely take advantage of RSC's capacity to encode conjunctions among sensory, motor, and spatial mapping information streams. Together, these functions provide the scaffold for intelligent actions, such as navigation, perspective taking, interaction with others, and error detection.

Reactivation of schema representation in lateral occipital cortex supports successful memory encoding

Schemas provide a scaffold onto which we can integrate new memories. Previous research has investigated the brain activity and connectivity underlying schema-related memory formation. However, how schemas are represented and reactivated in the brain, in order to enhance memory, remains unclear. To address this issue, we used an object–location spatial schema that was learned over multiple sessions, combined with similarity analyses of neural representations, to investigate the reactivation of schema representations of object–location memories when a new object–scene association is learned. In addition, we investigated how this reactivation affects subsequent memory performance under different strengths of schemas. We found that reactivation of a schema representation in the lateral occipital cortex (LOC) during object–scene encoding affected subsequent associative memory performance only in the schema-consistent condition and increased the functional connectivity between the LOC and the parahippocampal place area. Taken together, our findings provide new insight into how schema acts as a scaffold to support the integration of novel information into existing cortical networks and suggest a neural basis for schema-induced rapid cortical learning.

Environment Symmetry Drives a Multidirectional Code in Rat Retrosplenial Cortex

We investigated how environment symmetry shapes the neural processing of direction by recording directionally tuned retrosplenial neurons in male Lister hooded rats exploring multicompartment environments that had different levels of global rotational symmetry. Our hypothesis built on prior observations of twofold symmetry in the directional tuning curves of rats in a globally twofold-symmetric environment. To test whether environment symmetry was the relevant factor shaping the directional responses, here we deployed the same apparatus (two connected rectangular boxes) plus one with fourfold symmetry (a 2 × 2 array of connected square boxes) and one with onefold symmetry (a circular open-field arena). Consistent with our hypothesis we found many neurons with tuning curve symmetries that mirrored these environment symmetries, having twofold, fourfold, or onefold symmetric tuning, respectively. Some cells expressed this pattern only globally (across the whole environment), maintaining singular tuning curves in each subcompartment. However, others also expressed it locally within each subcompartment. Because multidirectionality has not been reported in naive rats in single environmental compartments, this suggests an experience-dependent effect of global environment symmetry on local firing symmetry. An intermingled population of directional neurons were classic head direction cells with globally referenced directional tuning. These cells were electrophysiologically distinct, with narrower tuning curves and a burstier firing pattern. Thus, retrosplenial directional neurons can simultaneously encode overall head direction and local head direction (relative to compartment layout). Furthermore, they can learn about global environment symmetry and express this locally. This may be important for the encoding of environment structure beyond immediate perceptual reach.SIGNIFICANCE STATEMENT We investigated how environment symmetry shapes the neural code for space by recording directionally tuned neurons from the retrosplenial cortex of rats exploring single- or multicompartment environments having onefold, twofold, or fourfold rotational symmetry. We found that many cells expressed a symmetry in their head direction tuning curves that matched the corresponding global environment symmetry, indicating plasticity of their directional tuning. They were also electrophysiologically distinct from canonical head directional cells. Notably, following exploration of the global space, many multidirectionally tuned neurons encoded global environment symmetry, even in local subcompartments. Our results suggest that multidirectional head direction codes contribute to the cognitive mapping of the complex structure of multicompartmented spaces.

Tackling the challenges of group network inference from intracranial EEG data

INTRODUCTION

Intracranial EEG (iEEG) data is a powerful way to map brain function, characterized by high temporal and spatial resolution, allowing the study of interactions among neuronal populations that orchestrate cognitive processing. However, the statistical inference and analysis of brain networks using iEEG data faces many challenges related to its sparse brain coverage, and its inhomogeneity across patients.

METHODS

We review these challenges and develop a methodological pipeline for estimation of network structure not obtainable from any single patient, illustrated on the inference of the interaction among visual streams using a dataset of 27 human iEEG recordings from a visual experiment employing visual scene stimuli. 100 ms sliding window and multiple band-pass filtered signals are used to provide temporal and spectral resolution. For the connectivity analysis we showcase two connectivity measures reflecting different types of interaction between regions of interest (ROI): Phase Locking Value as a symmetric measure of synchrony, and Directed Transfer Function-asymmetric measure describing causal interaction. For each two channels, initial uncorrected significance testing at p < 0.05 for every time-frequency point is carried out by comparison of the data-derived connectivity to a baseline surrogate-based null distribution, providing a binary time-frequency connectivity map. For each ROI pair, a connectivity density map is obtained by averaging across all pairs of channels spanning them, effectively agglomerating data across relevant channels and subjects. Finally, the difference of the mean map value after and before the stimulation is compared to the same statistic in surrogate data to assess link significance.

RESULTS

The analysis confirmed the function of the parieto-medial temporal pathway, mediating visuospatial information between dorsal and ventral visual streams during visual scene analysis. Moreover, we observed the anterior hippocampal connectivity with more posterior areas in the medial temporal lobe, and found the reciprocal information flow between early processing areas and medial place area.

DISCUSSION

To summarize, we developed an approach for estimating network connectivity, dealing with the challenge of sparse individual coverage of intracranial EEG electrodes. Its application provided new insights into the interaction between the dorsal and ventral visual streams, one of the iconic dualities in human cognition.

Rapid encoding of task regularities in the human hippocampus guides sensorimotor timing

The brain encodes the statistical regularities of the environment in a task-specific yet flexible and generalizable format. Here, we seek to understand this process by bridging two parallel lines of research, one centered on sensorimotor timing, and the other on cognitive mapping in the hippocampal system. By combining functional magnetic resonance imaging (fMRI) with a fast-paced time-to-contact (TTC) estimation task, we found that the hippocampus signaled behavioral feedback received in each trial as well as performance improvements across trials along with reward-processing regions. Critically, it signaled performance improvements independent from the tested intervals, and its activity accounted for the trial-wise regression-to-the-mean biases in TTC estimation. This is in line with the idea that the hippocampus supports the rapid encoding of temporal context even on short time scales in a behavior-dependent manner. Our results emphasize the central role of the hippocampus in statistical learning and position it at the core of a brain-wide network updating sensorimotor representations in real time for flexible behavior.

Scene-selective brain regions respond to embedded objects of a scene

Objects are fundamental to scene understanding. Scenes are defined by embedded objects and how we interact with them. Paradoxically, scene processing in the brain is typically discussed in contrast to object processing. Using the BOLD5000 dataset (Chang et al., 2019), we examined whether objects within a scene predicted the neural representation of scenes, as measured by functional magnetic resonance imaging in humans. Stimuli included 1,179 unique scenes across 18 semantic categories. Object composition of scenes were compared across scene exemplars in different semantic scene categories, and separately, in exemplars of the same scene category. Neural representations in scene- and object-preferring brain regions were significantly related to which objects were in a scene, with the effect at times stronger in the scene-preferring regions. The object model accounted for more variance when comparing scenes within the same semantic category to scenes from different categories. Here, we demonstrate the function of scene-preferring regions includes the processing of objects. This suggests visual processing regions may be better characterized by the processes, which are engaged when interacting with the stimulus kind, such as processing groups of objects in scenes, or processing a single object in our foreground, rather than the stimulus kind itself.

Ramp-shaped neural tuning supports graded population-level representation of the object-to-scene continuum

We can easily perceive the spatial scale depicted in a picture, regardless of whether it is a small space (e.g., a close-up view of a chair) or a much larger space (e.g., an entire class room). How does the human visual system encode this continuous dimension? Here, we investigated the underlying neural coding of depicted spatial scale, by examining the voxel tuning and topographic organization of brain responses. We created naturalistic yet carefully-controlled stimuli by constructing virtual indoor environments, and rendered a series of snapshots to smoothly sample between a close-up view of the central object and far-scale view of the full environment (object-to-scene continuum). Human brain responses were measured to each position using functional magnetic resonance imaging. We did not find evidence for a smooth topographic mapping for the object-to-scene continuum on the cortex. Instead, we observed large swaths of cortex with opposing ramp-shaped profiles, with highest responses to one end of the object-to-scene continuum or the other, and a small region showing a weak tuning to intermediate scale views. However, when we considered the population code of the entire ventral occipito-temporal cortex, we found smooth and linear representation of the object-to-scene continuum. Our results together suggest that depicted spatial scale information is encoded parametrically in large-scale population codes across the entire ventral occipito-temporal cortex.

Neural substrates of spatial processing and navigation in blindness: An activation likelihood estimation meta-analysis

Even though vision is considered the best suited sensory modality to acquire spatial information, blind individuals can form spatial representations to navigate and orient themselves efficiently in space. Consequently, many studies support the amodality hypothesis of spatial representations since sensory modalities other than vision contribute to the formation of spatial representations, independently of visual experience and imagery. However, given the high variability in abilities and deficits observed in blind populations, a clear consensus about the neural representations of space has yet to be established. To this end, we performed a meta-analysis of the literature on the neural correlates of spatial processing and navigation via sensory modalities other than vision, like touch and audition, in individuals with early and late onset blindness. An activation likelihood estimation (ALE) analysis of the neuroimaging literature revealed that early blind individuals and sighted controls activate the same neural networks in the processing of non-visual spatial information and navigation, including the posterior parietal cortex, frontal eye fields, insula, and the hippocampal complex. Furthermore, blind individuals also recruit primary and associative occipital areas involved in visuo-spatial processing via cross-modal plasticity mechanisms. The scarcity of studies involving late blind individuals did not allow us to establish a clear consensus about the neural substrates of spatial representations in this specific population. In conclusion, the results of our analysis on neuroimaging studies involving early blind individuals support the amodality hypothesis of spatial representations.

The hippocampus, ventromedial prefrontal cortex, and episodic and semantic memory

The human ventromedial prefrontal cortex (vmPFC)/anterior cingulate cortex is implicated in reward and emotion, but also in memory. It is shown how the human orbitofrontal cortex connecting with the vmPFC and anterior cingulate cortex provide a route to the hippocampus for reward and emotional value to be incorporated into episodic memory, enabling memory of where a reward was seen. It is proposed that this value component results in primarily episodic memories with some value component to be repeatedly recalled from the hippocampus so that they are more likely to become incorporated into neocortical semantic and autobiographical memories. The same orbitofrontal and anterior cingulate regions also connect in humans to the septal and basal forebrain cholinergic nuclei, thereby helping to consolidate memory, and helping to account for why damage to the vMPFC impairs memory. The human hippocampus and vmPFC thus contribute in complementary ways to forming episodic and semantic memories.

Multiple cortical visual streams in humans

The effective connectivity between 55 visual cortical regions and 360 cortical regions was measured in 171 HCP participants using the HCP-MMP atlas, and complemented with functional connectivity and diffusion tractography. A Ventrolateral Visual "What" Stream for object and face recognition projects hierarchically to the inferior temporal visual cortex, which projects to the orbitofrontal cortex for reward value and emotion, and to the hippocampal memory system. A Ventromedial Visual "Where" Stream for scene representations connects to the parahippocampal gyrus and hippocampus. An Inferior STS (superior temporal sulcus) cortex Semantic Stream receives from the Ventrolateral Visual Stream, from visual inferior parietal PGi, and from the ventromedial-prefrontal reward system and connects to language systems. A Dorsal Visual Stream connects via V2 and V3A to MT+ Complex regions (including MT and MST), which connect to intraparietal regions (including LIP, VIP and MIP) involved in visual motion and actions in space. It performs coordinate transforms for idiothetic update of Ventromedial Stream scene representations. A Superior STS cortex Semantic Stream receives visual inputs from the Inferior STS Visual Stream, PGi, and STV, and auditory inputs from A5, is activated by face expression, motion and vocalization, and is important in social behaviour, and connects to language systems.

Developmental changes in story-evoked responses in the neocortex and hippocampus

How does the representation of naturalistic life events change with age? Here, we analyzed fMRI data from 414 children and adolescents (5-19 years) as they watched a narrative movie. In addition to changes in the degree of inter-subject correlation (ISC) with age in sensory and medial parietal regions, we used a novel measure (between-group ISC) to reveal age-related shifts in the responses across the majority of the neocortex. Over the course of development, brain responses became more discretized into stable and coherent events and shifted earlier in time to anticipate upcoming perceived event transitions, measured behaviorally in an age-matched sample. However, hippocampal responses to event boundaries actually decreased with age, suggesting a shifting division of labor between episodic encoding processes and schematic event representations between the ages of 5 and 19.

The contribution of object identity and configuration to scene representation in convolutional neural networks

Scene perception involves extracting the identities of the objects comprising a scene in conjunction with their configuration (the spatial layout of the objects in the scene). How object identity and configuration information is weighted during scene processing and how this weighting evolves over the course of scene processing however, is not fully understood. Recent developments in convolutional neural networks (CNNs) have demonstrated their aptitude at scene processing tasks and identified correlations between processing in CNNs and in the human brain. Here we examined four CNN architectures (Alexnet, Resnet18, Resnet50, Densenet161) and their sensitivity to changes in object and configuration information over the course of scene processing. Despite differences among the four CNN architectures, across all CNNs, we observed a common pattern in the CNN's response to object identity and configuration changes. Each CNN demonstrated greater sensitivity to configuration changes in early stages of processing and stronger sensitivity to object identity changes in later stages. This pattern persists regardless of the spatial structure present in the image background, the accuracy of the CNN in classifying the scene, and even the task used to train the CNN. Importantly, CNNs' sensitivity to a configuration change is not the same as their sensitivity to any type of position change, such as that induced by a uniform translation of the objects without a configuration change. These results provide one of the first documentations of how object identity and configuration information are weighted in CNNs during scene processing.

Differential Abnormality in Functional Connectivity Density in Preclinical and Early-Stage Alzheimer's Disease

Background

Both subjective cognitive decline (SCD) and amnestic mild cognitive impairment (aMCI) have a high risk of progression to Alzheimer's disease (AD). While most of the available evidence described changes in functional connectivity (FC) in SCD and aMCI, there was no confirmation of changes in functional connectivity density (FCD) that have not been confirmed. Therefore, the purpose of this study was to investigate the specific alterations in resting-state FCD in SCD and aMCI and further assess the extent to which these changes can distinguish the preclinical and early-stage AD.

Methods

A total of 57 patients with SCD, 59 patients with aMCI, and 78 healthy controls (HC) were included. The global FCD, local FCD, and long-range FCD were calculated for each voxel to identify brain regions with significant FCD alterations. The brain regions with abnormal FCD were then used as regions of interest for FC analysis. In addition, we calculated correlations between neuroimaging alterations and cognitive function and performed receiver-operating characteristic analyses to assess the diagnostic effect of the FCD and FC alterations on SCD and aMCI.

Results

FCD mapping revealed significantly increased global FCD in the left parahippocampal gyrus (PHG.L) and increased long-range FCD in the left hippocampus for patients with SCD when compared to HCs. However, when compared to SCD, patients with aMCI showed significantly decreased global FCD and long-range FCD in the PHG.L. The follow-up FC analysis further revealed significant variations between the PHG.L and the occipital lobe in patients with SCD and aMCI. In addition, patients with SCD also presented significant changes in FC between the left hippocampus, the left cerebellum anterior lobe, and the inferior temporal gyrus. Moreover, changes in abnormal indicators in the SCD and aMCI groups were significantly associated with cognitive function. Finally, combining FCD and FC abnormalities allowed for a more precise differentiation of the clinical stages.

Conclusion

To our knowledge, this study is the first to investigate specific alterations in FCD and FC for both patients with SCD and aMCI and confirms differential abnormalities that can serve as potential imaging markers for preclinical and early-stage Alzheimer's disease (AD). Also, it adds a new dimension of understanding to the diagnosis of SCD and aMCI as well as the evaluation of disease progression.

The retrocalcarine sulcus maps different retinotopic representations in macaques and humans

Primate cerebral cortex is highly convoluted with much of the cortical surface buried in sulcal folds. The origins of cortical folding and its functional relevance have been a major focus of systems and cognitive neuroscience, especially when considering stereotyped patterns of cortical folding that are shared across individuals within a primate species and across multiple species. However, foundational questions regarding organizing principles shared across species remain unanswered. Taking a cross-species comparative approach with a careful consideration of historical observations, we investigate cortical folding relative to primary visual cortex (area V1). We identify two macroanatomical structures-the retrocalcarine and external calcarine sulci-in 24 humans and 6 macaque monkeys. We show that within species, these sulci are identifiable in all individuals, fall on a similar part of the V1 retinotopic map, and thus, serve as anatomical landmarks predictive of functional organization. Yet, across species, the underlying eccentricity representations corresponding to these macroanatomical structures differ strikingly across humans and macaques. Thus, the correspondence between retinotopic representation and cortical folding for an evolutionarily old structure like V1 is species-specific and suggests potential differences in developmental and experiential constraints across primates.

Texture-like representation of objects in human visual cortex

Humans are exquisitely sensitive to the spatial arrangement of visual features in objects and scenes, but not in visual textures. Category-selective regions in the visual cortex are widely believed to underlie object perception, suggesting such regions should distinguish natural images of objects from synthesized images containing similar visual features in scrambled arrangements. Contrarily, we demonstrate that representations in category-selective cortex do not discriminate natural images from feature-matched scrambles but can discriminate images of different categories, suggesting a texture-like encoding. We find similar insensitivity to feature arrangement in Imagenet-trained deep convolutional neural networks. This suggests the need to reconceptualize the role of category-selective cortex as representing a basis set of complex texture-like features, useful for a myriad of behaviors.

The human visual ability to recognize objects and scenes is widely thought to rely on representations in category-selective regions of the visual cortex. These representations could support object vision by specifically representing objects, or, more simply, by representing complex visual features regardless of the particular spatial arrangement needed to constitute real-world objects, that is, by representing visual textures. To discriminate between these hypotheses, we leveraged an image synthesis approach that, unlike previous methods, provides independent control over the complexity and spatial arrangement of visual features. We found that human observers could easily detect a natural object among synthetic images with similar complex features that were spatially scrambled. However, observer models built from BOLD responses from category-selective regions, as well as a model of macaque inferotemporal cortex and Imagenet-trained deep convolutional neural networks, were all unable to identify the real object. This inability was not due to a lack of signal to noise, as all observer models could predict human performance in image categorization tasks. How then might these texture-like representations in category-selective regions support object perception? An image-specific readout from category-selective cortex yielded a representation that was more selective for natural feature arrangement, showing that the information necessary for natural object discrimination is available. Thus, our results suggest that the role of the human category-selective visual cortex is not to explicitly encode objects but rather to provide a basis set of texture-like features that can be infinitely reconfigured to flexibly learn and identify new object categories.

Processing of visual and non-visual naturalistic spatial information in the "parahippocampal place area"

The "parahippocampal place area" (PPA) in the human ventral visual stream exhibits increased hemodynamic activity correlated with the perception of landscape photos compared to faces or objects. Here, we investigate the perception of scene-related, spatial information embedded in two naturalistic stimuli. The same 14 participants were watching a Hollywood movie and listening to its audio-description as part of the open-data resource studyforrest.org. We model hemodynamic activity based on annotations of selected stimulus features, and compare results to a block-design visual localizer. On a group level, increased activation correlating with visual spatial information occurring in the movie is overlapping with a traditionally localized PPA. Activation correlating with semantic spatial information occurring in the audio-description is more restricted to the anterior PPA. On an individual level, we find significant bilateral activity in the PPA of nine individuals and unilateral activity in one individual. Results suggest that activation in the PPA generalizes to spatial information embedded in a movie and an auditory narrative, and may call for considering a functional subdivision of the PPA.

Individual differences in mental imagery modulate effective connectivity of scene-selective regions during resting state

Successful navigation relies on the ability to identify, perceive, and correctly process the spatial structure of a scene. It is well known that visual mental imagery plays a crucial role in navigation. Indeed, cortical regions encoding navigationally relevant information are also active during mental imagery of navigational scenes. However, it remains unknown whether their intrinsic activity and connectivity reflect the individuals' ability to imagine a scene. Here, we primarily investigated the intrinsic causal interactions among scene-selective brain regions such as Parahipoccampal Place Area (PPA), Retrosplenial Complex, and Occipital Place Area (OPA) using Dynamic Causal Modelling for resting-state functional magnetic resonance data. Second, we tested whether resting-state effective connectivity parameters among scene-selective regions could reflect individual differences in mental imagery in our sample, as assessed by the self-reported Vividness of Visual Imagery Questionnaire. We found an inhibitory influence of occipito-medial on temporal regions, and an excitatory influence of more anterior on more medial and posterior brain regions. Moreover, we found that a key role in imagery is played by the connection strength from OPA to PPA, especially in the left hemisphere, since the influence of the signal between these scene-selective regions positively correlated with good mental imagery ability. Our investigation contributes to the understanding of the complexity of the causal interaction among brain regions involved in navigation and provides new insight in understanding how an essential ability, such as mental imagery, can be explained by the intrinsic fluctuation of brain signal.

Activity in perirhinal and entorhinal cortex predicts perceived visual similarities among category exemplars with highest precision

Vision neuroscience has made great strides in understanding the hierarchical organization of object representations along the ventral visual stream (VVS). How VVS representations capture fine-grained visual similarities between objects that observers subjectively perceive has received limited examination so far. In the current study, we addressed this question by focussing on perceived visual similarities among subordinate exemplars of real-world categories. We hypothesized that these perceived similarities are reflected with highest fidelity in neural activity patterns downstream from inferotemporal regions, namely in perirhinal (PrC) and anterolateral entorhinal cortex (alErC) in the medial temporal lobe. To address this issue with functional magnetic resonance imaging (fMRI), we administered a modified 1-back task that required discrimination between category exemplars as well as categorization. Further, we obtained observer-specific ratings of perceived visual similarities, which predicted behavioural discrimination performance during scanning. As anticipated, we found that activity patterns in PrC and alErC predicted the structure of perceived visual similarity relationships among category exemplars, including its observer-specific component, with higher precision than any other VVS region. Our findings provide new evidence that subjective aspects of object perception that rely on fine-grained visual differentiation are reflected with highest fidelity in the medial temporal lobe.

Three cortical scene systems and their development

Since the discovery of three scene-selective regions in the human brain, a central assumption has been that all three regions directly support navigation. We propose instead that cortical scene processing regions support three distinct computational goals (and one not for navigation at all): (i) The parahippocampal place area supports scene categorization, which involves recognizing the kind of place we are in; (ii) the occipital place area supports visually guided navigation, which involves finding our way through the immediately visible environment, avoiding boundaries and obstacles; and (iii) the retrosplenial complex supports map-based navigation, which involves finding our way from a specific place to some distant, out-of-sight place. We further hypothesize that these systems develop along different timelines, with both navigation systems developing slower than the scene categorization system.

Dataset of human intracranial recordings during famous landmark identification

For most people, recalling information about familiar items in a visual scene is an effortless task, but it is one that depends on coordinated interactions of multiple, distributed neural components. We leveraged the high spatiotemporal resolution of direct intracranial recordings to better delineate the network dynamics underpinning visual scene recognition. We present a dataset of recordings from a large cohort of humans while they identified images of famous landmarks (50 individuals, 52 recording sessions, 6,775 electrodes, 6,541 trials). This dataset contains local field potential recordings derived from subdural and penetrating electrodes covering broad areas of cortex across both hemispheres. We provide this pre-processed data with behavioural metrics (correct/incorrect, response times) and electrode localisation in a population-normalised cortical surface space. This rich dataset will allow further investigation into the spatiotemporal progression of multiple neural processes underlying visual processing, scene recognition and cued memory recall.

The limbic memory circuit and the neural basis of contextual memory

The hippocampus, retrosplenial cortex and anterior thalamus are key components of a neural circuit known to be involved in a variety of memory functions, including spatial, contextual and episodic memory. In this review, we focus on the role of this circuit in contextual memory processes. The background environment, or context, is a powerful cue for memory retrieval, and neural representations of the context provide a mechanism for efficiently retrieving relevant memories while avoiding interference from memories that belong to other contexts. Data from experimental lesions and neural manipulation techniques indicate that each of these regions is critical for contextual memory. Neurophysiological evidence from the hippocampus and retrosplenial cortex suggest that contextual information is represented within this circuit by population-level neural firing patterns that reliably differentiate each context a subject encounters. These findings indicate that encoding contextual information to support context-dependent memory retrieval is a key function of this circuit.

Extensive cortical functional connectivity of the human hippocampal memory system

The cortical connections of the human hippocampal memory system are fundamental to understanding its operation in health and disease, especially in the context of the great development of the human cortex. The functional connectivity of the human hippocampal system was analyzed in 172 participants imaged at 7T in the Human Connectome Project. The human hippocampus has high functional connectivity not only with the entorhinal cortex, but also with areas that are more distant in the ventral 'what' stream including the perirhinal cortex and temporal cortical visual areas. Parahippocampal gyrus TF in humans has connectivity with this ventral 'what' subsystem. Correspondingly for the dorsal stream, the hippocampus has high functional connectivity not only with the presubiculum, but also with areas more distant, the medial parahippocampal cortex TH which includes the parahippocampal place or scene area, the posterior cingulate including retrosplenial cortex, and the parietal cortex. Further, there is considerable cross connectivity between the ventral and dorsal streams with the hippocampus. The findings are supported by anatomical connections, which together provide an unprecedented and quantitative overview of the extensive cortical connectivity of the human hippocampal system that goes beyond hierarchically organised and segregated pathways connecting the hippocampus and neocortex, and leads to new concepts on the operation of the hippocampal memory system in humans.

Visuospatial coding as ubiquitous scaffolding for human cognition

For more than 100 years we have known that the visual field is mapped onto the surface of visual cortex, imposing an inherently spatial reference frame on visual information processing. Recent studies highlight visuospatial coding not only throughout visual cortex, but also brain areas not typically considered visual. Such widespread access to visuospatial coding raises important questions about its role in wider cognitive functioning. Here, we synthesise these recent developments and propose that visuospatial coding scaffolds human cognition by providing a reference frame through which neural computations interface with environmental statistics and task demands via perception-action loops.

One object, two networks? Assessing the relationship between the face and body-selective regions in the primate visual system

Faces and bodies are often treated as distinct categories that are processed separately by face- and body-selective brain regions in the primate visual system. These regions occupy distinct regions of visual cortex and are often thought to constitute independent functional networks. Yet faces and bodies are part of the same object and their presence inevitably covary in naturalistic settings. Here, we re-evaluate both the evidence supporting the independent processing of faces and bodies and the organizational principles that have been invoked to explain this distinction. We outline four hypotheses ranging from completely separate networks to a single network supporting the perception of whole people or animals. The current evidence, especially in humans, is compatible with all of these hypotheses, making it presently unclear how the representation of faces and bodies is organized in the cortex.

The role of contextual materials in object recognition

While scene context is known to facilitate object recognition, little is known about which contextual "ingredients" are at the heart of this phenomenon. Here, we address the question of whether the materials that frequently occur in scenes (e.g., tiles in a bathroom) associated with specific objects (e.g., a perfume) are relevant for the processing of that object. To this end, we presented photographs of consistent and inconsistent objects (e.g., perfume vs. pinecone) superimposed on scenes (e.g., a bathroom) and close-ups of materials (e.g., tiles). In Experiment 1, consistent objects on scenes were named more accurately than inconsistent ones, while there was only a marginal consistency effect for objects on materials. Also, we did not find any consistency effect for scrambled materials that served as color control condition. In Experiment 2, we recorded event-related potentials and found N300/N400 responses-markers of semantic violations-for objects on inconsistent relative to consistent scenes. Critically, objects on materials triggered N300/N400 responses of similar magnitudes. Our findings show that contextual materials indeed affect object processing-even in the absence of spatial scene structure and object content-suggesting that material is one of the contextual "ingredients" driving scene context effects.

The human brain uses spatial schemas to represent segmented environments

Humans and animals use cognitive maps to represent the spatial structure of the environment. Although these maps are typically conceptualized as extending in an equipotential manner across known space, psychological evidence suggests that people mentally segment complex environments into subspaces. To understand the neurocognitive mechanisms behind this operation, we familiarized participants with a virtual courtyard that was divided into two halves by a river; we then used behavioral testing and fMRI to understand how spatial locations were encoded within this environment. Participants' spatial judgments and multivoxel activation patterns were affected by the division of the courtyard, indicating that the presence of a boundary can induce mental segmentation even when all parts of the environment are co-visible. In the hippocampus and occipital place area (OPA), the segmented organization of the environment manifested in schematic spatial codes that represented geometrically equivalent locations in the two subspaces as similar. In the retrosplenial complex (RSC), responses were more consistent with an integrated spatial map. These results demonstrate that people use both local spatial schemas and integrated spatial maps to represent segmented environment. We hypothesize that schematization may serve as a general mechanism for organizing complex knowledge structures in terms of their component elements.

Two 'what' pathways for action and object recognition

The ventral visual stream is conceived as a pathway for object recognition. However, we also recognize the actions an object can be involved in. Here, we show that action recognition critically depends on a pathway in lateral occipitotemporal cortex, partially overlapping and topographically aligned with object representations that are precursors for action recognition. By contrast, object features that are more relevant for object recognition, such as color and texture, are typically found in ventral occipitotemporal cortex. We argue that occipitotemporal cortex contains similarly organized lateral and ventral 'what' pathways for action and object recognition, respectively. This account explains a number of observed phenomena, such as the duplication of object domains and the specific representational profiles in lateral and ventral cortex.

Parts and Wholes in Scene Processing

During natural vision, our brains are constantly exposed to complex, but regularly structured environments. Real-world scenes are defined by typical part-whole relationships, where the meaning of the whole scene emerges from configurations of localized information present in individual parts of the scene. Such typical part-whole relationships suggest that information from individual scene parts is not processed independently, but that there are mutual influences between the parts and the whole during scene analysis. Here, we review recent research that used a straightforward, but effective approach to study such mutual influences: By dissecting scenes into multiple arbitrary pieces, these studies provide new insights into how the processing of whole scenes is shaped by their constituent parts and, conversely, how the processing of individual parts is determined by their role within the whole scene. We highlight three facets of this research: First, we discuss studies demonstrating that the spatial configuration of multiple scene parts has a profound impact on the neural processing of the whole scene. Second, we review work showing that cortical responses to individual scene parts are shaped by the context in which these parts typically appear within the environment. Third, we discuss studies demonstrating that missing scene parts are interpolated from the surrounding scene context. Bridging these findings, we argue that efficient scene processing relies on an active use of the scene's part-whole structure, where the visual brain matches scene inputs with internal models of what the world should look like.

"Scene" from inside: The representation of Observer's space in high-level visual cortex

Human observers are remarkably adept at perceiving and interacting with visual stimuli around them. Compared to visual stimuli like objects or faces, scenes are unique in that they provide enclosures for observers. An observer looks at a scene by being physically inside the scene. The current research explored this unique observer-scene relationship by studying the neural representation of scenes' spatial boundaries. Previous studies hypothesized that scenes' boundaries were processed in sets of high-level visual cortices. Notably, the parahippocampal place area (PPA), exhibited neural sensitivity to scenes that had closed vs. open spatial boundaries (Kravitz et al., 2011; Park et al., 2011). We asked whether this sensitivity reflected the openness of landscape (e.g., forest vs. beach), or the openness of the environment immediately surrounding the observer (i.e., whether a scene was viewed from inside vs. outside a room). Across two human fMRI experiments, we found that the PPA, as well as another well-known navigation-processing area, the occipital place area (OPA), processed scenes' boundaries according to the observer's space rather than the landscape. Moreover, we found that the PPA's activation pattern was susceptible to manipulations involving mid-level perceptual properties of scenes (e.g., rectilinear pattern of window frames), while the OPA's response was not. Our results have important implications for research in visual scene processing and suggest an important role of an observer's location in representing the spatial boundary, beyond the low-level visual input of a landscape.

The Ties that Bind: Agnosia, Neglect and Selective Attention to Visual Scale

PURPOSE OF REVIEW

Historical and contemporary treatments of visual agnosia and neglect regard these disorders as largely unrelated. It is thought that damage to different neural processes leads directly to one or the other condition, yet apperceptive variants of agnosia and object-centered variants of neglect share remarkably similar deficits in the quality of conscious experience. Here we argue for a closer association between "apperceptive" variants of visual agnosia and "object-centered" variants of visual neglect. We introduce a theoretical framework for understanding these conditions based on "scale attention", which refers to selecting boundary and surface information at different levels of the structural hierarchy in the visual array.

RECENT FINDINGS

We review work on visual agnosia, the cortical structures and cortico-cortical pathways that underlie visual perception, visuospatial neglect and object-centered neglect, and attention to scale. We highlight direct and indirect pathways involved in these disorders and in attention to scale. The direct pathway involves the posterior vertical segments of the superior longitudinal fasciculus that are positioned to link the established dorsal and ventral attentional centers in the parietal cortex with structures in the inferior occipitotemporal cortex associated with visual apperceptive agnosia. The connections in the right hemisphere appear to be more important for visual conscious experience, whereas those in the left hemisphere appear to be more strongly associated with the planning and execution of visually guided grasps directed at multi-part objects such as tools. In the latter case, semantic and functional information must drive the selection of the appropriate hand posture and grasp points on the object. This view is supported by studies of grasping in patients with agnosia and in patients with neglect that show that the selection of grasp points when picking up a tool involves both scale attention and semantic contributions from inferotemporal cortex. The indirect pathways, which include the inferior fronto-occipital and horizontal components of the superior longitudinal fasciculi, involve the frontal lobe, working memory and the "multiple demands" network, which can shape the content of visual awareness through the maintenance of goal- and task-based abstractions and their influence on scale attention. Recent studies of human cortico-cortical pathways necessitate revisions to long-standing theoretical views on visual perception, visually guided action and their integrations. We highlight findings from a broad sample of seemingly disparate areas of research to support the proposal that attention to scale is necessary for typical conscious visual experience and for goal-directed actions that depend on functional and semantic information. Furthermore, we suggest that vertical pathways between the parietal and occipitotemporal cortex, along with indirect pathways that involve the premotor and prefrontal cortex, facilitate the operations of scale attention.

Computational models of category-selective brain regions enable high-throughput tests of selectivity

Cortical regions apparently selective to faces, places, and bodies have provided important evidence for domain-specific theories of human cognition, development, and evolution. But claims of category selectivity are not quantitatively precise and remain vulnerable to empirical refutation. Here we develop artificial neural network-based encoding models that accurately predict the response to novel images in the fusiform face area, parahippocampal place area, and extrastriate body area, outperforming descriptive models and experts. We use these models to subject claims of category selectivity to strong tests, by screening for and synthesizing images predicted to produce high responses. We find that these high-response-predicted images are all unambiguous members of the hypothesized preferred category for each region. These results provide accurate, image-computable encoding models of each category-selective region, strengthen evidence for domain specificity in the brain, and point the way for future research characterizing the functional organization of the brain with unprecedented computational precision.

Extensive Cortical Connectivity of the Human Hippocampal Memory System: Beyond the "What" and "Where" Dual Stream Model

The human hippocampus is involved in forming new memories: damage impairs memory. The dual stream model suggests that object "what" representations from ventral stream temporal cortex project to the hippocampus via the perirhinal and then lateral entorhinal cortex, and spatial "where" representations from the dorsal parietal stream via the parahippocampal gyrus and then medial entorhinal cortex. The hippocampus can then associate these inputs to form episodic memories of what happened where. Diffusion tractography was used to reveal the direct connections of hippocampal system areas in humans. This provides evidence that the human hippocampus has extensive direct cortical connections, with connections that bypass the entorhinal cortex to connect with the perirhinal and parahippocampal cortex, with the temporal pole, with the posterior and retrosplenial cingulate cortex, and even with early sensory cortical areas. The connections are less hierarchical and segregated than in the dual stream model. This provides a foundation for a conceptualization for how the hippocampal memory system connects with the cerebral cortex and operates in humans. One implication is that prehippocampal cortical areas such as the parahippocampal TF and TH subregions and perirhinal cortices may implement specialized computations that can benefit from inputs from the dorsal and ventral streams.

Neural Representations in the Prefrontal Cortex Are Task Dependent for Scene Attributes But Not for Scene Categories

Natural scenes deliver rich sensory information about the world. Decades of research has shown that the scene-selective network in the visual cortex represents various aspects of scenes. However, less is known about how such complex scene information is processed beyond the visual cortex, such as in the prefrontal cortex. It is also unknown how task context impacts the process of scene perception, modulating which scene content is represented in the brain. In this study, we investigate these questions using scene images from four natural scene categories, which also depict two types of scene attributes, temperature (warm or cold), and sound level (noisy or quiet). A group of healthy human subjects from both sexes participated in the present study using fMRI. In the study, participants viewed scene images under two different task conditions: temperature judgment and sound-level judgment. We analyzed how these scene attributes and categories are represented across the brain under these task conditions. Our findings show that scene attributes (temperature and sound level) are only represented in the brain when they are task relevant. However, scene categories are represented in the brain, in both the parahippocampal place area and the prefrontal cortex, regardless of task context. These findings suggest that the prefrontal cortex selectively represents scene content according to task demands, but this task selectivity depends on the types of scene content: task modulates neural representations of scene attributes but not of scene categories.SIGNIFICANCE STATEMENT Research has shown that visual scene information is processed in scene-selective regions in the occipital and temporal cortices. Here, we ask how scene content is processed and represented beyond the visual brain, in the prefrontal cortex (PFC). We show that both scene categories and scene attributes are represented in PFC, with interesting differences in task dependency: scene attributes are only represented in PFC when they are task relevant, but scene categories are represented in PFC regardless of the task context. Together, our work shows that scene information is processed beyond the visual cortex, and scene representation in PFC reflects how adaptively our minds extract relevant information from a scene.

Cortical networks of dynamic scene category representation in the human brain

Humans have an impressive ability to rapidly process global information in natural scenes to infer their category. Yet, it remains unclear whether and how scene categories observed dynamically in the natural world are represented in cerebral cortex beyond few canonical scene-selective areas. To address this question, here we examined the representation of dynamic visual scenes by recording whole-brain blood oxygenation level-dependent (BOLD) responses while subjects viewed natural movies. We fit voxelwise encoding models to estimate tuning for scene categories that reflect statistical ensembles of objects and actions in the natural world. We find that this scene-category model explains a significant portion of the response variance broadly across cerebral cortex. Cluster analysis of scene-category tuning profiles across cortex reveals nine spatially-segregated networks of brain regions consistently across subjects. These networks show heterogeneous tuning for a diverse set of dynamic scene categories related to navigation, human activity, social interaction, civilization, natural environment, non-human animals, motion-energy, and texture, suggesting that the organization of scene category representation is quite complex.

Lower-Resolution Retrieval of Scenes in Older Adults With Subjective Cognitive Decline

Objective

Scenes with more perceptual detail can help detect subtle memory deficits more than scenes with less detail. Here, we investigated whether older adults with subjective cognitive decline (SCD) show less brain activation and more memory deficits to scenes with more (vs. scenes with less) perceptual detail compared to controls (CON).

Method

In 37 healthy older adults (SCD: 16), we measured blood oxygenation level-dependent-functional magnetic resonance imaging during encoding and behavioral performance during retrieval.

Results

During encoding, higher activation to scenes with more (vs. less) perceptual detail in the parahippocampal place area predicted better memory performance in SCD and CON. During retrieval, superior performance for new scenes with more (vs. less) perceptual detail was significantly more pronounced in CON than inSCD.

Conclusions

Together, these results suggest a present, but attenuated benefit from perceptual detail for memory retrieval in SCD. Memory complaints in SCD might, thus, refer to a decreased availability of perceptual detail of previously encoded stimuli.

Unveiling functions of the visual cortex using task-specific deep neural networks

The human visual cortex enables visual perception through a cascade of hierarchical computations in cortical regions with distinct functionalities. Here, we introduce an AI-driven approach to discover the functional mapping of the visual cortex. We related human brain responses to scene images measured with functional MRI (fMRI) systematically to a diverse set of deep neural networks (DNNs) optimized to perform different scene perception tasks. We found a structured mapping between DNN tasks and brain regions along the ventral and dorsal visual streams. Low-level visual tasks mapped onto early brain regions, 3-dimensional scene perception tasks mapped onto the dorsal stream, and semantic tasks mapped onto the ventral stream. This mapping was of high fidelity, with more than 60% of the explainable variance in nine key regions being explained. Together, our results provide a novel functional mapping of the human visual cortex and demonstrate the power of the computational approach.

Learning Invariant Object and Spatial View Representations in the Brain Using Slow Unsupervised Learning

First, neurophysiological evidence for the learning of invariant representations in the inferior temporal visual cortex is described. This includes object and face representations with invariance for position, size, lighting, view and morphological transforms in the temporal lobe visual cortex; global object motion in the cortex in the superior temporal sulcus; and spatial view representations in the hippocampus that are invariant with respect to eye position, head direction, and place. Second, computational mechanisms that enable the brain to learn these invariant representations are proposed. For the ventral visual system, one key adaptation is the use of information available in the statistics of the environment in slow unsupervised learning to learn transform-invariant representations of objects. This contrasts with deep supervised learning in artificial neural networks, which uses training with thousands of exemplars forced into different categories by neuronal teachers. Similar slow learning principles apply to the learning of global object motion in the dorsal visual system leading to the cortex in the superior temporal sulcus. The learning rule that has been explored in VisNet is an associative rule with a short-term memory trace. The feed-forward architecture has four stages, with convergence from stage to stage. This type of slow learning is implemented in the brain in hierarchically organized competitive neuronal networks with convergence from stage to stage, with only 4-5 stages in the hierarchy. Slow learning is also shown to help the learning of coordinate transforms using gain modulation in the dorsal visual system extending into the parietal cortex and retrosplenial cortex. Representations are learned that are in allocentric spatial view coordinates of locations in the world and that are independent of eye position, head direction, and the place where the individual is located. This enables hippocampal spatial view cells to use idiothetic, self-motion, signals for navigation when the view details are obscured for short periods.

Shared cognitive mechanisms involved in the processing of scene texture and scene shape

Recent research has demonstrated that the parahippocampal place area represents both the shape and texture features of scenes, with the importance of each feature varying according to perceived scene category. Namely, shape features are predominately more diagnostic to the processing of artificial human–made scenes, while shape and texture are equally diagnostic in natural scene processing. However, to date little is known regarding the degree of interactivity or independence observed in the processing of these scene features. Furthermore, manipulating the scope of visual attention (i.e., globally vs. locally) when processing ensembles of multiple objects—stimuli that share a functional neuroanatomical link with scenes—has been shown to affect their cognitive visual representation. It remains unknown whether manipulating the scope of attention impacts scene processing in a similar manner. Using the well-established Garner speeded-classification behavioral paradigm, we investigated the influence of both feature diagnosticity and the scope of visual attention on potential interactivity or independence in the shape and texture processing of artificial human–made scenes. The results revealed asymmetric interference between scene shape and texture processing, with the more diagnostic feature (i.e., shape) interfering with the less diagnostic feature (i.e., texture), but not vice versa. Furthermore, this interference was attenuated and enhanced with more local and global visual processing strategies, respectively. These findings suggest that the scene shape and texture processing are mediated by shared cognitive mechanisms and that, although these representations are governed primarily via feature diagnosticity, they can nevertheless be influenced by the scope of visual attention.