Evaluating the correspondence between face-, scene-, and object-selectivity and retinotopic organization within lateral occipitotemporal cortex

Natural scenes reveal diverse representations of 2D and 3D body pose in the human brain

Human pose, defined as the spatial relationships between body parts, carries instrumental information supporting the understanding of motion and action of a person. A substantial body of previous work has identified cortical areas responsive to images of bodies and different body parts. However, the neural basis underlying the visual perception of body part relationships has received less attention. To broaden our understanding of body perception, we analyzed high-resolution fMRI responses to a wide range of poses from over 4,000 complex natural scenes. Using ground-truth annotations and an application of three-dimensional (3D) pose reconstruction algorithms, we compared similarity patterns of cortical activity with similarity patterns built from human pose models with different levels of depth availability and viewpoint dependency. Targeting the challenge of explaining variance in complex natural image responses with interpretable models, we achieved statistically significant correlations between pose models and cortical activity patterns (though performance levels are substantially lower than the noise ceiling). We found that the 3D view-independent pose model, compared with two-dimensional models, better captures the activation from distinct cortical areas, including the right posterior superior temporal sulcus (pSTS). These areas, together with other pose-selective regions in the LOTC, form a broader, distributed cortical network with greater view-tolerance in more anterior patches. We interpret these findings in light of the computational complexity of natural body images, the wide range of visual tasks supported by pose structures, and possible shared principles for view-invariant processing between articulated objects and ordinary, rigid objects.

Temporal dynamics of short-term neural adaptation across human visual cortex

Neural responses in visual cortex adapt to prolonged and repeated stimuli. While adaptation occurs across the visual cortex, it is unclear how adaptation patterns and computational mechanisms differ across the visual hierarchy. Here we characterize two signatures of short-term neural adaptation in time-varying intracranial electroencephalography (iEEG) data collected while participants viewed naturalistic image categories varying in duration and repetition interval. Ventral- and lateral-occipitotemporal cortex exhibit slower and prolonged adaptation to single stimuli and slower recovery from adaptation to repeated stimuli compared to V1-V3. For category-selective electrodes, recovery from adaptation is slower for preferred than non-preferred stimuli. To model neural adaptation we augment our delayed divisive normalization (DN) model by scaling the input strength as a function of stimulus category, enabling the model to accurately predict neural responses across multiple image categories. The model fits suggest that differences in adaptation patterns arise from slower normalization dynamics in higher visual areas interacting with differences in input strength resulting from category selectivity. Our results reveal systematic differences in temporal adaptation of neural population responses between lower and higher visual brain areas and show that a single computational model of history-dependent normalization dynamics, fit with area-specific parameters, accounts for these differences.

A previously undescribed scene-selective site is the key to encoding ego-motion in naturalistic environments

Current models of scene processing in the human brain include three scene-selective areas: the parahippocampal place area (or the temporal place areas), the restrosplenial cortex (or the medial place area), and the transverse occipital sulcus (or the occipital place area). Here, we challenged this model by showing that at least one other scene-selective site can also be detected within the human posterior intraparietal gyrus. Despite the smaller size of this site compared to the other scene-selective areas, the posterior intraparietal gyrus scene-selective (PIGS) site was detected consistently in a large pool of subjects (n = 59; 33 females). The reproducibility of this finding was tested based on multiple criteria, including comparing the results across sessions, utilizing different scanners (3T and 7T) and stimulus sets. Furthermore, we found that this site (but not the other three scene-selective areas) is significantly sensitive to ego-motion in scenes, thus distinguishing the role of PIGS in scene perception relative to other scene-selective areas. These results highlight the importance of including finer scale scene-selective sites in models of scene processing - a crucial step toward a more comprehensive understanding of how scenes are encoded under dynamic conditions.

"Walking selectivity" in the occipital place area in 8-year-olds, not 5-year-olds

A recent neuroimaging study in adults found that the occipital place area (OPA)-a cortical region involved in "visually guided navigation" (i.e. moving about the immediately visible environment, avoiding boundaries, and obstacles)-represents visual information about walking, not crawling, suggesting that OPA is late developing, emerging only when children are walking, not beforehand. But when precisely does this "walking selectivity" in OPA emerge-when children first begin to walk in early childhood, or perhaps counterintuitively, much later in childhood, around 8 years of age, when children are adult-like walking? To directly test these two hypotheses, using functional magnetic resonance imaging (fMRI) in two groups of children, 5- and 8-year-olds, we measured the responses in OPA to first-person perspective videos through scenes from a "walking" perspective, as well as three control perspectives ("crawling," "flying," and "scrambled"). We found that the OPA in 8-year-olds-like adults-exhibited walking selectivity (i.e. responding significantly more to the walking videos than to any of the others, and no significant differences across the crawling, flying, and scrambled videos), while the OPA in 5-year-olds exhibited no walking selectively. These findings reveal that OPA undergoes protracted development, with walking selectivity only emerging around 8 years of age.

A retinotopic code structures the interaction between perception and memory systems

Conventional views of brain organization suggest that regions at the top of the cortical hierarchy processes internally oriented information using an abstract amodal neural code. Despite this, recent reports have described the presence of retinotopic coding at the cortical apex, including the default mode network. What is the functional role of retinotopic coding atop the cortical hierarchy? Here we report that retinotopic coding structures interactions between internally oriented (mnemonic) and externally oriented (perceptual) brain areas. Using functional magnetic resonance imaging, we observed robust inverted (negative) retinotopic coding in category-selective memory areas at the cortical apex, which is functionally linked to the classic (positive) retinotopic coding in category-selective perceptual areas in high-level visual cortex. These functionally linked retinotopic populations in mnemonic and perceptual areas exhibit spatially specific opponent responses during both bottom-up perception and top-down recall, suggesting that these areas are interlocked in a mutually inhibitory dynamic. These results show that retinotopic coding structures interactions between perceptual and mnemonic neural systems, providing a scaffold for their dynamic interaction.

Organization of pRF size along the AP axis of the hippocampus and adjacent medial temporal cortex is related to specialization for scenes versus faces

The hippocampus is largely recognized for its integral contributions to memory processing. By contrast, its role in perceptual processing remains less clear. Hippocampal properties vary along the anterior-posterior (AP) axis. Based on past research suggesting a gradient in the scale of features processed along the AP extent of the hippocampus, the representations have been proposed to vary as a function of granularity along this axis. One way to quantify such granularity is with population receptive field (pRF) size measured during visual processing, which has so far received little attention. In this study, we compare the pRF sizes within the hippocampus to its activation for images of scenes versus faces. We also measure these functional properties in surrounding medial temporal lobe (MTL) structures. Consistent with past research, we find pRFs to be larger in the anterior than in the posterior hippocampus. Critically, our analysis of surrounding MTL regions, the perirhinal cortex, entorhinal cortex, and parahippocampal cortex shows a similar correlation between scene sensitivity and larger pRF size. These findings provide conclusive evidence for a tight relationship between the pRF size and the sensitivity to image content in the hippocampus and adjacent medial temporal cortex.

Distinct encoding and post-encoding representational formats contribute to episodic sequence memory formation

In episodic encoding, an unfolding experience is rapidly transformed into a memory representation that binds separate episodic elements into a memory form to be later recollected. However, it is unclear how brain activity changes over time to accommodate the encoding of incoming information. This study aimed to investigate the dynamics of the representational format that contributed to memory formation of sequential episodes. We combined representational similarity analysis and multivariate decoding approaches on EEG data to compare whether "category-level" or "item-level" representations supported memory formation during the online encoding of a picture triplet sequence and offline, in the period that immediately followed encoding. The findings revealed a gradual integration of category-level representation during the online encoding of the picture sequence and a rapid item-based neural reactivation of the encoded sequence at the episodic offset. However, we found that only memory reinstatement at episodic offset was associated with successful memory retrieval from long-term memory. These results suggest that post-encoding memory reinstatement is crucial for the rapid formation of unique memory for episodes that unfold over time. Overall, the study sheds light on the dynamics of representational format changes that take place during the formation of episodic memories.

A stimulus-driven approach reveals vertical luminance gradient as a stimulus feature that drives human cortical scene selectivity

Human neuroimaging studies have revealed a dedicated cortical system for visual scene processing. But what is a "scene"? Here, we use a stimulus-driven approach to identify a stimulus feature that selectively drives cortical scene processing. Specifically, using fMRI data from BOLD5000, we examined the images that elicited the greatest response in the cortical scene processing system, and found that there is a common "vertical luminance gradient" (VLG), with the top half of a scene image brighter than the bottom half; moreover, across the entire set of images, VLG systematically increases with the neural response in the scene-selective regions (Study 1). Thus, we hypothesized that VLG is a stimulus feature that selectively engages cortical scene processing, and directly tested the role of VLG in driving cortical scene selectivity using tightly controlled VLG stimuli (Study 2). Consistent with our hypothesis, we found that the scene-selective cortical regions-but not an object-selective region or early visual cortex-responded significantly more to images of VLG over control stimuli with minimal VLG. Interestingly, such selectivity was also found for images with an "inverted" VLG, resembling the luminance gradient in night scenes. Finally, we also tested the behavioral relevance of VLG for visual scene recognition (Study 3); we found that participants even categorized tightly controlled stimuli of both upright and inverted VLG to be a place more than an object, indicating that VLG is also used for behavioral scene recognition. Taken together, these results reveal that VLG is a stimulus feature that selectively engages cortical scene processing, and provide evidence for a recent proposal that visual scenes can be characterized by a set of common and unique visual features.

Peripheral visual field loss and activities of daily living

PURPOSE OF REVIEW

Peripheral visual field (VF) loss affects 13% of the population over 65. Its effect on activities of daily living and higher order visual processing is as important as it is inadequately understood. The purpose of this review is to summarize available literature on the impact of peripheral vision loss on driving, reading, face recognition, scene recognition and scene navigation.

RECENT FINDINGS

In this review, glaucoma and retrochiasmal cortical damage are utilized as examples of peripheral field loss which typically spare central vision and have patterns respecting the horizontal and vertical meridians, respectively. In both glaucoma and retrochiasmal damage, peripheral field loss causes driving difficulty - especially with lane maintenance - leading to driving cessation, loss of independence, and depression. Likewise, peripheral field loss can lead to slower reading speeds and decreased enjoyment from reading, and anxiety. In glaucoma and retrochiasmal field loss, face processing is impaired which impacts social functioning. Finally, scene recognition and navigation are also adversely affected, impacting wayfinding and hazard detection leading to decreased independence as well as more frequent injury.

SUMMARY

Peripheral VF loss is an under-recognized cause of patient distress and disability. All peripheral field loss is not the same, differential patterns of loss affect parameters of activities of daily living (ADL) and visual processing in particular ways. Future research should aim to further characterize patterns of deranged ADL and visual processing, their correlation with types of field loss, and associated mechanisms.

Functionally and structurally distinct fusiform face area(s) in over 1000 participants

The fusiform face area (FFA) is a widely studied region causally involved in face perception. Even though cognitive neuroscientists have been studying the FFA for over two decades, answers to foundational questions regarding the function, architecture, and connectivity of the FFA from a large (N>1000) group of participants are still lacking. To fill this gap in knowledge, we quantified these multimodal features of fusiform face-selective regions in 1053 participants in the Human Connectome Project. After manually defining over 4,000 fusiform face-selective regions, we report five main findings. First, 68.76% of hemispheres have two cortically separate regions (pFus-faces/FFA-1 and mFus-faces/FFA-2). Second, in 26.69% of hemispheres, pFus-faces/FFA-1 and mFus-faces/FFA-2 are spatially contiguous, yet are distinct based on functional, architectural, and connectivity metrics. Third, pFus-faces/FFA-1 is more face-selective than mFus-faces/FFA-2, and the two regions have distinct functional connectivity fingerprints. Fourth, pFus-faces/FFA-1 is cortically thinner and more heavily myelinated than mFus-faces/FFA-2. Fifth, face-selective patterns and functional connectivity fingerprints of each region are more similar in monozygotic than dizygotic twins and more so than architectural gradients. As we share our areal definitions with the field, future studies can explore how structural and functional features of these regions will inform theories regarding how visual categories are represented in the brain.

Functionally and structurally distinct fusiform face area(s) in over 1000 participants

The fusiform face area (FFA) is a widely studied region causally involved in face perception. Even though cognitive neuroscientists have been studying the FFA for over two decades, answers to foundational questions regarding the function, architecture, and connectivity of the FFA from a large (N>1000) group of participants are still lacking. To fill this gap in knowledge, we quantified these multimodal features of fusiform face-selective regions in 1053 participants in the Human Connectome Project. After manually defining over 4,000 fusiform face-selective regions, we report five main findings. First, 68.76% of hemispheres have two cortically separate regions (pFus-faces/FFA-1 and mFus-faces/FFA-2). Second, in 26.69% of hemispheres, pFus-faces/FFA-1 and mFus-faces/FFA-2 are spatially contiguous, yet are distinct based on functional, architectural, and connectivity metrics. Third, pFus-faces/FFA-1 is more face-selective than mFus-faces/FFA-2, and the two regions have distinct functional connectivity fingerprints. Fourth, pFus-faces/FFA-1 is cortically thinner and more heavily myelinated than mFus-faces/FFA-2. Fifth, face-selective patterns and functional connectivity fingerprints of each region are more similar in monozygotic than dizygotic twins and more so than architectural gradients. As we share our areal definitions with the field, future studies can explore how structural and functional features of these regions will inform theories regarding how visual categories are represented in the brain.

Relative tuning of holistic face processing towards the fovea

Humans quickly detect and gaze at faces in the world, which reflects their importance in cognition and may lead to tuning of face recognition toward the central visual field. Although sometimes reported, foveal selectivity in face processing is debated: brain imaging studies have found evidence for a central field bias specific to faces, but behavioral studies have found little foveal selectivity in face recognition. These conflicting results are difficult to reconcile, but they could arise from stimulus-specific differences. Recent studies, for example, suggest that individual faces vary in the degree to which they require holistic processing. Holistic processing is the perception of faces as a whole rather than as a set of separate features. We hypothesized that the dissociation between behavioral and neuroimaging studies arises because of this stimulus-specific dependence on holistic processing. Specifically, the central bias found in neuroimaging studies may be specific to holistic processing. Here, we tested whether the eccentricity-dependence of face perception is determined by the degree to which faces require holistic processing. We first measured the holistic-ness of individual Mooney faces (two-tone shadow images readily perceived as faces). In a group of independent observers, we then used a gender discrimination task to measured recognition of these Mooney faces as a function of their eccentricity. Face gender was recognized across the visual field, even at substantial eccentricities, replicating prior work. Importantly, however, holistic face gender recognition was relatively tuned-slightly, but reliably stronger in the central visual field. Our results may reconcile the debate on the eccentricity-dependance of face perception and reveal a spatial inhomogeneity specifically in the holistic representations of faces.

Visuospatial coding as ubiquitous scaffolding for human cognition

For more than 100 years we have known that the visual field is mapped onto the surface of visual cortex, imposing an inherently spatial reference frame on visual information processing. Recent studies highlight visuospatial coding not only throughout visual cortex, but also brain areas not typically considered visual. Such widespread access to visuospatial coding raises important questions about its role in wider cognitive functioning. Here, we synthesise these recent developments and propose that visuospatial coding scaffolds human cognition by providing a reference frame through which neural computations interface with environmental statistics and task demands via perception-action loops.

One object, two networks? Assessing the relationship between the face and body-selective regions in the primate visual system

Faces and bodies are often treated as distinct categories that are processed separately by face- and body-selective brain regions in the primate visual system. These regions occupy distinct regions of visual cortex and are often thought to constitute independent functional networks. Yet faces and bodies are part of the same object and their presence inevitably covary in naturalistic settings. Here, we re-evaluate both the evidence supporting the independent processing of faces and bodies and the organizational principles that have been invoked to explain this distinction. We outline four hypotheses ranging from completely separate networks to a single network supporting the perception of whole people or animals. The current evidence, especially in humans, is compatible with all of these hypotheses, making it presently unclear how the representation of faces and bodies is organized in the cortex.

Direct comparison of contralateral bias and face/scene selectivity in human occipitotemporal cortex

Human visual cortex is organised broadly according to two major principles: retinotopy (the spatial mapping of the retina in cortex) and category-selectivity (preferential responses to specific categories of stimuli). Historically, these principles were considered anatomically separate, with retinotopy restricted to the occipital cortex and category-selectivity emerging in the lateral-occipital and ventral-temporal cortex. However, recent studies show that category-selective regions exhibit systematic retinotopic biases, for example exhibiting stronger activation for stimuli presented in the contra- compared to the ipsilateral visual field. It is unclear, however, whether responses within category-selective regions are more strongly driven by retinotopic location or by category preference, and if there are systematic differences between category-selective regions in the relative strengths of these preferences. Here, we directly compare contralateral and category preferences by measuring fMRI responses to scene and face stimuli presented in the left or right visual field and computing two bias indices: a contralateral bias (response to the contralateral minus ipsilateral visual field) and a face/scene bias (preferred response to scenes compared to faces, or vice versa). We compare these biases within and between scene- and face-selective regions and across the lateral and ventral surfaces of the visual cortex more broadly. We find an interaction between surface and bias: lateral surface regions show a stronger contralateral than face/scene bias, whilst ventral surface regions show the opposite. These effects are robust across and within subjects, and appear to reflect large-scale, smoothly varying gradients. Together, these findings support distinct functional roles for the lateral and ventral visual cortex in terms of the relative importance of the spatial location of stimuli during visual information processing.

Inferior Occipital Gyrus Is Organized along Common Gradients of Spatial and Face-Part Selectivity

The ventral visual stream of the human brain is subdivided into patches with categorical stimulus preferences, like faces or scenes. However, the functional organization within these areas is less clear. Here, we used functional magnetic resonance imaging and vertex-wise tuning models to independently probe spatial and face-part preferences in the inferior occipital gyrus (IOG) of healthy adult males and females. The majority of responses were well explained by Gaussian population tuning curves for both retinotopic location and the preferred relative position within a face. Parameter maps revealed a common gradient of spatial and face-part selectivity, with the width of tuning curves drastically increasing from posterior to anterior IOG. Tuning peaks clustered more idiosyncratically but were also correlated across maps of visual and face space. Preferences for the upper visual field went along with significantly increased coverage of the upper half of the face, matching recently discovered biases in human perception. Our findings reveal a broad range of neural face-part selectivity in IOG, ranging from narrow to "holistic." IOG is functionally organized along this gradient, which in turn is correlated with retinotopy.SIGNIFICANCE STATEMENT Brain imaging has revealed a lot about the large-scale organization of the human brain and visual system. For example, occipital cortex contains map-like representations of the visual field, while neurons in ventral areas cluster into patches with categorical preferences, like faces or scenes. Much less is known about the functional organization within these areas. Here, we focused on a well established face-preferring area-the inferior occipital gyrus (IOG). A novel neuroimaging paradigm allowed us to map the retinotopic and face-part tuning of many recording sites in IOG independently. We found a steep posterior-anterior gradient of decreasing face-part selectivity, which correlated with retinotopy. This suggests the functional role of ventral areas is not uniform and may follow retinotopic "protomaps."

Theta-burst TMS of lateral occipital cortex reduces BOLD responses across category-selective areas in ventral temporal cortex

Human visual cortex contains three scene-selective regions in the lateral, medial and ventral cortex, termed the occipital place area (OPA), medial place area (MPA) and parahippocampal place area (PPA). Using functional magnetic resonance imaging (fMRI), all three regions respond more strongly when viewing visual scenes compared with isolated objects or faces. To determine how these regions are functionally and causally connected, we applied transcranial magnetic stimulation to OPA and measured fMRI responses before and after stimulation, using a theta-burst paradigm (TBS). To test for stimulus category-selectivity, we presented a range of visual categories (scenes, buildings, objects, faces). To test for specificity of any effects to TBS of OPA we employed two control conditions: Sham, with no TBS stimulation, and an active TBS-control with TBS to a proximal face-selective cortical region (occipital face area, or OFA). We predicted that TBS to OPA (but not OFA) would lead to decreased responses to scenes and buildings (but not other categories) in other scene-selective cortical regions. Across both ROI and whole-volume analyses, we observed decreased responses to scenes in PPA as a result of TBS. However, these effects were neither category specific, with decreased responses to all stimulus categories, nor limited to scene-selective regions, with decreases also observed in face-selective fusiform face area (FFA). Furthermore, similar effects were observed with TBS to OFA, thus effects were not specific to the stimulation site in the lateral occipital cortex. Whilst these data are suggestive of a causal, but non-specific relationship between lateral occipital and ventral temporal cortex, we discuss several factors that could have underpinned this result, such as the differences between TBS and online TMS, the role of anatomical distance between stimulated regions and how TMS effects are operationalised. Furthermore, our findings highlight the importance of active control conditions in brain stimulation experiments to accurately assess functional and causal connectivity between specific brain regions.

Shape coding in occipito-temporal cortex relies on object silhouette, curvature, and medial axis

Object recognition relies on different transformations of the retinal input, carried out by the visual system, that range from local contrast to object shape and category. While some of those transformations are thought to occur at specific stages of the visual hierarchy, the features they represent are correlated (e.g., object shape and identity) and selectivity for the same feature overlaps in many brain regions. This may be explained either by collinearity across representations or may instead reflect the coding of multiple dimensions by the same cortical population. Moreover, orthogonal and shared components may differently impact distinctive stages of the visual hierarchy. We recorded functional MRI activity while participants passively attended to object images and employed a statistical approach that partitioned orthogonal and shared object representations to reveal their relative impact on brain processing. Orthogonal shape representations (silhouette, curvature, and medial axis) independently explained distinct and overlapping clusters of selectivity in the occitotemporal and parietal cortex. Moreover, we show that the relevance of shared representations linearly increases moving from posterior to anterior regions. These results indicate that the visual cortex encodes shared relations between different features in a topographic fashion and that object shape is encoded along different dimensions, each representing orthogonal features.NEW & NOTEWORTHY There are several possible ways of characterizing the shape of an object. Which shape description better describes our brain responses while we passively perceive objects? Here, we employed three competing shape models to explain brain representations when viewing real objects. We found that object shape is encoded in a multidimensional fashion and thus defined by the interaction of multiple features.

A common neural substrate for processing scenes and egomotion-compatible visual motion

Neuroimaging studies have revealed two separate classes of category-selective regions specialized in optic flow (egomotion-compatible) processing and in scene/place perception. Despite the importance of both optic flow and scene/place recognition to estimate changes in position and orientation within the environment during self-motion, the possible functional link between egomotion- and scene-selective regions has not yet been established. Here we reanalyzed functional magnetic resonance images from a large sample of participants performing two well-known “localizer” fMRI experiments, consisting in passive viewing of navigationally relevant stimuli such as buildings and places (scene/place stimulus) and coherently moving fields of dots simulating the visual stimulation during self-motion (flow fields). After interrogating the egomotion-selective areas with respect to the scene/place stimulus and the scene-selective areas with respect to flow fields, we found that the egomotion-selective areas V6+ and pIPS/V3A responded bilaterally more to scenes/places compared to faces, and all the scene-selective areas (parahippocampal place area or PPA, retrosplenial complex or RSC, and occipital place area or OPA) responded more to egomotion-compatible optic flow compared to random motion. The conjunction analysis between scene/place and flow field stimuli revealed that the most important focus of common activation was found in the dorsolateral parieto-occipital cortex, spanning the scene-selective OPA and the egomotion-selective pIPS/V3A. Individual inspection of the relative locations of these two regions revealed a partial overlap and a similar response profile to an independent low-level visual motion stimulus, suggesting that OPA and pIPS/V3A may be part of a unique motion-selective complex specialized in encoding both egomotion- and scene-relevant information, likely for the control of navigation in a structured environment.

Similarity judgments and cortical visual responses reflect different properties of object and scene categories in naturalistic images

Numerous factors have been reported to underlie the representation of complex images in high-level human visual cortex, including categories (e.g. faces, objects, scenes), animacy, and real-world size, but the extent to which this organization reflects behavioral judgments of real-world stimuli is unclear. Here, we compared representations derived from explicit behavioral similarity judgments and ultra-high field (7T) fMRI of human visual cortex for multiple exemplars of a diverse set of naturalistic images from 48 object and scene categories. While there was a significant correlation between similarity judgments and fMRI responses, there were striking differences between the two representational spaces. Behavioral judgements primarily revealed a coarse division between man-made (including humans) and natural (including animals) images, with clear groupings of conceptually-related categories (e.g. transportation, animals), while these conceptual groupings were largely absent in the fMRI representations. Instead, fMRI responses primarily seemed to reflect a separation of both human and non-human faces/bodies from all other categories. Further, comparison of the behavioral and fMRI representational spaces with those derived from the layers of a deep neural network (DNN) showed a strong correspondence with behavior in the top-most layer and with fMRI in the mid-level layers. These results suggest a complex relationship between localized responses in high-level visual cortex and behavioral similarity judgments - each domain reflects different properties of the images, and responses in high-level visual cortex may correspond to intermediate stages of processing between basic visual features and the conceptual categories that dominate the behavioral response.

Scene Perception in the Human Brain

Humans are remarkably adept at perceiving and understanding complex real-world scenes. Uncovering the neural basis of this ability is an important goal of vision science. Neuroimaging studies have identified three cortical regions that respond selectively to scenes: parahippocampal place area, retrosplenial complex/medial place area, and occipital place area. Here, we review what is known about the visual and functional properties of these brain areas. Scene-selective regions exhibit retinotopic properties and sensitivity to low-level visual features that are characteristic of scenes. They also mediate higher-level representations of layout, objects, and surface properties that allow individual scenes to be recognized and their spatial structure ascertained. Challenges for the future include developing computational models of information processing in scene regions, investigating how these regions support scene perception under ecologically realistic conditions, and understanding how they operate in the context of larger brain networks.

Object Vision in a Structured World

In natural vision, objects appear at typical locations, both with respect to visual space (e.g., an airplane in the upper part of a scene) and other objects (e.g., a lamp above a table). Recent studies have shown that object vision is strongly adapted to such positional regularities. In this review we synthesize these developments, highlighting that adaptations to positional regularities facilitate object detection and recognition, and sharpen the representations of objects in visual cortex. These effects are pervasive across various types of high-level content. We posit that adaptations to real-world structure collectively support optimal usage of limited cortical processing resources. Taking positional regularities into account will thus be essential for understanding efficient object vision in the real world.

Category-selective areas in human visual cortex exhibit preferences for stimulus depth

Multiple regions in the human brain are dedicated to accomplish the feat of object recognition; yet our brains must also compute the 2D and 3D locations of the objects we encounter in order to make sense of our visual environments. A number of studies have explored how various object category-selective regions are sensitive to and have preferences for specific 2D spatial locations in addition to processing their preferred-stimulus categories, but there is no survey of how these regions respond to depth information. In a blocked functional MRI experiment, subjects viewed a series of category-specific (i.e., faces, objects, scenes) and unspecific (e.g., random moving dots) stimuli with red/green anaglyph glasses. Critically, these stimuli were presented at different depth planes such that they appeared in front of, behind, or at the same (i.e., middle) depth plane as the fixation point (Experiment 1) or simultaneously in front of and behind fixation (i.e., mixed depth; Experiment 2). Comparisons of mean response magnitudes between back, middle, and front depth planes reveal that face and object regions OFA and LOC exhibit a preference for front depths, and motion area MT+ exhibits a strong linear preference for front, followed by middle, followed by back depth planes. In contrast, scene-selective regions PPA and OPA prefer front and/or back depth planes (relative to middle). Moreover, the occipital place area demonstrates a strong preference for "mixed" depth above and beyond back alone, raising potential implications about its particular role in scene perception. Crucially, the observed depth preferences in nearly all areas were evoked irrespective of the semantic stimulus category being viewed. These results reveal that the object category-selective regions may play a role in processing or incorporating depth information that is orthogonal to their primary processing of object category information.

Scene complexity modulates degree of feedback activity during object detection in natural scenes

Selective brain responses to objects arise within a few hundreds of milliseconds of neural processing, suggesting that visual object recognition is mediated by rapid feed-forward activations. Yet disruption of neural responses in early visual cortex beyond feed-forward processing stages affects object recognition performance. Here, we unite these discrepant findings by reporting that object recognition involves enhanced feedback activity (recurrent processing within early visual cortex) when target objects are embedded in natural scenes that are characterized by high complexity. Human participants performed an animal target detection task on natural scenes with low, medium or high complexity as determined by a computational model of low-level contrast statistics. Three converging lines of evidence indicate that feedback was selectively enhanced for high complexity scenes. First, functional magnetic resonance imaging (fMRI) activity in early visual cortex (V1) was enhanced for target objects in scenes with high, but not low or medium complexity. Second, event-related potentials (ERPs) evoked by target objects were selectively enhanced at feedback stages of visual processing (from ~220 ms onwards) for high complexity scenes only. Third, behavioral performance for high complexity scenes deteriorated when participants were pressed for time and thus less able to incorporate the feedback activity. Modeling of the reaction time distributions using drift diffusion revealed that object information accumulated more slowly for high complexity scenes, with evidence accumulation being coupled to trial-to-trial variation in the EEG feedback response. Together, these results suggest that while feed-forward activity may suffice to recognize isolated objects, the brain employs recurrent processing more adaptively in naturalistic settings, using minimal feedback for simple scenes and increasing feedback for complex scenes.

Idiosyncratic, Retinotopic Bias in Face Identification Modulated by Familiarity

The perception of gender and age of unfamiliar faces is reported to vary idiosyncratically across retinal locations such that, for example, the same androgynous face may appear to be male at one location but female at another. Here, we test spatial heterogeneity for the recognition of the identity of personally familiar faces in human participants. We found idiosyncratic biases that were stable within participants and that varied more across locations for low as compared to high familiar faces. These data suggest that like face gender and age, face identity is processed, in part, by independent populations of neurons monitoring restricted spatial regions and that the recognition responses vary for the same face across these different locations. Moreover, repeated and varied social interactions appear to lead to adjustments of these independent face recognition neurons so that the same familiar face is eventually more likely to elicit the same recognition response across widely separated visual field locations. We provide a mechanistic account of this reduced retinotopic bias based on computational simulations.

Transcranial Magnetic Stimulation to the Occipital Place Area Biases Gaze During Scene Viewing

We can understand viewed scenes and extract task-relevant information within a few hundred milliseconds. This process is generally supported by three cortical regions that show selectivity for scene images: parahippocampal place area (PPA), medial place area (MPA) and occipital place area (OPA). Prior studies have focused on the visual information each region is responsive to, usually within the context of recognition or navigation. Here, we move beyond these tasks to investigate gaze allocation during scene viewing. Eye movements rely on a scene’s visual representation to direct saccades, and thus foveal vision. In particular, we focus on the contribution of OPA, which is: (i) located in occipito-parietal cortex, likely feeding information into parts of the dorsal pathway critical for eye movements; and (ii) contains strong retinotopic representations of the contralateral visual field. Participants viewed scene images for 1034 ms while their eye movements were recorded. On half of the trials, a 500 ms train of five transcranial magnetic stimulation (TMS) pulses was applied to the participant’s cortex, starting at scene onset. TMS was applied to the right hemisphere over either OPA or the occipital face area (OFA), which also exhibits a contralateral visual field bias but shows selectivity for face stimuli. Participants generally made an overall left-to-right, top-to-bottom pattern of eye movements across all conditions. When TMS was applied to OPA, there was an increased saccade latency for eye movements toward the contralateral relative to the ipsilateral visual field after the final TMS pulse (400 ms). Additionally, TMS to the OPA biased fixation positions away from the contralateral side of the scene compared to the control condition, while the OFA group showed no such effect. There was no effect on horizontal saccade amplitudes. These combined results suggest that OPA might serve to represent local scene information that can then be utilized by visuomotor control networks to guide gaze allocation in natural scenes.

Computational mechanisms underlying cortical responses to the affordance properties of visual scenes

Biologically inspired deep convolutional neural networks (CNNs), trained for computer vision tasks, have been found to predict cortical responses with remarkable accuracy. However, the internal operations of these models remain poorly understood, and the factors that account for their success are unknown. Here we develop a set of techniques for using CNNs to gain insights into the computational mechanisms underlying cortical responses. We focused on responses in the occipital place area (OPA), a scene-selective region of dorsal occipitoparietal cortex. In a previous study, we showed that fMRI activation patterns in the OPA contain information about the navigational affordances of scenes; that is, information about where one can and cannot move within the immediate environment. We hypothesized that this affordance information could be extracted using a set of purely feedforward computations. To test this idea, we examined a deep CNN with a feedforward architecture that had been previously trained for scene classification. We found that responses in the CNN to scene images were highly predictive of fMRI responses in the OPA. Moreover the CNN accounted for the portion of OPA variance relating to the navigational affordances of scenes. The CNN could thus serve as an image-computable candidate model of affordance-related responses in the OPA. We then ran a series of in silico experiments on this model to gain insights into its internal operations. These analyses showed that the computation of affordance-related features relied heavily on visual information at high-spatial frequencies and cardinal orientations, both of which have previously been identified as low-level stimulus preferences of scene-selective visual cortex. These computations also exhibited a strong preference for information in the lower visual field, which is consistent with known retinotopic biases in the OPA. Visualizations of feature selectivity within the CNN suggested that affordance-based responses encoded features that define the layout of the spatial environment, such as boundary-defining junctions and large extended surfaces. Together, these results map the sensory functions of the OPA onto a fully quantitative model that provides insights into its visual computations. More broadly, they advance integrative techniques for understanding visual cortex across multiple level of analysis: from the identification of cortical sensory functions to the modeling of their underlying algorithms.

How does visual cortex compute behaviorally relevant properties of the local environment from sensory inputs? For decades, computational models have been able to explain only the earliest stages of biological vision, but recent advances in deep neural networks have yielded a breakthrough in the modeling of high-level visual cortex. However, these models are not explicitly designed for testing neurobiological theories, and, like the brain itself, their internal operations remain poorly understood. We examined a deep neural network for insights into the cortical representation of navigational affordances in visual scenes. In doing so, we developed a set of high-throughput techniques and statistical tools that are broadly useful for relating the internal operations of neural networks with the information processes of the brain. Our findings demonstrate that a deep neural network with purely feedforward computations can account for the processing of navigational layout in high-level visual cortex. We next performed a series of experiments and visualization analyses on this neural network. These analyses characterized a set of stimulus input features that may be critical for computing navigationally related cortical representations, and they identified a set of high-level, complex scene features that may serve as a basis set for the cortical coding of navigational layout. These findings suggest a computational mechanism through which high-level visual cortex might encode the spatial structure of the local navigational environment, and they demonstrate an experimental approach for leveraging the power of deep neural networks to understand the visual computations of the brain.

A cross-validated cytoarchitectonic atlas of the human ventral visual stream

The human ventral visual stream consists of several areas that are considered processing stages essential for perception and recognition. A fundamental microanatomical feature differentiating areas is cytoarchitecture, which refers to the distribution, size, and density of cells across cortical layers. Because cytoarchitectonic structure is measured in 20-micron-thick histological slices of postmortem tissue, it is difficult to assess (a) how anatomically consistent these areas are across brains and (b) how they relate to brain parcellations obtained with prevalent neuroimaging methods, acquired at the millimeter and centimeter scale. Therefore, the goal of this study was to (a) generate a cross-validated cytoarchitectonic atlas of the human ventral visual stream on a whole brain template that is commonly used in neuroimaging studies and (b) to compare this atlas to a recently published retinotopic parcellation of visual cortex (Wang et al., 2014). To achieve this goal, we generated an atlas of eight cytoarchitectonic areas: four areas in the occipital lobe (hOc1-hOc4v) and four in the fusiform gyrus (FG1-FG4), then we tested how the different alignment techniques affect the accuracy of the resulting atlas. Results show that both cortex-based alignment (CBA) and nonlinear volumetric alignment (NVA) generate an atlas with better cross-validation performance than affine volumetric alignment (AVA). Additionally, CBA outperformed NVA in 6/8 of the cytoarchitectonic areas. Finally, the comparison of the cytoarchitectonic atlas to a retinotopic atlas shows a clear correspondence between cytoarchitectonic and retinotopic areas in the ventral visual stream. The successful performance of CBA suggests a coupling between cytoarchitectonic areas and macroanatomical landmarks in the human ventral visual stream, and furthermore, that this coupling can be utilized for generating an accurate group atlas. In addition, the coupling between cytoarchitecture and retinotopy highlights the potential use of this atlas in understanding how anatomical features contribute to brain function. We make this cytoarchitectonic atlas freely available in both BrainVoyager and FreeSurfer formats (http://vpnl.stanford.edu/vcAtlas). The availability of this atlas will enable future studies to link cytoarchitectonic organization to other parcellations of the human ventral visual stream with potential to advance the understanding of this pathway in typical and atypical populations.

Development differentially sculpts receptive fields across early and high-level human visual cortex

Receptive fields (RFs) processing information in restricted parts of the visual field are a key property of visual system neurons. However, how RFs develop in humans is unknown. Using fMRI and population receptive field (pRF) modeling in children and adults, we determine where and how pRFs develop across the ventral visual stream. Here we report that pRF properties in visual field maps, from the first visual area, V1, through the first ventro-occipital area, VO1, are adult-like by age 5. However, pRF properties in face-selective and character-selective regions develop into adulthood, increasing the foveal coverage bias for faces in the right hemisphere and words in the left hemisphere. Eye-tracking indicates that pRF changes are related to changing fixation patterns on words and faces across development. These findings suggest a link between face and word viewing behavior and the differential development of pRFs across visual cortex, potentially due to competition on foveal coverage.

Population receptive fields (pRFs) in the visual system are key information-processors, but how they develop is unknown. Here, authors use fMRI and pRF modeling in children and adults to show that in the ventral stream only pRFs in face- and word-selective regions continue to develop, mirroring changes in viewing behavior.

Differential Sampling of Visual Space in Ventral and Dorsal Early Visual Cortex

A fundamental feature of cortical visual processing is the separation of visual processing for the upper and lower visual fields. In early visual cortex (EVC), the upper visual field is processed ventrally, with the lower visual field processed dorsally. This distinction persists into several category-selective regions of occipitotemporal cortex, with ventral and lateral scene-, face-, and object-selective regions biased for the upper and lower visual fields, respectively. Here, using an elliptical population receptive field (pRF) model, we systematically tested the sampling of visual space within ventral and dorsal divisions of human EVC in both male and female participants. We found that (1) pRFs tend to be elliptical and oriented toward the fovea with distinct angular distributions for ventral and dorsal divisions of EVC, potentially reflecting a radial bias; and (2) pRFs in ventral areas were larger (∼1.5×) and more elliptical (∼1.2×) than those in dorsal areas. These differences potentially reflect a tendency for receptive fields in ventral temporal cortex to overlap the fovea with less emphasis on precise localization and isotropic representation of space compared with dorsal areas. Collectively, these findings suggest that ventral and dorsal divisions of EVC sample visual space differently, likely contributing to and/or stemming from the functional differentiation of visual processing observed in higher-level regions of the ventral and dorsal cortical visual pathways.SIGNIFICANCE STATEMENT The processing of visual information from the upper and lower visual fields is separated in visual cortex. Although ventral and dorsal divisions of early visual cortex (EVC) are commonly assumed to sample visual space equivalently, we demonstrate systematic differences using an elliptical population receptive field (pRF) model. Specifically, we demonstrate that (1) ventral and dorsal divisions of EVC exhibit diverging distributions of pRF angle, which are biased toward the fovea; and (2) ventral pRFs exhibit higher aspect ratios and cover larger areas than dorsal pRFs. These results suggest that ventral and dorsal divisions of EVC sample visual space differently and that such differential sampling likely contributes to different functional roles attributed to the ventral and dorsal pathways, such as object recognition and visually guided attention, respectively.

Retinotopic Organization of Scene Areas in Macaque Inferior Temporal Cortex

Primates have specialized domains in inferior temporal (IT) cortex that are responsive to particular image categories. Though IT traditionally has been regarded as lacking retinotopy, several recent studies in monkeys have shown that retinotopic maps extend to face patches along the lower bank of the superior temporal sulcus (STS) and neighboring regions of IT cortex. Here, we used fMRI to map the retinotopic organization of medial ventral temporal cortex in four monkeys (2 male and 2 female). We confirm the presence of visual field maps within and around the lower bank of the STS and extend these prior findings to scene-selective cortex in the ventral-most regions of IT. Within the occipitotemporal sulcus (OTS), we identified two retinotopic areas, OTS1 and OTS2. The polar angle representation of OTS2 was a mirror reversal of the OTS1 representation. These regions contained representations of the contralateral periphery and were selectively active for scene versus face, body, or object images. The extent of this retinotopy parallels that in humans and shows that the organization of the scene network is preserved across primate species. In addition retinotopic maps were identified in dorsal extrastriate, posterior parietal, and frontal cortex as well as the thalamus, including both the lateral geniculate nucleus and pulvinar. Together, it appears that most, if not all, of the macaque visual system contains organized representations of visual space.SIGNIFICANCE STATEMENT Primates have specialized domains in inferior temporal (IT) cortex that are responsive to particular image categories. Though retinotopic maps are considered a fundamental organizing principle of posterior visual cortex, IT traditionally has been regarded as lacking retinotopy. Recent imaging studies have demonstrated the presence of several visual field maps within the lateral IT. Using neuroimaging, we found multiple representations of visual space within ventral IT cortex of macaques that included scene-selective cortex. Scene domains were biased toward the peripheral visual field. These data demonstrate the prevalence of visual field maps throughout the primate visual system, including late stages in the ventral visual hierarchy, and support the idea that domains representing different categories are biased toward different parts of the visual field.

Patterns of response to scrambled scenes reveal the importance of visual properties in the organization of scene-selective cortex

Neuroimaging studies have found distinct patterns of neural response to different categories of scene in scene-selective regions of the human brain. However, it is not clear how information about scene category is represented in these regions. Images from different categories vary systematically in their visual properties as well as their semantic category. So, it is possible that patterns of neural response could reflect variation in visual properties. To address this question, we used fMRI to measure patterns of neural response to intact and scrambled scene categories. Although scrambling preserved many of their visual characteristics, perception of scene categories was severely impaired. Nevertheless, we found distinct patterns of response to different scene categories in the parahippocampal place area (PPA) and the occipital place area (OPA) for both intact and scrambled scenes. Moreover, intact and scrambled scenes produced highly similar patterns of response. Our finding that reliable and distinct patterns of response in scene-selective regions are still evident when categorical perception is impaired suggests that visual properties play an important role in the topographic organization of these regions.

Coding of navigational affordances in the human visual system

A central component of spatial navigation is determining where one can and cannot go in the immediate environment. We used fMRI to test the hypothesis that the human visual system solves this problem by automatically identifying the navigational affordances of the local scene. Multivoxel pattern analyses showed that a scene-selective region of dorsal occipitoparietal cortex, known as the occipital place area, represents pathways for movement in scenes in a manner that is tolerant to variability in other visual features. These effects were found in two experiments: One using tightly controlled artificial environments as stimuli, the other using a diverse set of complex, natural scenes. A reconstruction analysis demonstrated that the population codes of the occipital place area could be used to predict the affordances of novel scenes. Taken together, these results reveal a previously unknown mechanism for perceiving the affordance structure of navigable space.

Contributions of low- and high-level properties to neural processing of visual scenes in the human brain

Visual scene analysis in humans has been characterized by the presence of regions in extrastriate cortex that are selectively responsive to scenes compared with objects or faces. While these regions have often been interpreted as representing high-level properties of scenes (e.g. category), they also exhibit substantial sensitivity to low-level (e.g. spatial frequency) and mid-level (e.g. spatial layout) properties, and it is unclear how these disparate findings can be united in a single framework. In this opinion piece, we suggest that this problem can be resolved by questioning the utility of the classical low- to high-level framework of visual perception for scene processing, and discuss why low- and mid-level properties may be particularly diagnostic for the behavioural goals specific to scene perception as compared to object recognition. In particular, we highlight the contributions of low-level vision to scene representation by reviewing (i) retinotopic biases and receptive field properties of scene-selective regions and (ii) the temporal dynamics of scene perception that demonstrate overlap of low- and mid-level feature representations with those of scene category. We discuss the relevance of these findings for scene perception and suggest a more expansive framework for visual scene analysis.This article is part of the themed issue 'Auditory and visual scene analysis'.

Two Distinct Scene-Processing Networks Connecting Vision and Memory

A number of regions in the human brain are known to be involved in processing natural scenes, but the field has lacked a unifying framework for understanding how these different regions are organized and interact. We provide evidence from functional connectivity and meta-analyses for a new organizational principle, in which scene processing relies upon two distinct networks that split the classically defined parahippocampal place area (PPA). The first network of strongly connected regions consists of the occipital place area/transverse occipital sulcus and posterior PPA, which contain retinotopic maps and are not strongly coupled to the hippocampus at rest. The second network consists of the caudal inferior parietal lobule, retrosplenial complex, and anterior PPA, which connect to the hippocampus (especially anterior hippocampus), and are implicated in both visual and nonvisual tasks, including episodic memory and navigation. We propose that these two distinct networks capture the primary functional division among scene-processing regions, between those that process visual features from the current view of a scene and those that connect information from a current scene view with a much broader temporal and spatial context. This new framework for understanding the neural substrates of scene-processing bridges results from many lines of research, and makes specific functional predictions.

Making Sense of Real-World Scenes

To interact with the world, we have to make sense of the continuous sensory input conveying information about our environment. A recent surge of studies has investigated the processes enabling scene understanding, using increasingly complex stimuli and sophisticated analyses to highlight the visual features and brain regions involved. However, there are two major challenges to producing a comprehensive framework for scene understanding. First, scene perception is highly dynamic, subserving multiple behavioral goals. Second, a multitude of different visual properties co-occur across scenes and may be correlated or independent. We synthesize the recent literature and argue that for a complete view of scene understanding, it is necessary to account for both differing observer goals and the contribution of diverse scene properties.

Scene-Selectivity and Retinotopy in Medial Parietal Cortex

Functional imaging studies in human reliably identify a trio of scene-selective regions, one on each of the lateral [occipital place area (OPA)], ventral [parahippocampal place area (PPA)], and medial [retrosplenial complex (RSC)] cortical surfaces. Recently, we demonstrated differential retinotopic biases for the contralateral lower and upper visual fields within OPA and PPA, respectively. Here, using functional magnetic resonance imaging, we combine detailed mapping of both population receptive fields (pRF) and category-selectivity, with independently acquired resting-state functional connectivity analyses, to examine scene and retinotopic processing within medial parietal cortex. We identified a medial scene-selective region, which was contained largely within the posterior and ventral bank of the parieto-occipital sulcus (POS). While this region is typically referred to as RSC, the spatial extent of our scene-selective region typically did not extend into retrosplenial cortex, and thus we adopt the term medial place area (MPA) to refer to this visually defined scene-selective region. Intriguingly MPA co-localized with a region identified solely on the basis of retinotopic sensitivity using pRF analyses. We found that MPA demonstrates a significant contralateral visual field bias, coupled with large pRF sizes. Unlike OPA and PPA, MPA did not show a consistent bias to a single visual quadrant. MPA also co-localized with a region identified by strong differential functional connectivity with PPA and the human face-selective fusiform face area (FFA), commensurate with its functional selectivity. Functional connectivity with OPA was much weaker than with PPA, and similar to that with face-selective occipital face area (OFA), suggesting a closer link with ventral than lateral cortex. Consistent with prior research, we also observed differential functional connectivity in medial parietal cortex for anterior over posterior PPA, as well as a region on the lateral surface, the caudal inferior parietal lobule (cIPL). However, the differential connectivity in medial parietal cortex was found principally anterior of MPA. We suggest that there is posterior–anterior gradient within medial parietal cortex, with posterior regions in the POS showing retinotopically based scene-selectivity and more anterior regions showing connectivity that may be more reflective of abstract, navigationally pertinent and possibly mnemonic representations.