Where do objects become scenes?

Natural scene statistics account for the representation of scene categories in human visual cortex

During natural vision, humans categorize the scenes they encounter: an office, the beach, and so on. These categories are informed by knowledge of the way that objects co-occur in natural scenes. How does the human brain aggregate information about objects to represent scene categories? To explore this issue, we used statistical learning methods to learn categories that objectively capture the co-occurrence statistics of objects in a large collection of natural scenes. Using the learned categories, we modeled fMRI brain signals evoked in human subjects when viewing images of scenes. We find that evoked activity across much of anterior visual cortex is explained by the learned categories. Furthermore, a decoder based on these scene categories accurately predicts the categories and objects comprising novel scenes from brain activity evoked by those scenes. These results suggest that the human brain represents scene categories that capture the co-occurrence statistics of objects in the world.

Spatial encoding and underlying circuitry in scene-selective cortex

Three cortical areas (Retro-Splenial Cortex (RSC), Transverse Occipital Sulcus (TOS) and Parahippocampal Place Area (PPA)) respond selectively to scenes. However, their wider role in spatial encoding and their functional connectivity remain unclear. Using fMRI, first we tested the responses of these areas during spatial comparison tasks using dot targets on white noise. Activity increased during task performance in both RSC and TOS, but not in PPA. However, the amplitude of task-driven activity and behavioral measures of task demand were correlated only in RSC. A control experiment showed that none of these areas were activated during a comparable shape comparison task. Secondly, we analyzed functional connectivity of these areas during the resting state. Results revealed a significant connection between RSC and frontal association areas (known to be involved in perceptual decision-making). In contrast, TOS showed functional connections dorsally with the Inferior Parietal Sulcus, and ventrally with the Lateral Occipital Complex--but not with RSC and/or frontal association areas. Moreover, RSC and TOS showed differentiable functional connections with the anterior-medial and posterior-lateral parts of PPA, respectively. These results suggest two parallel pathways for spatial encoding, including RSC and TOS respectively. Only the RSC network was involved in active spatial comparisons.

Neural representations of contextual guidance in visual search of real-world scenes

Exploiting scene context and object-object co-occurrence is critical in guiding eye movements and facilitating visual search, yet the mediating neural mechanisms are unknown. We used functional magnetic resonance imaging while observers searched for target objects in scenes and used multivariate pattern analyses (MVPA) to show that the lateral occipital complex (LOC) can predict the coarse spatial location of observers' expectations about the likely location of 213 different targets absent from the scenes. In addition, we found weaker but significant representations of context location in an area related to the orienting of attention (intraparietal sulcus, IPS) as well as a region related to scene processing (retrosplenial cortex, RSC). Importantly, the degree of agreement among 100 independent raters about the likely location to contain a target object in a scene correlated with LOC's ability to predict the contextual location while weaker but significant effects were found in IPS, RSC, the human motion area, and early visual areas (V1, V3v). When contextual information was made irrelevant to observers' behavioral task, the MVPA analysis of LOC and the other areas' activity ceased to predict the location of context. Thus, our findings suggest that the likely locations of targets in scenes are represented in various visual areas with LOC playing a key role in contextual guidance during visual search of objects in real scenes.

Functional activation during reading in relation to the sulci of the angular gyrus region

Neurological studies suggest that the angular gyrus region of the inferior parietal lobule may be critical for reading. However, unambiguous demonstration of angular gyrus involvement from lesion and functional neuroimaging studies is lacking, partly because of the absence of detailed morphological descriptions of this region. On the basis of our recent anatomical examination of this region and a tightly controlled functional magnetic resonance imaging paradigm, the present investigation demonstrated reading-related activity in the region of the angular gyrus that lies between the central and posterior branches of the caudal superior temporal sulcus, namely cytoarchitectonic area PG. Analysis of functional connectivity showed increased functional coupling during reading of area PG with the language areas of Broca and Wernicke, and a region previously identified as the visual word form area. Thus, the parietal reading area has been precisely localized, and its interactions with other cortical areas during reading have been demonstrated.

Transcranial Magnetic Stimulation to the Transverse Occipital Sulcus Affects Scene but Not Object Processing

Traditionally, it has been theorized that the human visual system identifies and classifies scenes in an object-centered approach, such that scene recognition can only occur once key objects within a scene are identified. Recent research points toward an alternative approach, suggesting that the global image features of a scene are sufficient for the recognition and categorization of a scene. We have previously shown that disrupting object processing with repetitive TMS to object-selective cortex enhances scene processing possibly through a release of inhibitory mechanisms between object and scene pathways [Mullin, C. R., & Steeves, J. K. E. TMS to the lateral occipital cortex disrupts object processing but facilitates scene processing. Journal of Cognitive Neuroscience, 23, 4174–4184, 2011]. Here we show the effects of TMS to the transverse occipital sulcus (TOS), an area implicated in scene perception, on scene and object processing. TMS was delivered to the TOS or the vertex (control site) while participants performed an object and scene natural/nonnatural categorization task. Transiently interrupting the TOS resulted in significantly lower accuracies for scene categorization compared with control conditions. This demonstrates a causal role of the TOS in scene processing and indicates its importance, in addition to the parahippocampal place area and retrosplenial cortex, in the scene processing network. Unlike TMS to object-selective cortex, which facilitates scene categorization, disrupting scene processing through stimulation of the TOS did not affect object categorization. Further analysis revealed a higher proportion of errors for nonnatural scenes that led us to speculate that the TOS may be involved in processing the higher spatial frequency content of a scene. This supports a nonhierarchical model of scene recognition.

Early spatial frequency processing of natural images: an ERP study

The present study examined the role of spatial stimulus frequencies in the early visual processing of natural scenes. The content of initially degraded (low- or high-pass filtered) pictures was progressively revealed in a sequence of steps by adding high or low spatial frequencies. Event Related Potentials (ERPs) were used to track the early stages of visual processing. Picture degradation modulated the topography of the P1, with an occipital midline distribution for the most degraded pictures, which became progressively more laterally distributed as pictures became more complete. Picture degradation also modulated the amplitude of the P2. For both low-passed and high-passed scenes, a linear relationship between the spectral power and the amplitude of the P1 and P2 was observed. These results are likely to reflect the progressive engagement of the lateral occipital complex as the amount of information in both the low and high portions of the frequency spectrum increased.

The distributed representation of random and meaningful object pairs in human occipitotemporal cortex: the weighted average as a general rule

Natural scenes typically contain multiple visual objects, often in interaction, such as when a bottle is used to fill a glass. Previous studies disagree about the representation of multiple objects and the role of object position herein, nor did they pinpoint the effect of potential interactions between the objects. In an fMRI study, we presented four single objects in two different positions and object pairs consisting of all possible combinations of the single objects. Objects pairs could form either a meaningful action configuration in which they interact with each other or a non-meaningful configuration. We found that for single objects and object pairs both identity and position were represented in multi-voxel activity patterns in LOC. The response patterns of object pairs were best predicted by a weighted average of the response patterns of the constituent objects, with the strongest single-object response (the max response) weighted more than the min response. The difference in weight between the max and the min object was larger for familiar action pairs than for other pairs when participants attended to the configuration. A weighted average thus relates the response patterns of object pairs to the response patterns of single objects, even when the objects interact.

Deconstructing visual scenes in cortex: gradients of object and spatial layout information

Real-world visual scenes are complex cluttered, and heterogeneous stimuli engaging scene- and object-selective cortical regions including parahippocampal place area (PPA), retrosplenial complex (RSC), and lateral occipital complex (LOC). To understand the unique contribution of each region to distributed scene representations, we generated predictions based on a neuroanatomical framework adapted from monkey and tested them using minimal scenes in which we independently manipulated both spatial layout (open, closed, and gradient) and object content (furniture, e.g., bed, dresser). Commensurate with its strong connectivity with posterior parietal cortex, RSC evidenced strong spatial layout information but no object information, and its response was not even modulated by object presence. In contrast, LOC, which lies within the ventral visual pathway, contained strong object information but no background information. Finally, PPA, which is connected with both the dorsal and the ventral visual pathway, showed information about both objects and spatial backgrounds and was sensitive to the presence or absence of either. These results suggest that 1) LOC, PPA, and RSC have distinct representations, emphasizing different aspects of scenes, 2) the specific representations in each region are predictable from their patterns of connectivity, and 3) PPA combines both spatial layout and object information as predicted by connectivity.

The benefit of object interactions arises in the lateral occipital cortex independent of attentional modulation from the intraparietal sulcus: a transcranial magnetic stimulation study

Our visual experience is generally not of isolated objects, but of scenes, where multiple objects are interacting. Such interactions (e.g., a watering can positioned to pour water toward a plant) have been shown to facilitate object identification compared with when the objects are depicted as not interacting (e.g., a watering can positioned away from the plant) (Green and Hummel, 2004, 2006). What is the neural basis for this advantage? Recent fMRI studies have identified the lateral occipital cortex (LO) as a potential neural origin of this behavioral benefit, as LO showed greater responses to object pairs depicted as interacting compared with when they are not (Kim and Biederman, 2010; Roberts and Humphreys, 2010). However, it is possible that LO was modulated by an attention-sensitive region, the intraparietal sulcus (IPS), which sometimes showed a similar pattern of responses as that of LO in the Kim and Biederman (2010) investigation. To test this hypothesis, we delivered transcranial magnetic stimulation (TMS) to human subjects' LO and IPS while they detected a target object that was or was not interacting with another object to form a scene. TMS delivered to LO but not IPS abolished the facilitation in identifying interacting objects compared with noninteracting depictions observed in the absence of TMS, suggesting that it is LO and not IPS that is critical for the coding of object interactions.

Simple line drawings suffice for functional MRI decoding of natural scene categories

Humans are remarkably efficient at categorizing natural scenes. In fact, scene categories can be decoded from functional MRI (fMRI) data throughout the ventral visual cortex, including the primary visual cortex, the parahippocampal place area (PPA), and the retrosplenial cortex (RSC). Here we ask whether, and where, we can still decode scene category if we reduce the scenes to mere lines. We collected fMRI data while participants viewed photographs and line drawings of beaches, city streets, forests, highways, mountains, and offices. Despite the marked difference in scene statistics, we were able to decode scene category from fMRI data for line drawings just as well as from activity for color photographs, in primary visual cortex through PPA and RSC. Even more remarkably, in PPA and RSC, error patterns for decoding from line drawings were very similar to those from color photographs. These data suggest that, in these regions, the information used to distinguish scene category is similar for line drawings and photographs. To determine the relative contributions of local and global structure to the human ability to categorize scenes, we selectively removed long or short contours from the line drawings. In a category-matching task, participants performed significantly worse when long contours were removed than when short contours were removed. We conclude that global scene structure, which is preserved in line drawings, plays an integral part in representing scene categories.