Semantic control of feature extraction from natural scenes

Human sensory adaptation to the ecological structure of environmental statistics

Humans acquire sensory information via fast, highly specialized detectors: For example, edge detectors monitor restricted regions of visual space over timescales of 100-200 ms. Surprisingly, this study demonstrates that their operation is nevertheless shaped by the ecological consistency of slow global statistical structure in the environment. In the experiments, humans acquired feature information from brief localized elements embedded within a virtual environment. Cast shadows are important for determining the appearance and layout of the environment. When the statistical reliability of shadows was manipulated, human feature detectors implicitly adapted to these changes over minutes, adjusting their response properties to emphasize either "image-based" or "object-based" anchoring of local visual elements. More specifically, local visual operators were more firmly anchored around object representations when shadows were reliable. As shadow reliability was reduced, visual operators disengaged from objects and became anchored around image features. These results indicate that the notion of sensory adaptation must be reframed around complex statistical constructs with ecological validity. These constructs far exceed the spatiotemporal selectivity bandwidth of sensory detectors, thus demonstrating the highly integrated nature of sensory processing during natural behavior.

The influence of natural image statistics on upright orientation judgements

Humans have well-documented priors for many features present in nature that guide visual perception. Despite being putatively grounded in the statistical regularities of the environment, scene priors are frequently violated due to the inherent variability of visual features from one scene to the next. However, these repeated violations do not appreciably challenge visuo-cognitive function, necessitating the broad use of priors in conjunction with context-specific information. We investigated the trade-off between participants' internal expectations formed from both longer-term priors and those formed from immediate contextual information using a perceptual inference task and naturalistic stimuli. Notably, our task required participants to make perceptual inferences about naturalistic images using their own internal criteria, rather than making comparative judgements. Nonetheless, we show that observers' performance is well approximated by a model that makes inferences using a prior for low-level image statistics, aggregated over many images. We further show that the dependence on this prior is rapidly re-weighted against contextual information, even when misleading. Our results therefore provide insight into how apparent high-level interpretations of scene appearances follow from the most basic of perceptual processes, which are grounded in the statistics of natural images.

Characterising and dissecting human perception of scene complexity

Humans can effortlessly assess the complexity of the visual stimuli they encounter. However, our understanding of how we do this, and the relevant factors that result in our perception of scene complexity remain unclear; especially for the natural scenes in which we are constantly immersed. We introduce several new datasets to further understanding of human perception of scene complexity. Our first dataset (VISC-C) contains 800 scenes and 800 corresponding two-dimensional complexity annotations gathered from human observers, allowing exploration for how complexity perception varies across a scene. Our second dataset, (VISC-CI) consists of inverted scenes (reflection on the horizontal axis) with corresponding complexity maps, collected from human observers. Inverting images in this fashion is associated with destruction of semantic scene characteristics when viewed by humans, and hence allows analysis of the impact of semantics on perceptual complexity. We analysed perceptual complexity from both a single-score and a two-dimensional perspective, by evaluating a set of calculable and observable perceptual features based upon grounded psychological research (clutter, symmetry, entropy and openness). We considered these factors' relationship to complexity via hierarchical regressions analyses, tested the efficacy of various neural models against our datasets, and validated our perceptual features against a large and varied complexity dataset consisting of nearly 5000 images. Our results indicate that both global image properties and semantic features are important for complexity perception. We further verified this by combining identified perceptual features with the output of a neural network predictor capable of extracting semantics, and found that we could increase the amount of explained human variance in complexity beyond that of low-level measures alone. Finally, we dissect our best performing prediction network, determining that artificial neurons learn to extract both global image properties and semantic details from scenes for complexity prediction. Based on our experimental results, we propose the "dual information" framework of complexity perception, hypothesising that humans rely on both low-level image features and high-level semantic content to evaluate the complexity of images.

Luminance and Contrast of Images in the THINGS Database

The THINGS database is a freely available stimulus set that has the potential to facilitate the generation of theory that bridges multiple areas within cognitive neuroscience. The database consists of 26,107 high quality digital photos that are sorted into 1,854 concepts. While a valuable resource, relatively few technical details relevant to the design of studies in cognitive neuroscience have been described. We present an analysis of two key low-level properties of THINGS images, luminance and luminance contrast. These image statistics are known to influence common physiological and neural correlates of perceptual and cognitive processes. In general, we found that the distributions of luminance and contrast are in close agreement with the statistics of natural images reported previously. However, we found that image concepts are separable in their luminance and contrast: we show that luminance and contrast alone are sufficient to classify images into their concepts with above chance accuracy. We describe how these factors may confound studies using the THINGS images, and suggest simple controls that can be implemented a priori or post-hoc. We discuss the importance of using such natural images as stimuli in psychological research.

Redundancy between spectral and higher-order texture statistics for natural image segmentation

Visual texture, defined by local image statistics, provides important information to the human visual system for perceptual segmentation. Second-order or spectral statistics (equivalent to the Fourier power spectrum) are a well-studied segmentation cue. However, the role of higher-order statistics (HOS) in segmentation remains unclear, particularly for natural images. Recent experiments indicate that, in peripheral vision, the HOS of the widely adopted Portilla-Simoncelli texture model are a weak segmentation cue compared to spectral statistics, despite the fact that both are necessary to explain other perceptual phenomena and to support high-quality texture synthesis. Here we test whether this discrepancy reflects a property of natural image statistics. First, we observe that differences in spectral statistics across segments of natural images are redundant with differences in HOS. Second, using linear and nonlinear classifiers, we show that each set of statistics individually affords high performance in natural scenes and texture segmentation tasks, but combining spectral statistics and HOS produces relatively small improvements. Third, we find that HOS improve segmentation for a subset of images, although these images are difficult to identify. We also find that different subsets of HOS improve segmentation to a different extent, in agreement with previous physiological and perceptual work. These results show that the HOS add modestly to spectral statistics for natural image segmentation. We speculate that tuning to natural image statistics under resource constraints could explain the weak contribution of HOS to perceptual segmentation in human peripheral vision.

A deep-learning framework for human perception of abstract art composition

Artistic composition (the structural organization of pictorial elements) is often characterized by some basic rules and heuristics, but art history does not offer quantitative tools for segmenting individual elements, measuring their interactions and related operations. To discover whether a metric description of this kind is even possible, we exploit a deep-learning algorithm that attempts to capture the perceptual mechanism underlying composition in humans. We rely on a robust behavioral marker with known relevance to higher-level vision: orientation judgements, that is, telling whether a painting is hung “right-side up.” Humans can perform this task, even for abstract paintings. To account for this finding, existing models rely on “meaningful” content or specific image statistics, often in accordance with explicit rules from art theory. Our approach does not commit to any such assumptions/schemes, yet it outperforms previous models and for a larger database, encompassing a wide range of painting styles. Moreover, our model correctly reproduces human performance across several measurements from a new web-based experiment designed to test whole paintings, as well as painting fragments matched to the receptive-field size of different depths in the model. By exploiting this approach, we show that our deep learning model captures relevant characteristics of human orientation perception across styles and granularities. Interestingly, the more abstract the painting, the more our model relies on extended spatial integration of cues, a property supported by deeper layers.

Mechanisms of Spectrotemporal Modulation Detection for Normal- and Hearing-Impaired Listeners

Spectrotemporal modulations (STM) are essential features of speech signals that make them intelligible. While their encoding has been widely investigated in neurophysiology, we still lack a full understanding of how STMs are processed at the behavioral level and how cochlear hearing loss impacts this processing. Here, we introduce a novel methodological framework based on psychophysical reverse correlation deployed in the modulation space to characterize the mechanisms underlying STM detection in noise. We derive perceptual filters for young normal-hearing and older hearing-impaired individuals performing a detection task of an elementary target STM (a given product of temporal and spectral modulations) embedded in other masking STMs. Analyzed with computational tools, our data show that both groups rely on a comparable linear (band-pass)-nonlinear processing cascade, which can be well accounted for by a temporal modulation filter bank model combined with cross-correlation against the target representation. Our results also suggest that the modulation mistuning observed for the hearing-impaired group results primarily from broader cochlear filters. Yet, we find idiosyncratic behaviors that cannot be captured by cochlear tuning alone, highlighting the need to consider variability originating from additional mechanisms. Overall, this integrated experimental-computational approach offers a principled way to assess suprathreshold processing distortions in each individual and could thus be used to further investigate interindividual differences in speech intelligibility.

Common spatiotemporal processing of visual features shapes object representation

Biological vision relies on representations of the physical world at different levels of complexity. Relevant features span from simple low-level properties, as contrast and spatial frequencies, to object-based attributes, as shape and category. However, how these features are integrated into coherent percepts is still debated. Moreover, these dimensions often share common biases: for instance, stimuli from the same category (e.g., tools) may have similar shapes. Here, using magnetoencephalography, we revealed the temporal dynamics of feature processing in human subjects attending to objects from six semantic categories. By employing Relative Weights Analysis, we mitigated collinearity between model-based descriptions of stimuli and showed that low-level properties (contrast and spatial frequencies), shape (medial-axis) and category are represented within the same spatial locations early in time: 100–150 ms after stimulus onset. This fast and overlapping processing may result from independent parallel computations, with categorical representation emerging later than the onset of low-level feature processing, yet before shape coding. Categorical information is represented both before and after shape, suggesting a role for this feature in the refinement of categorical matching.

Presaccadic attention improves or impairs performance by enhancing sensitivity to higher spatial frequencies

Right before we move our eyes, visual performance and neural responses for the saccade target are enhanced. This effect, presaccadic attention, is considered to prioritize the saccade target and to enhance behavioral performance for the saccade target. Recent evidence has shown that presaccadic attention modulates the processing of feature information. Hitherto, it remains unknown whether presaccadic modulations on feature information are flexible, to improve performance for the task at hand, or automatic, so that they alter the featural representation similarly regardless of the task. Using a masking procedure, here we report that presaccadic attention can either improve or impair performance depending on the spatial frequency content of the visual input. These counterintuitive modulations were significant at a time window right before saccade onset. Furthermore, merely deploying covert attention within the same temporal interval without preparing a saccade did not affect performance. This study reveals that presaccadic attention not only prioritizes the saccade target, but also automatically modifies its featural representation.

The empirical characteristics of human pattern vision defy theoretically-driven expectations

Contrast is the most fundamental property of images. Consequently, any comprehensive model of biological vision must incorporate this attribute and provide a veritable description of its impact on visual perception. Current theoretical and computational models predict that vision should modify its characteristics at low contrast: for example, it should become broader (more lowpass) to protect from noise, as often demonstrated by individual neurons. We find that the opposite is true for human discrimination of elementary image elements: vision becomes sharper, not broader, as contrast approaches threshold levels. Furthermore, it suffers from increased internal variability at low contrast and it transitions from a surprisingly linear regime at high contrast to a markedly nonlinear processing mode in the low-contrast range. These characteristics are hard-wired in that they happen on a single trial without memory or expectation. Overall, the empirical results urge caution when attempting to interpret human vision from the standpoint of optimality and related theoretical constructs. Direct measurements of this phenomenon indicate that the actual constraints derive from intrinsic architectural features, such as the co-existence of complex-cell-like and simple-cell-like components. Small circuits built around these elements can indeed account for the empirical results, but do not appear to operate in a manner that conforms to optimality even approximately. More generally, our results provide a compelling demonstration of how far we still are from securing an adequate computational account of the most basic operations carried out by human vision.

Prior object-knowledge sharpens properties of early visual feature-detectors

Early stages of visual processing are carried out by neural circuits activated by simple and specific features, such as the orientation of an edge. A fundamental question in human vision is how the brain organises such intrinsically local information into meaningful properties of objects. Classic models of visual processing emphasise a one-directional flow of information from early feature-detectors to higher-level information-processing. By contrast to this view, and in line with predictive-coding models of perception, here, we provide evidence from human vision that high-level object representations dynamically interact with the earliest stages of cortical visual processing. In two experiments, we used ambiguous stimuli that, depending on the observer’s prior object-knowledge, can be perceived as either coherent objects or as a collection of meaningless patches. By manipulating object knowledge we were able to determine its impact on processing of low-level features while keeping sensory stimulation identical. Both studies demonstrate that perception of local features is facilitated in a manner consistent with an observer’s high-level object representation (i.e., with no effect on object-inconsistent features). Our results cannot be ascribed to attentional influences. Rather, they suggest that high-level object representations interact with and sharpen early feature-detectors, optimising their performance for the current perceptual context.

Diagnostic and prognostic role of semantic processing in preclinical Alzheimer's disease

Relatively spared during most of the timeline of normal aging, semantic memory shows a subtle yet measurable decline even during the pre-clinical stage of Alzheimer's disease. This decline is thought to reflect early neurofibrillary changes and impairment is detectable using tests of language relying on lexical-semantic abilities. A promising approach is the characterization of semantic parameters such as typicality and age of acquisition of words, and propositional density from verbal output. Seminal research like the Nun Study or the analysis of the linguistic decline of famous writers and politicians later diagnosed with Alzheimer's disease supports the early diagnostic value of semantic processing and semantic memory. Moreover, measures of these skills may play an important role for the prognosis of patients with mild cognitive impairment.

Object Recognition: Complexity of Recognition Strategies

Primate brains and state-of-the-art convolutional neural networks can recognize many faces, objects and scenes, though how they do so is often mysterious. New research unveils some of the mystery, revealing unexpected complexity in the recognition strategies of rodents.

Object segmentation controls image reconstruction from natural scenes

The structure of the physical world projects images onto our eyes. However, those images are often poorly representative of environmental structure: well-defined boundaries within the eye may correspond to irrelevant features of the physical world, while critical features of the physical world may be nearly invisible at the retinal projection. The challenge for the visual cortex is to sort these two types of features according to their utility in ultimately reconstructing percepts and interpreting the constituents of the scene. We describe a novel paradigm that enabled us to selectively evaluate the relative role played by these two feature classes in signal reconstruction from corrupted images. Our measurements demonstrate that this process is quickly dominated by the inferred structure of the environment, and only minimally controlled by variations of raw image content. The inferential mechanism is spatially global and its impact on early visual cortex is fast. Furthermore, it retunes local visual processing for more efficient feature extraction without altering the intrinsic transduction noise. The basic properties of this process can be partially captured by a combination of small-scale circuit models and large-scale network architectures. Taken together, our results challenge compartmentalized notions of bottom-up/top-down perception and suggest instead that these two modes are best viewed as an integrated perceptual mechanism.

Biological vision is designed to discover the structure of the environment around us. To do this, it relies on ambiguous and often misleading information from the eyes: the boundary of a critical object may be invisible against a background of similar appearance, and may be overlooked in favour of the sharp contour projected by an irrelevant shadow. It remains unclear how human vision sorts different image features according to their relevance to the layout of objects within the scene. We demonstrate that vision achieves this goal via a specialized perceptual system for object segmentation that is one and the same with the feature extraction system: immediately after information is relayed to cortex by the eyes, the process of reconstructing image content from local features is controlled by a dedicated inferential mechanism that attempts to recover the underlying environmental structure; perception is quickly organized around the operation of this mechanism, which becomes the primary contextual influence on image reconstruction. The integrated nature of this perceptual mechanism defies current notions of separate top-down and bottom-up processes, offering a fresh view of how human vision operates on natural signals.

No Evidence of Narrowly Defined Cognitive Penetrability in Unambiguous Vision

The classical notion of cognitive impenetrability suggests that perceptual processing is an automatic modular system and not under conscious control. Near consensus is now emerging that this classical notion is untenable. However, as recently pointed out by Firestone and Scholl, this consensus is built on quicksand. In most studies claiming perception is cognitively penetrable, it remains unclear which actual process has been affected (perception, memory, imagery, input selection or judgment). In fact, the only available "proofs" for cognitive penetrability are proxies for perception, such as behavioral responses and neural correlates. We suggest that one can interpret cognitive penetrability in two different ways, a broad sense and a narrow sense. In the broad sense, attention and memory are not considered as "just" pre- and post-perceptual systems but as part of the mechanisms by which top-down processes influence the actual percept. Although many studies have proven top-down influences in this broader sense, it is still debatable whether cognitive penetrability remains tenable in a narrow sense. The narrow sense states that cognitive penetrability only occurs when top-down factors are flexible and cause a clear illusion from a first person perspective. So far, there is no strong evidence from a first person perspective that visual illusions can indeed be driven by high-level flexible factors. One cannot be cognitively trained to see and unsee visual illusions. We argue that this lack of convincing proof for cognitive penetrability in the narrow sense can be explained by the fact that most research focuses on foveal vision only. This type of perception may be too unambiguous for transient high-level factors to control perception. Therefore, illusions in more ambiguous perception, such as peripheral vision, can offer a unique insight into the matter. They produce a clear subjective percept based on unclear, degraded visual input: the optimal basis to study narrowly defined cognitive penetrability.

Changing What You See by Changing What You Know: The Role of Attention

Attending is a cognitive process that incorporates a person's knowledge, goals, and expectations. What we perceive when we attend to one thing is different from what we perceive when we attend to something else. Yet, it is often argued that attentional effects do not count as evidence that perception is influenced by cognition. I investigate two arguments often given to justify excluding attention. The first is arguing that attention is a post-perceptual process reflecting selection between fully constructed perceptual representations. The second is arguing that attention as a pre-perceptual process that simply changes the input to encapsulated perceptual systems. Both of these arguments are highly problematic. Although some attentional effects can indeed be construed as post-perceptual, others operate by changing perceptual content across the entire visual hierarchy. Although there is a natural analogy between spatial attention and a change of input, the analogy falls apart when we consider other forms of attention. After dispelling these arguments, I make a case for thinking of attention not as a confound, but as one of the mechanisms by which cognitive states affect perception by going through cases in which the same or similar visual inputs are perceived differently depending on the observer's cognitive state, and instances where cuing an observer using language affects what one sees. Lastly, I provide two compelling counter-examples to the critique that although cognitive influences on perception can be demonstrated in the laboratory, it is impossible to really experience them for oneself in a phenomenologically compelling way. Taken together, the current evidence strongly supports the thesis that what we know routinely influences what we see, that the same sensory input can be perceived differently depending on the current cognitive state of the viewer, and that phenomenologically salient demonstrations are possible if certain conditions are met.

Dynamic Reweighting of Auditory Modulation Filters

Sound waveforms convey information largely via amplitude modulations (AM). A large body of experimental evidence has provided support for a modulation (bandpass) filterbank. Details of this model have varied over time partly reflecting different experimental conditions and diverse datasets from distinct task strategies, contributing uncertainty to the bandwidth measurements and leaving important issues unresolved. We adopt here a solely data-driven measurement approach in which we first demonstrate how different models can be subsumed within a common 'cascade' framework, and then proceed to characterize the cascade via system identification analysis using a single stimulus/task specification and hence stable task rules largely unconstrained by any model or parameters. Observers were required to detect a brief change in level superimposed onto random level changes that served as AM noise; the relationship between trial-by-trial noisy fluctuations and corresponding human responses enables targeted identification of distinct cascade elements. The resulting measurements exhibit a dynamic complex picture in which human perception of auditory modulations appears adaptive in nature, evolving from an initial lowpass to bandpass modes (with broad tuning, Q∼1) following repeated stimulus exposure.

Saccade Preparation Reshapes Sensory Tuning

Human observers make large rapid eye movements-saccades-to bring behaviorally relevant information into the fovea, where spatial resolution is high. In some visual tasks [1-4], performance at the location of a saccade target improves before the eyes move. Although these findings provide evidence that extra-retinal signals evoked by saccades can enhance visual perception, it remains unknown whether and how presaccadic modulations change the processing of feature information and thus modulate visual representations. To answer this question, one must go beyond the use of methods that only probe performance accuracy (d') in different tasks. Here, using a psychophysical reverse correlation approach [5-8], we investigated how saccade preparation influences the processing of orientation and spatial frequency-two building blocks of early vision. We found that saccade preparation selectively enhanced the gain of high spatial frequency information and narrowed orientation tuning at the upcoming saccade landing position. These modulations were time locked to saccade onset, peaking right before the eyes moved (-50-0 ms). Moreover, merely deploying covert attention within the same temporal interval without preparing a saccade did not alter performance. The observed presaccadic tuning changes may correspond to the presaccadic enhancement [9-11] and receptive field shifts reported in neurophysiological studies [12-14]. Saccade preparation may support transaccadic integration by reshaping the representation of the saccade target to be more fovea-like just before the eyes move. The presaccadic modulations on spatial frequency and orientation processing illustrate a strong perception-action coupling by revealing that the visual system dynamically reshapes feature selectivity contingent upon eye movements.

How to (and how not to) think about top-down influences on visual perception

The question of whether cognition can influence perception has a long history in neuroscience and philosophy. Here, we outline a novel approach to this issue, arguing that it should be viewed within the framework of top-down information-processing. This approach leads to a reversal of the standard explanatory order of the cognitive penetration debate: we suggest studying top-down processing at various levels without preconceptions of perception or cognition. Once a clear picture has emerged about which processes have influences on those at lower levels, we can re-address the extent to which they should be considered perceptual or cognitive. Using top-down processing within the visual system as a model for higher-level influences, we argue that the current evidence indicates clear constraints on top-down influences at all stages of information processing; it does, however, not support the notion of a boundary between specific types of information-processing as proposed by the cognitive impenetrability hypothesis.

Shift toward prior knowledge confers a perceptual advantage in early psychosis and psychosis-prone healthy individuals

Many neuropsychiatric illnesses are associated with psychosis, i.e., hallucinations (perceptions in the absence of causative stimuli) and delusions (irrational, often bizarre beliefs). Current models of brain function view perception as a combination of two distinct sources of information: bottom-up sensory input and top-down influences from prior knowledge. This framework may explain hallucinations and delusions. Here, we characterized the balance between visual bottom-up and top-down processing in people with early psychosis (study 1) and in psychosis-prone, healthy individuals (study 2) to elucidate the mechanisms that might contribute to the emergence of psychotic experiences. Through a specialized mental-health service, we identified unmedicated individuals who experience early psychotic symptoms but fall below the threshold for a categorical diagnosis. We observed that, in early psychosis, there was a shift in information processing favoring prior knowledge over incoming sensory evidence. In the complementary study, we capitalized on subtle variations in perception and belief in the general population that exhibit graded similarity with psychotic experiences (schizotypy). We observed that the degree of psychosis proneness in healthy individuals, and, specifically, the presence of subtle perceptual alterations, is also associated with stronger reliance on prior knowledge. Although, in the current experimental studies, this shift conferred a performance benefit, under most natural viewing situations, it may provoke anomalous perceptual experiences. Overall, we show that early psychosis and psychosis proneness both entail a basic shift in visual information processing, favoring prior knowledge over incoming sensory evidence. The studies provide complementary insights to a mechanism by which psychotic symptoms may emerge.

Receptive fields of visual neurons: the early years

This paper traces the history of the visual receptive field (RF) from Hartline to Hubel and Wiesel. Hartline (1938, 1940) found that an isolated optic nerve fiber in the frog could be excited by light falling on a small circular area of the retina. He called this area the RF, using a term first introduced by Sherrington (1906) in the tactile domain. In 1953 Kuffler discovered the antagonistic center-surround organization of cat RFs, and Barlow, Fitzhugh, and Kuffler (1957) extended this work to stimulus size and state of adaptation. Shortly thereafter, Lettvin and colleagues (1959) in an iconic paper asked "what the frog's eye tells the frog's brain". Meanwhile, Jung and colleagues (1952-1973) searched for the perceptual correlates of neuronal responses, and Jung and Spillmann (1970) proposed the term perceptive field (PF) as a psychophysical correlate of the RF. The Westheimer function (1967) enabled psychophysical measurements of the PF center and surround in human and monkey, which correlated closely with the underlying RF organization. The sixties and seventies were marked by rapid progress in RF research. Hubel and Wiesel (1959-1974), recording from neurons in the visual cortex of the cat and monkey, found elongated RFs selective for the shape, orientation, and position of the stimulus, as well as for movement direction and ocularity. These findings prompted the emergence in visual psychophysics of the concept of feature detectors selective for lines, bars, and edges, and contributed to a model of the RF in terms of difference of Gaussians (DOG) and Fourier channels. The distinction between simple, complex, and hypercomplex neurons followed. Although RF size increases towards the peripheral retina, its cortical representation remains constant due to the reciprocal relationship with the cortical magnification factor (M). This constitutes a uniform yardstick for M-scaled stimuli across the retina. Developmental studies have shown that RF properties are not fixed. RFs possess their full response inventory already at birth, but require the interaction with appropriate stimuli within a critical time window for refinement and consolidation. Taken together these findings paved the way for a better understanding of how objective properties of the external world are encoded to become subjective properties of the subjective, perceptual world.

The neural basis of perceiving person interactions

This study examined whether the grouping of people into meaningful social scenes (e.g., two people having a chat) impacts the basic perceptual analysis of each partaking individual. To explore this issue, we measured neural activity using functional magnetic resonance imaging (fMRI) while participants sex-categorized congruent as well as incongruent person dyads (i.e., two people interacting in a plausible or implausible manner). Incongruent person dyads elicited enhanced neural processing in several high-level visual areas dedicated to face and body encoding and in the posterior middle temporal gyrus compared to congruent person dyads. Incongruent and congruent person scenes were also successfully differentiated by a linear multivariate pattern classifier in the right fusiform body area and the left extrastriate body area. Finally, increases in the person scenes' meaningfulness as judged by independent observers was accompanied by enhanced activity in the bilateral posterior insula. These findings demonstrate that the processing of person scenes goes beyond a mere stimulus-bound encoding of their partaking agents, suggesting that changes in relations between agents affect their representation in category-selective regions of the visual cortex and beyond.

Inversion improves the recognition of facial expression in thatcherized images

The Thatcher illusion provides a compelling example of the face inversion effect. However, the marked effect of inversion in the Thatcher illusion contrasts to other studies that report only a small effect of inversion on the recognition of facial expressions. To address this discrepancy, we compared the effects of inversion and thatcherization on the recognition of facial expressions. We found that inversion of normal faces caused only a small reduction in the recognition of facial expressions. In contrast, local inversion of facial features in upright thatcherized faces resulted in a much larger reduction in the recognition of facial expressions. Paradoxically, inversion of thatcherized faces caused a relative increase in the recognition of facial expressions. Together, these results suggest that different processes explain the effects of inversion on the recognition of facial expressions and on the perception of the Thatcher illusion. The grotesque perception of thatcherized images is based on a more orientation-sensitive representation of the face. In contrast, the recognition of facial expression is dependent on a more orientation-insensitive representation. A similar pattern of results was evident when only the mouth or eye region was visible. These findings demonstrate that a key component of the Thatcher illusion is to be found in orientation-specific encoding of the features of the face.

Orientation-sensitivity to facial features explains the Thatcher illusion

The Thatcher illusion provides a compelling example of the perceptual cost of face inversion. The Thatcher illusion is often thought to result from a disruption to the processing of spatial relations between face features. Here, we show the limitations of this account and instead demonstrate that the effect of inversion in the Thatcher illusion is better explained by a disruption to the processing of purely local facial features. Using a matching task, we found that participants were able to discriminate normal and Thatcherized versions of the same face when they were presented in an upright orientation, but not when the images were inverted. Next, we showed that the effect of inversion was also apparent when only the eye region or only the mouth region was visible. These results demonstrate that a key component of the Thatcher illusion is to be found in orientation-specific encoding of the expressive features (eyes and mouth) of the face.

Evidence for adjustable bandwidth orientation channels

The standard model of early vision claims that orientation and spatial frequency are encoded with multiple, quasi-independent channels that have fixed spatial frequency and orientation bandwidths. The standard model was developed using detection and discrimination data collected from experiments that used deterministic patterns such as Gabor patches and gratings used as stimuli. However, detection data from experiments using noise as a stimulus suggests that the visual system may use adjustable-bandwidth, rather than fixed-bandwidth, channels. In our previous work, we used classification images as a key piece of evidence against the hypothesis that pattern detection is based on the responses of channels with an adjustable spatial frequency bandwidth. Here we tested the hypothesis that channels with adjustable orientation bandwidths are used to detect two-dimensional, filtered noise targets that varied in orientation bandwidth and were presented in white noise. Consistent with our previous work that examined spatial frequency bandwidth, we found that detection thresholds were consistent with the hypothesis that observers sum information across a broad range of orientations nearly optimally: absolute efficiency for stimulus detection was 20-30% and approximately constant across a wide range of orientation bandwidths. Unlike what we found with spatial frequency bandwidth, the results of our classification image experiment were consistent with the hypothesis that the orientation bandwidth of internal filters were adjustable. Thus, for orientation summation, both detection thresholds and classification images support the adjustable channels hypothesis. Classification images also revealed hallmarks of inhibition or suppression from uninformative spatial frequencies and/or orientations. This work highlights the limitations of the standard model of summation for orientation. The standard model of orientation summation and tuning was chiefly developed with narrow-band stimuli that were not presented in noise, stimuli that are arguably less naturalistic than the variable bandwidth stimuli presented in noise used in our experiments. Finally, the disagreement between the results from our experiments on spatial frequency summation with the data presented in this paper suggests that orientation may be encoded more flexibly than spatial frequency channels.