Phase noise and the classification of natural images

Attention deployment in natural scenes: Higher-order scene statistics rather than semantics modulate the N2pc component

Which properties of a natural scene affect visual search? We consider the alternative hypotheses that low-level statistics, higher-level statistics, semantics, or layout affect search difficulty in natural scenes. Across three experiments (n = 20 each), we used four different backgrounds that preserve distinct scene properties: (a) natural scenes (all experiments); (b) 1/f noise (pink noise, which preserves only low-level statistics and was used in Experiments 1 and 2); (c) textures that preserve low-level and higher-level statistics but not semantics or layout (Experiments 2 and 3); and (d) inverted (upside-down) scenes that preserve statistics and semantics but not layout (Experiment 2). We included "split scenes" that contained different backgrounds left and right of the midline (Experiment 1, natural/noise; Experiment 3, natural/texture). Participants searched for a Gabor patch that occurred at one of six locations (all experiments). Reaction times were faster for targets on noise and slower on inverted images, compared to natural scenes and textures. The N2pc component of the event-related potential, a marker of attentional selection, had a shorter latency and a higher amplitude for targets in noise than for all other backgrounds. The background contralateral to the target had an effect similar to that on the target side: noise led to faster reactions and shorter N2pc latencies than natural scenes, although we observed no difference in N2pc amplitude. There were no interactions between the target side and the non-target side. Together, this shows that-at least when searching simple targets without own semantic content-natural scenes are more effective distractors than noise and that this results from higher-order statistics rather than from semantics or layout.

Unconscious Manipulation of Conceptual Representations with Decoded Neurofeedback Impacts Search Behavior

The necessity of conscious awareness in human learning has been a long-standing topic in psychology and neuroscience. Previous research on non-conscious associative learning is limited by the low signal-to-noise ratio of the subliminal stimulus, and the evidence remains controversial, including failures to replicate. Using functional MRI decoded neurofeedback, we guided participants from both sexes to generate neural patterns akin to those observed when visually perceiving real-world entities (e.g., dogs). Importantly, participants remained unaware of the actual content represented by these patterns. We utilized an associative DecNef approach to imbue perceptual meaning (e.g., dogs) into Japanese hiragana characters that held no inherent meaning for our participants, bypassing a conscious link between the characters and the dogs concept. Despite their lack of awareness regarding the neurofeedback objective, participants successfully learned to activate the target perceptual representations in the bilateral fusiform. The behavioral significance of our training was evaluated in a visual search task. DecNef and control participants searched for dogs or scissors targets that were pre-cued by the hiragana used during DecNef training or by a control hiragana. The DecNef hiragana did not prime search for its associated target but, strikingly, participants were impaired at searching for the targeted perceptual category. Hence, conscious awareness may function to support higher-order associative learning. Meanwhile, lower-level forms of re-learning, modification, or plasticity in existing neural representations can occur unconsciously, with behavioral consequences outside the original training context. The work also provides an account of DecNef effects in terms of neural representational drift.

Axial super-resolution optical coherence tomography via complex-valued network

Optical coherence tomography (OCT) is a fast and non-invasive optical interferometric imaging technique that can provide high-resolution cross-sectional images of biological tissues. OCT's key strength is its depth resolving capability which remains invariant along the imaging depth and is determined by the axial resolution. The axial resolution is inversely proportional to the bandwidth of the OCT light source. Thus, the use of broadband light sources can effectively improve the axial resolution and however leads to an increased cost. In recent years, real-valued deep learning technique has been introduced to obtain super-resolution optical imaging. In this study, we proposed a complex-valued super-resolution network (CVSR-Net) to achieve an axial super-resolution for OCT by fully utilizing the amplitude and phase of OCT signal. The method was evaluated on three OCT datasets. The results show that the CVSR-Net outperforms its real-valued counterpart with a better depth resolving capability. Furthermore, comparisons were made between our network, six prevailing real-valued networks and their complex-valued counterparts. The results demonstrate that the complex-valued network exhibited a better super-resolution performance than its real-valued counterpart and our proposed CVSR-Net achieved the best performance. In addition, the CVSR-Net was tested on out-of-distribution domain datasets and its super-resolution performance was well maintained as compared to that on source domain datasets, indicating a good generalization capability.

The ratio effect in visual numerosity comparisons is preserved despite spatial frequency equalisation

How non-symbolic numerosity is visually extracted remains a matter of intense debate. Most evidence suggests that numerosity is directly extracted on individual objects following Weber's law, at least for a moderate numerical range. Alternative accounts propose that, whatever the range, numerosity is indirectly derived from summary texture-statistics of the raw image such as spatial frequency (SF). Here, to disentangle these accounts, we tested whether the well-known behavioural signature of numerosity encoding (ratio effect) is preserved despite the equalisation of the SF content. In Experiment 1, participants had to select the numerically larger of two briefly presented moderate-range numerical sets (i.e., 8-18 dots) carefully matched for SF; the ratio between numerosities was manipulated by levels of increasing difficulty (e.g., 0.66, 0.75, 0.8). In Experiment 2, participants performed the same task, but they were presented with both the original and SF equalised stimuli. In both experiments, the results clearly showed a ratio-dependence of the performance: numerosity discrimination became harder and slower as the ratio between numerosities increased. Moreover, this effect was found to be independent of the stimulus type, although the overall performance was better with the original rather than the SF equalised stimuli (Experiment 2). Taken together, these findings indicate that the power spectrum per se cannot explain the main behavioural signature of Weber-like encoding of numerosities (the ratio effect), at least over the tested numerical range, partially challenging alternative indirect accounts of numerosity processing.

Synergy of spatial frequency and orientation bandwidth in texture segregation

Defining target textures by increased bandwidths in spatial frequency and orientation, we observed strong cue combination effects in a combined texture figure detection and discrimination task. Performance for double-cue targets was better than predicted by independent processing of either cue and even better than predicted from linear cue integration. Application of a texture-processing model revealed that the oversummative cue combination effect is captured by calculating a low-level summary statistic (\(\Delta CE_m\)), which describes the differential contrast energy to target and reference textures, from multiple scales and orientations, and integrating this statistic across channels with a winner-take-all rule. Modeling detection performance using a signal detection theory framework showed that the observers' sensitivity to single-cue and double-cue texture targets, measured in \(d^{\prime }\) units, could be reproduced with plausible settings for filter and noise parameters. These results challenge models assuming separate channeling of elementary features and their later integration, since oversummative cue combination effects appear as an inherent property of local energy mechanisms, at least for spatial frequency and orientation bandwidth-modulated textures.

Time will tell: Object categorization and emotional engagement during processing of degraded natural scenes

The aim of the present study was to examine the relationship between object categorization in natural scenes and the engagement of cortico-limbic appetitive and defensive systems (emotional engagement) by manipulating both the bottom-up information and the top-down context. Concerning the bottom-up information, we manipulated the computational load by scrambling the phase of the spatial frequency spectrum, and asked participants to classify natural scenes as containing an animal or a person. The role of the top-down context was assessed by comparing an incremental condition, in which pictures were progressively revealed, to a condition in which no probabilistic relationship existed between each stimulus and the following one. In two experiments, the categorization and response to emotional and neutral scenes were similarly modulated by the computational load. The Late Positive Potential (LPP) was affected by the emotional content of the scenes, and by categorization accuracy. When the phase of the spatial frequency spectrum was scrambled by a large amount (>58%), chance categorization resulted, and affective LPP modulation was eliminated. With less degraded scenes, categorization accuracy was higher (.82 in Experiment 1, .86 in Experiment 2) and affective modulation of the LPP was observed at a late window (>800 ms), indicating that it is possible to delay the time of engagement of the motivational systems which are responsible for the LPP affective modulation. The present data strongly support the view that semantic analysis of visual scenes, operationalized here as object categorization, is a necessary condition for emotional engagement at the electrocortical level (LPP).

Object Clusters or Spectral Energy? Assessing the Relative Contributions of Image Phase and Amplitude Spectra to Trypophobia

Trypophobia refers to the visual discomfort experienced by some people when viewing clustered patterns (e.g., clusters of holes). Trypophobic images deviate from the 1/f amplitude spectra typically characterizing natural images by containing excess energy at mid-range spatial frequencies. While recent work provides partial support for the idea of excess mid-range spatial frequency energy causing visual discomfort when viewing trypophobic images, a full factorial manipulation of image phase and amplitude spectra has yet to be conducted in order to determine whether the phase spectrum (sinusoidal waveform patterns that comprise image details like edge and texture elements) also plays a role in trypophobic discomfort. Here, we independently manipulated the phase and amplitude spectra of 31 Trypophobic images using a standard Fast Fourier Transform (FFT). Participants rated the four different versions of each image for levels of visual comfort, and completed the Trypophobia Questionnaire (TQ). Images having the original phase spectra intact (with either original or 1/f amplitude) explained the most variance in comfort ratings and were rated lowest in comfort. However, images with the original amplitude spectra but scrambled phase spectra were rated higher in comfort, with a smaller amount of variance in comfort attributed to the amplitude spectrum. Participant TQ scores correlated with comfort ratings only for images having the original phase spectra intact. There was no correlation between TQ scores and comfort levels when participants viewed the original amplitude / phase-scrambled images. Taken together, the present findings show that the phase spectrum of trypophobic images, which determines the pattern of small clusters of objects, plays a much larger role than the amplitude spectrum in determining visual discomfort.

The role of low-level image features in the affective categorization of rapidly presented scenes

It remains unclear how the visual system is able to extract affective content from complex scenes even with extremely brief (< 100 millisecond) exposures. One possibility, suggested by findings in machine vision, is that low-level features such as unlocalized, two-dimensional (2-D) Fourier spectra can be diagnostic of scene content. To determine whether Fourier image amplitude carries any information about the affective quality of scenes, we first validated the existence of image category differences through a support vector machine (SVM) model that was able to discriminate our intact aversive and neutral images with ~ 70% accuracy using amplitude-only features as inputs. This model allowed us to confirm that scenes belonging to different affective categories could be mathematically distinguished on the basis of amplitude spectra alone. The next question is whether these same features are also exploited by the human visual system. Subsequently, we tested observers' rapid classification of affective and neutral naturalistic scenes, presented briefly (~33.3 ms) and backward masked with synthetic textures. We tested categorization accuracy across three distinct experimental conditions, using: (i) original images, (ii) images having their amplitude spectra swapped within a single affective image category (e.g., an aversive image whose amplitude spectrum has been swapped with another aversive image) or (iii) images having their amplitude spectra swapped between affective categories (e.g., an aversive image containing the amplitude spectrum of a neutral image). Despite its discriminative potential, the human visual system does not seem to use Fourier amplitude differences as the chief strategy for affectively categorizing scenes at a glance. The contribution of image amplitude to affective categorization is largely dependent on interactions with the phase spectrum, although it is impossible to completely rule out a residual role for unlocalized 2-D amplitude measures.

Human sensitivity to perturbations constrained by a model of the natural image manifold

Humans are remarkably well tuned to the statistical properties of natural images. However, quantitative characterization of processing within the domain of natural images has been difficult because most parametric manipulations of a natural image make that image appear less natural. We used generative adversarial networks (GANs) to constrain parametric manipulations to remain within an approximation of the manifold of natural images. In the first experiment, seven observers decided which one of two synthetic perturbed images matched a synthetic unperturbed comparison image. Observers were significantly more sensitive to perturbations that were constrained to an approximate manifold of natural images than they were to perturbations applied directly in pixel space. Trial-by-trial errors were consistent with the idea that these perturbations disrupt configural aspects of visual structure used in image segmentation. In a second experiment, five observers discriminated paths along the image manifold as recovered by the GAN. Observers were remarkably good at this task, confirming that observers are tuned to fairly detailed properties of an approximate manifold of natural images. We conclude that human tuning to natural images is more general than detecting deviations from natural appearance, and that humans have, to some extent, access to detailed interrelations between natural images.

Asymmetric Compression of Representational Space for Object Animacy Categorization under Degraded Viewing Conditions

Animacy is a robust organizing principle among object category representations in the human brain. Using multivariate pattern analysis methods, it has been shown that distance to the decision boundary of a classifier trained to discriminate neural activation patterns for animate and inanimate objects correlates with observer RTs for the same animacy categorization task [Ritchie, J. B., Tovar, D. A., & Carlson, T. A. Emerging object representations in the visual system predict reaction times for categorization. PLoS Computational Biology, 11, e1004316, 2015; Carlson, T. A., Ritchie, J. B., Kriegeskorte, N., Durvasula, S., & Ma, J. Reaction time for object categorization is predicted by representational distance. Journal of Cognitive Neuroscience, 26, 132–142, 2014]. Using MEG decoding, we tested if the same relationship holds when a stimulus manipulation (degradation) increases task difficulty, which we predicted would systematically decrease the distance of activation patterns from the decision boundary and increase RTs. In addition, we tested whether distance to the classifier boundary correlates with drift rates in the linear ballistic accumulator [Brown, S. D., & Heathcote, A. The simplest complete model of choice response time: Linear ballistic accumulation. Cognitive Psychology, 57, 153–178, 2008]. We found that distance to the classifier boundary correlated with RT, accuracy, and drift rates in an animacy categorization task. Split by animacy, the correlations between brain and behavior were sustained longer over the time course for animate than for inanimate stimuli. Interestingly, when examining the distance to the classifier boundary during the peak correlation between brain and behavior, we found that only degraded versions of animate, but not inanimate, objects had systematically shifted toward the classifier decision boundary as predicted. Our results support an asymmetry in the representation of animate and inanimate object categories in the human brain.

Effect of silhouetting and inversion on view invariance in the monkey inferotemporal cortex

We easily recognize objects across changes in viewpoint, but the underlying features are unknown. Here, we show that view invariance in monkey inferotemporal cortex is driven mainly by external object contours and is not specialized for object orientation. We also find that the responses to natural objects match with that of their silhouettes early in the response, and with inverted versions later in the response—indicative of a coarse-to-fine processing sequence in the brain.

We effortlessly recognize objects across changes in viewpoint, but we know relatively little about the features that underlie viewpoint invariance in the brain. Here, we set out to characterize how viewpoint invariance in monkey inferior temporal (IT) neurons is influenced by two image manipulations—silhouetting and inversion. Reducing an object into its silhouette removes internal detail, so this would reveal how much viewpoint invariance depends on the external contours. Inverting an object retains but rearranges features, so this would reveal how much viewpoint invariance depends on the arrangement and orientation of features. Our main findings are 1) view invariance is weakened by silhouetting but not by inversion; 2) view invariance was stronger in neurons that generalized across silhouetting and inversion; 3) neuronal responses to natural objects matched early with that of silhouettes and only later to that of inverted objects, indicative of coarse-to-fine processing; and 4) the impact of silhouetting and inversion depended on object structure. Taken together, our results elucidate the underlying features and dynamics of view-invariant object representations in the brain.

NEW & NOTEWORTHY We easily recognize objects across changes in viewpoint, but the underlying features are unknown. Here, we show that view invariance in the monkey inferotemporal cortex is driven mainly by external object contours and is not specialized for object orientation. We also find that the responses to natural objects match with that of their silhouettes early in the response, and with inverted versions later in the response—indicative of a coarse-to-fine processing sequence in the brain.

Spatial Frequency Priming of Scene Perception in Adolescents With and Without ASD

While most typically developing (TD) participants have a coarse-to-fine processing style, people with autism spectrum disorder (ASD) seem to be less globally and more locally biased when processing visual information. The stimulus-specific spatial frequency content might be directly relevant to determine this temporal hierarchy of visual information processing in people with and without ASD. We implemented a semantic priming task in which (in)congruent coarse and/or fine spatial information preceded target categorization. Our results indicated that adolescents with ASD made more categorization errors than TD adolescents and needed more time to process the prime stimuli. Simultaneously, however, our findings argued for a processing advantage in ASD, when the prime stimulus contains detailed spatial information and presentation time permits explicit visual processing.

Non-uniform phase sensitivity in spatial frequency maps of the human visual cortex

KEY POINTS

Just as a portrait painting can come from a collection of coarse and fine details, natural vision can be decomposed into coarse and fine components. Previous studies have shown that the early visual areas in the brain represent these components in a map-like fashion. Other studies have shown that these same visual areas can be sensitive to how coarse and fine features line up in space. We found that the brain actually jointly represents both the scale of the feature (fine, medium, or coarse) and the alignment of these features in space. The results suggest that the visual cortex has an optimized representation particularly for the alignment of fine details, which are crucial in understanding the visual scene.

ABSTRACT

Complex natural scenes can be decomposed into their oriented spatial frequency (SF) and phase relationships, both of which are represented locally at the earliest stages of cortical visual processing. The SF preference map in the human cortex, obtained using synthetic stimuli, is orderly and correlates strongly with eccentricity. In addition, early visual areas show sensitivity to the phase information that describes the relationship between SFs and thereby dictates the structure of the image. Taken together, two possibilities arise for the joint representation of SF and phase: either the entirety of the cortical SF map is uniformly sensitive to phase, or a particular set of SFs is selectively phase sensitive - for example, greater phase sensitivity for higher SFs that define fine-scale edges in a complex scene. To test between these two possibilities, we constructed a novel continuous natural scene video whereby phase information was maintained in one SF band but scrambled elsewhere. By shifting the central frequency of the phase-aligned band in time, we mapped the phase-sensitive SF preference of the visual cortex. Using functional magnetic resonance imaging, we found that phase sensitivity in early visual areas is biased toward higher SFs. Compared to a SF map of the same scene obtained using linear-filtered stimuli, a much larger patch of areas V1 and V2 is sensitive to the phase alignment of higher SFs. The results of early areas cannot be explained by attention. Our results suggest non-uniform sensitivity to phase alignment in population-level SF representations, with phase alignment being particularly important for fine-scale edge representations of natural scenes.

Stimulus-Driven Population Activity Patterns in Macaque Primary Visual Cortex

Dimensionality reduction has been applied in various brain areas to study the activity of populations of neurons. To interpret the outputs of dimensionality reduction, it is important to first understand its outputs for brain areas for which the relationship between the stimulus and neural response is well characterized. Here, we applied principal component analysis (PCA) to trial-averaged neural responses in macaque primary visual cortex (V1) to study two fundamental, population-level questions. First, we characterized how neural complexity relates to stimulus complexity, where complexity is measured using relative comparisons of dimensionality. Second, we assessed the extent to which responses to different stimuli occupy similar dimensions of the population activity space using a novel statistical method. For comparison, we performed the same dimensionality reduction analyses on the activity of a recently-proposed V1 receptive field model and a deep convolutional neural network. Our results show that the dimensionality of the population response changes systematically with alterations in the properties and complexity of the visual stimulus.

A central goal in systems neuroscience is to understand how large populations of neurons work together to enable us to sense, to reason, and to act. To go beyond single-neuron and pairwise analyses, recent studies have applied dimensionality reduction methods to neural population activity to reveal tantalizing evidence of neural mechanisms underlying a wide range of brain functions. To aid in interpreting the outputs of dimensionality reduction, it is important to vary the inputs to a brain area and ask whether the outputs of dimensionality reduction change in a sensible manner, which has not yet been shown. In this study, we recorded the activity of tens of neurons in the primary visual cortex (V1) of macaque monkeys while presenting different visual stimuli. We found that the dimensionality of the population activity grows with stimulus complexity, and that the population responses to different stimuli occupy similar dimensions of the population firing rate space, in accordance with the visual stimuli themselves. For comparison, we applied the same analysis methods to the activity of a recently-proposed V1 receptive field model and a deep convolutional neural network. Overall, we found dimensionality reduction to yield interpretable results, providing encouragement for the use of dimensionality reduction in other brain areas.

The Time-Course of Ultrarapid Categorization: The Influence of Scene Congruency and Top-Down Processing

Although categorization can take place at different levels of abstraction, classic studies on semantic labeling identified the basic level, for example, dog, as entry point for categorization. Ultrarapid categorization tasks have contradicted these findings, indicating that participants are faster at detecting superordinate-level information, for example, animal, in a complex visual image. We argue that both seemingly contradictive findings can be reconciled within the framework of parallel distributed processing and its successor Leabra (Local, Error-driven and Associative, Biologically Realistic Algorithm). The current study aimed at verifying this prediction in an ultrarapid categorization task with a dynamically changing presentation time (PT) for each briefly presented object, followed by a perceptual mask. Furthermore, we manipulated two defining task variables: level of categorization (basic vs. superordinate categorization) and object presentation mode (object-in-isolation vs. object-in-context). In contradiction with previous ultrarapid categorization research, focusing on reaction time, we used accuracy as our main dependent variable. Results indicated a consistent superordinate processing advantage, coinciding with an overall improvement in performance with longer PT and a significantly more accurate detection of objects in isolation, compared with objects in context, at lower stimulus PT. This contextual disadvantage disappeared when PT increased, indicating that figure-ground separation with recurrent processing is vital for meaningful contextual processing to occur.

#TheDress: The Role of Illumination Information and Individual Differences in the Psychophysics of Perceiving White-Blue Ambiguities

In early 2015, a public debate about a perceptual phenomenon that impressively demonstrated the subjective nature of human perception was running round the globe: the debate about #TheDress, a poorly lit photograph of a lace dress that was perceived as white–gold by some, but as blue–black by others. In the present research (N = 48), we found that the perceptual difference between white–gold perceivers (n₁ = 24, 12 women, M_age = 25.4 years) and blue–black perceivers (n₂ = 24, 12 women, M_age = 24.3 years) decreased significantly when the illumination information provided by the original digital photo was reduced by means of image scrambling (Experiment 1). This indicates that the illumination information is one potentially important factor contributing to the color ambiguity of #TheDress—possibly by amplification of a slight principal difference in psychophysics of color perception which the two observer groups showed for abstract uniformly colored fields displaying a white–blue ambiguity (Experiment 2).

Representation of higher-order statistical structures in natural scenes via spatial phase distributions

Natural scenes contain richer perceptual information in their spatial phase structure than their amplitudes. Modeling phase structure of natural scenes may explain higher-order structure inherent to the natural scenes, which is neglected in most classical models of redundancy reduction. Only recently, a few models have represented images using a complex form of receptive fields (RFs) and analyze their complex responses in terms of amplitude and phase. However, these complex representation models often tacitly assume a uniform phase distribution without empirical support. The structure of spatial phase distributions of natural scenes in the form of relative contributions of paired responses of RFs in quadrature has not been explored statistically until now. Here, we investigate the spatial phase structure of natural scenes using complex forms of various Gabor-like RFs. To analyze distributions of the spatial phase responses, we constructed a mixture model that accounts for multi-modal circular distributions, and the EM algorithm for estimation of the model parameters. Based on the likelihood, we report presence of both uniform and structured bimodal phase distributions in natural scenes. The latter bimodal distributions were symmetric with two peaks separated by about 180°. Thus, the redundancy in the natural scenes can be further removed by using the bimodal phase distributions obtained from these RFs in the complex representation models. These results predict that both phase invariant and phase sensitive complex cells are required to represent the regularities of natural scenes in visual systems.

Edge co-occurrences can account for rapid categorization of natural versus animal images

Making a judgment about the semantic category of a visual scene, such as whether it contains an animal, is typically assumed to involve high-level associative brain areas. Previous explanations require progressively analyzing the scene hierarchically at increasing levels of abstraction, from edge extraction to mid-level object recognition and then object categorization. Here we show that the statistics of edge co-occurrences alone are sufficient to perform a rough yet robust (translation, scale, and rotation invariant) scene categorization. We first extracted the edges from images using a scale-space analysis coupled with a sparse coding algorithm. We then computed the “association field” for different categories (natural, man-made, or containing an animal) by computing the statistics of edge co-occurrences. These differed strongly, with animal images having more curved configurations. We show that this geometry alone is sufficient for categorization, and that the pattern of errors made by humans is consistent with this procedure. Because these statistics could be measured as early as the primary visual cortex, the results challenge widely held assumptions about the flow of computations in the visual system. The results also suggest new algorithms for image classification and signal processing that exploit correlations between low-level structure and the underlying semantic category.

Sensitivity to gaze-contingent contrast increments in naturalistic movies: An exploratory report and model comparison

Sensitivity to luminance contrast is a prerequisite for all but the simplest visual systems. To examine contrast increment detection performance in a way that approximates the natural environmental input of the human visual system, we presented contrast increments gaze-contingently within naturalistic video freely viewed by observers. A band-limited contrast increment was applied to a local region of the video relative to the observer's current gaze point, and the observer made a forced-choice response to the location of the target (≈25,000 trials across five observers). We present exploratory analyses showing that performance improved as a function of the magnitude of the increment and depended on the direction of eye movements relative to the target location, the timing of eye movements relative to target presentation, and the spatiotemporal image structure at the target location. Contrast discrimination performance can be modeled by assuming that the underlying contrast response is an accelerating nonlinearity (arising from a nonlinear transducer or gain control). We implemented one such model and examined the posterior over model parameters, estimated using Markov-chain Monte Carlo methods. The parameters were poorly constrained by our data; parameters constrained using strong priors taken from previous research showed poor cross-validated prediction performance. Atheoretical logistic regression models were better constrained and provided similar prediction performance to the nonlinear transducer model. Finally, we explored the properties of an extended logistic regression that incorporates both eye movement and image content features. Models of contrast transduction may be better constrained by incorporating data from both artificial and natural contrast perception settings.

Rapid gist perception of meaningful real-life scenes: Exploring individual and gender differences in multiple categorization tasks

In everyday life, we are generally able to dynamically understand and adapt to socially (ir)elevant encounters, and to make appropriate decisions about these. All of this requires an impressive ability to directly filter and obtain the most informative aspects of a complex visual scene. Such rapid gist perception can be assessed in multiple ways. In the ultrafast categorization paradigm developed by Simon Thorpe et al. (1996), participants get a clear categorization task in advance and succeed at detecting the target object of interest (animal) almost perfectly (even with 20 ms exposures). Since this pioneering work, follow-up studies consistently reported population-level reaction time differences on different categorization tasks, indicating a superordinate advantage (animal versus dog) and effects of perceptual similarity (animals versus vehicles) and object category size (natural versus animal versus dog). In this study, we replicated and extended these separate findings by using a systematic collection of different categorization tasks (varying in presentation time, task demands, and stimuli) and focusing on individual differences in terms of e.g., gender and intelligence. In addition to replicating the main findings from the literature, we find subtle, yet consistent gender differences (women faster than men).

From image statistics to scene gist: evoked neural activity reveals transition from low-level natural image structure to scene category

The visual system processes natural scenes in a split second. Part of this process is the extraction of "gist," a global first impression. It is unclear, however, how the human visual system computes this information. Here, we show that, when human observers categorize global information in real-world scenes, the brain exhibits strong sensitivity to low-level summary statistics. Subjects rated a specific instance of a global scene property, naturalness, for a large set of natural scenes while EEG was recorded. For each individual scene, we derived two physiologically plausible summary statistics by spatially pooling local contrast filter outputs: contrast energy (CE), indexing contrast strength, and spatial coherence (SC), indexing scene fragmentation. We show that behavioral performance is directly related to these statistics, with naturalness rating being influenced in particular by SC. At the neural level, both statistics parametrically modulated single-trial event-related potential amplitudes during an early, transient window (100-150 ms), but SC continued to influence activity levels later in time (up to 250 ms). In addition, the magnitude of neural activity that discriminated between man-made versus natural ratings of individual trials was related to SC, but not CE. These results suggest that global scene information may be computed by spatial pooling of responses from early visual areas (e.g., LGN or V1). The increased sensitivity over time to SC in particular, which reflects scene fragmentation, suggests that this statistic is actively exploited to estimate scene naturalness.

Patterns with different phases but same statistics

Many successful methods in various vision tasks rely on statistical analysis of visual patterns. However, we are interested in covering the gap between the underlying mathematical representation of the visual patterns and their statistics. With this general trend, in this paper a relationship between phase structure of a class of patterns and their moments after and before filtering have been considered. First, a general formula between the phase structure and moments of the images is obtained. Second, a theorem is developed that states under which conditions two visual patterns with the same frequencies but different phases have the same moments up to a certain moment. Finally, a theorem is developed that explains, given a set of filters, under which conditions two visual patterns with both different frequencies and different phases have the same subband statistics.

BOLD human responses to chromatic spatial features

Animal physiological and human psychophysical studies suggest that an early step in visual processing involves the detection and identification of features such as lines and edges, by neural mechanisms with even- and odd-symmetric receptive fields. Functional imaging studies also demonstrate mechanisms with even- and odd-receptive fields in early visual areas, in response to luminance-modulated stimuli. In this study we measured fMRI BOLD responses to 2-D stimuli composed of only even or only odd symmetric features, and to an amplitude-matched random noise control, modulated in red-green equiluminant colour contrast. All these stimuli had identical power but different phase spectra, either highly congruent (even or odd symmetry stimuli) or random (noise). At equiluminance, V1 BOLD activity showed no preference between congruent- and random-phase stimuli, as well as no preference between even and odd symmetric stimuli. Areas higher in the visual hierarchy, both along the dorsal pathway (caudal part of the intraparietal sulcus, dorsal LO and V3A) and the ventral pathway (V4), responded preferentially to odd symmetry over even symmetry stimuli, and to congruent over random phase stimuli. Interestingly, V1 showed an equal increase in BOLD activity at each alternation between stimuli of different symmetry, suggesting the existence of specialised mechanisms for the detection of edges and lines such as even- and odd-chromatic receptive fields. Overall the results indicate a high selectivity of colour-selective neurons to spatial phase along both the dorsal and the ventral pathways in humans.

How sensitive is the human visual system to the local statistics of natural images?

A key hypothesis in sensory system neuroscience is that sensory representations are adapted to the statistical regularities in sensory signals and thereby incorporate knowledge about the outside world. Supporting this hypothesis, several probabilistic models of local natural image regularities have been proposed that reproduce neural response properties. Although many such physiological links have been made, these models have not been linked directly to visual sensitivity. Previous psychophysical studies of sensitivity to natural image regularities focus on global perception of large images, but much less is known about sensitivity to local natural image regularities. We present a new paradigm for controlled psychophysical studies of local natural image regularities and compare how well such models capture perceptually relevant image content. To produce stimuli with precise statistics, we start with a set of patches cut from natural images and alter their content to generate a matched set whose joint statistics are equally likely under a probabilistic natural image model. The task is forced choice to discriminate natural patches from model patches. The results show that human observers can learn to discriminate the higher-order regularities in natural images from those of model samples after very few exposures and that no current model is perfect for patches as small as 5 by 5 pixels or larger. Discrimination performance was accurately predicted by model likelihood, an information theoretic measure of model efficacy, indicating that the visual system possesses a surprisingly detailed knowledge of natural image higher-order correlations, much more so than current image models. We also perform three cue identification experiments to interpret how model features correspond to perceptually relevant image features.

Several aspects of primate visual physiology have been identified as adaptations to local regularities of natural images. However, much less work has measured visual sensitivity to local natural image regularities. Most previous work focuses on global perception of large images and shows that observers are more sensitive to visual information when image properties resemble those of natural images. In this work we measure human sensitivity to local natural image regularities using stimuli generated by patch-based probabilistic natural image models that have been related to primate visual physiology. We find that human observers can learn to discriminate the statistical regularities of natural image patches from those represented by current natural image models after very few exposures and that discriminability depends on the degree of regularities captured by the model. The quick learning we observed suggests that the human visual system is biased for processing natural images, even at very fine spatial scales, and that it has a surprisingly large knowledge of the regularities in natural images, at least in comparison to the state-of-the-art statistical models of natural images.

Similarity relations in visual search predict rapid visual categorization

How do we perform rapid visual categorization?It is widely thought that categorization involves evaluating the similarity of an object to other category items, but the underlying features and similarity relations remain unknown. Here, we hypothesized that categorization performance is based on perceived similarity relations between items within and outside the category. To this end, we measured the categorization performance of human subjects on three diverse visual categories (animals, vehicles, and tools) and across three hierarchical levels (superordinate, basic, and subordinate levels among animals). For the same subjects, we measured their perceived pair-wise similarities between objects using a visual search task. Regardless of category and hierarchical level, we found that the time taken to categorize an object could be predicted using its similarity to members within and outside its category. We were able to account for several classic categorization phenomena, such as (a) the longer times required to reject category membership; (b) the longer times to categorize atypical objects; and (c) differences in performance across tasks and across hierarchical levels. These categorization times were also accounted for by a model that extracts coarse structure from an image. The striking agreement observed between categorization and visual search suggests that these two disparate tasks depend on a shared coarse object representation.

Spatially pooled contrast responses predict neural and perceptual similarity of naturalistic image categories

The visual world is complex and continuously changing. Yet, our brain transforms patterns of light falling on our retina into a coherent percept within a few hundred milliseconds. Possibly, low-level neural responses already carry substantial information to facilitate rapid characterization of the visual input. Here, we computationally estimated low-level contrast responses to computer-generated naturalistic images, and tested whether spatial pooling of these responses could predict image similarity at the neural and behavioral level. Using EEG, we show that statistics derived from pooled responses explain a large amount of variance between single-image evoked potentials (ERPs) in individual subjects. Dissimilarity analysis on multi-electrode ERPs demonstrated that large differences between images in pooled response statistics are predictive of more dissimilar patterns of evoked activity, whereas images with little difference in statistics give rise to highly similar evoked activity patterns. In a separate behavioral experiment, images with large differences in statistics were judged as different categories, whereas images with little differences were confused. These findings suggest that statistics derived from low-level contrast responses can be extracted in early visual processing and can be relevant for rapid judgment of visual similarity. We compared our results with two other, well- known contrast statistics: Fourier power spectra and higher-order properties of contrast distributions (skewness and kurtosis). Interestingly, whereas these statistics allow for accurate image categorization, they do not predict ERP response patterns or behavioral categorization confusions. These converging computational, neural and behavioral results suggest that statistics of pooled contrast responses contain information that corresponds with perceived visual similarity in a rapid, low-level categorization task.

Humans excel in rapid and accurate processing of visual scenes. However, it is unclear which computations allow the visual system to convert light hitting the retina into a coherent representation of visual input in a rapid and efficient way. Here we used simple, computer-generated image categories with similar low-level structure as natural scenes to test whether a model of early integration of low-level information can predict perceived category similarity. Specifically, we show that summarized (spatially pooled) responses of model neurons covering the entire visual field (the population response) to low-level properties of visual input (contrasts) can already be informative about differences in early visual evoked activity as well as behavioral confusions of these categories. These results suggest that low-level population responses can carry relevant information to estimate similarity of controlled images, and put forward the exciting hypothesis that the visual system may exploit these responses to rapidly process real natural scenes. We propose that the spatial pooling that allows for the extraction of this information may be a plausible first step in extracting scene gist to form a rapid impression of the visual input.

Exploiting sparsity and low-rank structure for the recovery of multi-slice breast MRIs with reduced sampling error

It has been shown that, magnetic resonance images (MRIs) with sparsity representation in a transformed domain, e.g. spatial finite-differences (FD), or discrete cosine transform (DCT), can be restored from undersampled k-space via applying current compressive sampling theory. The paper presents a model-based method for the restoration of MRIs. The reduced-order model, in which a full-system-response is projected onto a subspace of lower dimensionality, has been used to accelerate image reconstruction by reducing the size of the involved linear system. In this paper, the singular value threshold (SVT) technique is applied as a denoising scheme to reduce and select the model order of the inverse Fourier transform image, and to restore multi-slice breast MRIs that have been compressively sampled in k-space. The restored MRIs with SVT for denoising show reduced sampling errors compared to the direct MRI restoration methods via spatial FD, or DCT. Compressive sampling is a technique for finding sparse solutions to underdetermined linear systems. The sparsity that is implicit in MRIs is to explore the solution to MRI reconstruction after transformation from significantly undersampled k-space. The challenge, however, is that, since some incoherent artifacts result from the random undersampling, noise-like interference is added to the image with sparse representation. These recovery algorithms in the literature are not capable of fully removing the artifacts. It is necessary to introduce a denoising procedure to improve the quality of image recovery. This paper applies a singular value threshold algorithm to reduce the model order of image basis functions, which allows further improvement of the quality of image reconstruction with removal of noise artifacts. The principle of the denoising scheme is to reconstruct the sparse MRI matrices optimally with a lower rank via selecting smaller number of dominant singular values. The singular value threshold algorithm is performed by minimizing the nuclear norm of difference between the sampled image and the recovered image. It has been illustrated that this algorithm improves the ability of previous image reconstruction algorithms to remove noise artifacts while significantly improving the quality of MRI recovery.

Hippocampal involvement in processing of indistinct visual motion stimuli

Perception of known patterns results from the interaction of current sensory input with existing internal representations. It is unclear how perceptual and mnemonic processes interact when visual input is dynamic and structured such that it does not allow immediate recognition of obvious objects and forms. In an fMRI experiment, meaningful visual motion stimuli depicting movement through a virtual tunnel and indistinct, meaningless visual motion stimuli, achieved through phase scrambling of the same stimuli, were presented while participants performed an optic flow task. We found that our indistinct visual motion stimuli evoked hippocampal activation, whereas the corresponding meaningful stimuli did not. Using independent component analysis, we were able to demonstrate a functional connectivity between the hippocampus and early visual areas, with increased activity for indistinct stimuli. In a second experiment, we used the same stimuli to test whether our results depended on the participants' task. We found task-independent bilateral hippocampal activation in response to indistinct motion stimuli. For both experiments, psychophysiological interaction analysis revealed a coupling from posterior hippocampus to dorsal visuospatial and ventral visual object processing areas when viewing indistinct stimuli. These results indicate a close functional link between stimulus-dependent perceptual and mnemonic processes. The observed pattern of hippocampal functional connectivity, in the absence of an explicit memory task, suggests that cortical-hippocampal networks are recruited when visual stimuli are temporally uncertain and do not immediately reveal a clear meaning.

Low-level cues and ultra-fast face detection

Recent experimental work has demonstrated the existence of extremely rapid saccades toward faces in natural scenes that can be initiated only 100 ms after image onset (Crouzet et al., 2010). These ultra-rapid saccades constitute a major challenge to current models of processing in the visual system because they do not seem to leave enough time for even a single feed-forward pass through the ventral stream. Here we explore the possibility that the information required to trigger these very fast saccades could be extracted very early on in visual processing using relatively low-level amplitude spectrum (AS) information in the Fourier domain. Experiment 1 showed that AS normalization can significantly alter face-detection performance. However, a decrease of performance following AS normalization does not alone prove that AS-based information is used (Gaspar and Rousselet, 2009). In Experiment 2, following the Gaspar and Rousselet paper, we used a swapping procedure to clarify the role of AS information in fast object detection. Our experiment is composed of three conditions: (i) original images, (ii) category swapped, in which the face image has the AS of a vehicle, and the vehicle has the AS of a face, and (iii) identity swapped, where the face has the AS of another face image, and the vehicle has the AS of another vehicle image. The results showed very similar levels of performance in the original and identity swapped conditions, and a clear drop in the category swapped condition. This result demonstrates that, in the early temporal window offered by the saccadic choice task, the visual saccadic system does indeed rely on low-level AS information in order to rapidly detect faces. This sort of crude diagnostic information could potentially be derived very early on in the visual system, possibly as early as V1 and V2.

Spatiotemporal phase-scrambling increases visual cortex activity

The hemodynamic response of the visual cortex to continuously moving spatial stimuli of virtual tunnels and phase-scrambled versions thereof was examined using functional magnetic resonance imaging. Earlier functional magnetic resonance imaging studies found either no difference or less early visual cortex (VC) activation when presenting normal versus phase-manipulated static natural images. Here we describe an increase in VC activation while viewing phase-scrambled films compared with normal films, although basic image statistics and average local flow were the same. The normal films, in contrast, resulted in an increased lateral occipital and precuneus activity sparing VC. In summary, our results show that earlier findings for scrambling of static images no longer hold for spatiotemporal stimuli.

Measuring perceived differences in surface texture due to changes in higher order statistics

We investigate the ability of humans to perceive changes in the appearance of images of surface texture caused by the variation of their higher order statistics. We incrementally randomize their phase spectra while holding their first and second order statistics constant in order to ensure that the change in the appearance is due solely to changes in third and other higher order statistics. Stimuli comprise both natural and synthetically generated naturalistic images, with the latter being used to prevent observers from making pixel-wise comparisons. A difference scaling method is used to derive the perceptual scales for each observer, which show a sigmoidal relationship with the degree of randomization. Observers were maximally sensitive to changes within the 20%-60% randomization range. In order to account for this behavior we propose a biologically plausible model that computes the variance of local measurements of phase congruency.

Representation of cross-frequency spatial phase relationships in human visual cortex

An image patch can be locally decomposed into sinusoidal waves of different orientations, spatial frequencies, amplitudes, and phases. The local phase information is essential for perception, because important visual features like edges emerge at locations of maximal local phase coherence. Detection of phase coherence requires integration of spatial frequency information across multiple spatial scales. Models of early visual processing suggest that the visual system should implement phase-sensitive pooling of spatial frequency information in the identification of broadband edges. We used functional magnetic resonance imaging (fMRI) adaptation to look for phase-sensitive neural responses in the human visual cortex. We found sensitivity to the phase difference between spatial frequency components in all studied visual areas, including the primary visual cortex (V1). Control experiments demonstrated that these results were not explained by differences in contrast or position. Next, we compared fMRI responses for broadband compound grating stimuli with congruent and random phase structures. All studied visual areas showed stronger responses for the stimuli with congruent phase structure. In addition, selectivity to phase congruency increased from V1 to higher-level visual areas along both the ventral and dorsal streams. We conclude that human V1 already shows phase-sensitive pooling of spatial frequencies, but only higher-level visual areas might be capable of pooling spatial frequency information across spatial scales typical for broadband natural images.

How do amplitude spectra influence rapid animal detection?

Amplitude spectra might provide information for natural scene classification. Amplitude does play a role in animal detection because accuracy suffers when amplitude is normalized. However, this effect could be due to an interaction between phase and amplitude, rather than to a loss of amplitude-only information. We used an amplitude-swapping paradigm to establish that animal detection is partly based on an interaction between phase and amplitude. A difference in false alarms for two subsets of our distractor stimuli suggests that the classification of scene environment (man-made versus natural) may also be based on an interaction between phase and amplitude. Examples of interaction between amplitude and phase are discussed.

Parametric study of EEG sensitivity to phase noise during face processing

BACKGROUND

The present paper examines the visual processing speed of complex objects, here faces, by mapping the relationship between object physical properties and single-trial brain responses. Measuring visual processing speed is challenging because uncontrolled physical differences that co-vary with object categories might affect brain measurements, thus biasing our speed estimates. Recently, we demonstrated that early event-related potential (ERP) differences between faces and objects are preserved even when images differ only in phase information, and amplitude spectra are equated across image categories. Here, we use a parametric design to study how early ERP to faces are shaped by phase information. Subjects performed a two-alternative force choice discrimination between two faces (Experiment 1) or textures (two control experiments). All stimuli had the same amplitude spectrum and were presented at 11 phase noise levels, varying from 0% to 100% in 10% increments, using a linear phase interpolation technique. Single-trial ERP data from each subject were analysed using a multiple linear regression model.

RESULTS

Our results show that sensitivity to phase noise in faces emerges progressively in a short time window between the P1 and the N170 ERP visual components. The sensitivity to phase noise starts at about 120-130 ms after stimulus onset and continues for another 25-40 ms. This result was robust both within and across subjects. A control experiment using pink noise textures, which had the same second-order statistics as the faces used in Experiment 1, demonstrated that the sensitivity to phase noise observed for faces cannot be explained by the presence of global image structure alone. A second control experiment used wavelet textures that were matched to the face stimuli in terms of second- and higher-order image statistics. Results from this experiment suggest that higher-order statistics of faces are necessary but not sufficient to obtain the sensitivity to phase noise function observed in response to faces.

CONCLUSION

Our results constitute the first quantitative assessment of the time course of phase information processing by the human visual brain. We interpret our results in a framework that focuses on image statistics and single-trial analyses.

Structural sparseness and spatial phase alignment in natural scenes

The Fourier phase spectrum plays a central role regarding where in an image contours occur, thereby defining the spatial relationship between those structures in the overall scene. Only a handful of studies have demonstrated psychophysically the relevance of the Fourier phase spectrum with respect to human visual processing, and none have demonstrated the relative amount of local cross-scale spatial phase alignment needed to perceptually extract meaningful structure from an image. We investigated the relative amount of spatial phase alignment needed for humans to perceptually match natural scene image structures at three different spatial frequencies [3, 6, and 12 cycles per degree (cpd)] as a function of the number of structures within the image (i.e., "structural sparseness"). The results showed that (1) the amount of spatial phase alignment needed to match structures depends on structural sparseness, with a bias for matching structures at 6 cpd and (2) the ability to match partially phase-randomized images at a given spatial frequency is independent of structural sparseness at other spatial frequencies. The findings of the current study are discussed in terms of a network of feature integrators in the human visual system.