Affective and contextual values modulate spatial frequency use in object recognition

Spatial frequency processing and its modulation by emotional content in severe alcohol use disorder

RATIONALE

Visuo-perceptive deficits in severe alcohol use disorder (SAUD) remain little understood, notably regarding the respective involvement of the two main human visual streams, i.e., magnocellular (MC) and parvocellular (PC) pathways, in these deficits. Besides, in healthy populations, low-level visual perception can adapt depending on the nature of visual cues, among which emotional features, but this MC and PC pathway adaptation to emotional content is unexplored in SAUD.

OBJECTIVES

To assess MC and PC functioning as well as their emotional modulations in SAUD.

METHODS

We used sensitivity indices (d') and repeated-measures analyses of variance to compare orientation judgments of Gabor patches sampled at various MC- and PC-related spatial frequencies in 35 individuals with SAUD and 38 matched healthy controls. We then explored how emotional content modulated performances by introducing neutral or fearful face cues immediately before the Gabor patches and added the type of cue in the analyses.

RESULTS

SAUD patients showed a general reduction in sensitivity across all spatial frequencies, indicating impoverished processing of both coarse and fine-scale visual content. However, we observed selective impairments depending on facial cues: individuals with SAUD processed intermediate spatial frequencies less efficiently than healthy controls following neutral faces, whereas group differences emerged for the highest spatial frequencies following fearful faces. Altogether, SAUD was associated with mixed MC and PC deficits that may vary according to emotional content, in line with a flexible but suboptimal use of low-level visual content. Such subtle alterations could have implications for everyday life's complex visual judgments.

Probing machine-learning classifiers using noise, bubbles, and reverse correlation

BACKGROUND

Many scientific fields now use machine-learning tools to assist with complex classification tasks. In neuroscience, automatic classifiers may be useful to diagnose medical images, monitor electrophysiological signals, or decode perceptual and cognitive states from neural signals. However, such tools often remain black-boxes: they lack interpretability. A lack of interpretability has obvious ethical implications for clinical applications, but it also limits the usefulness of these tools to formulate new theoretical hypotheses.

NEW METHOD

We propose a simple and versatile method to help characterize the information used by a classifier to perform its task. Specifically, noisy versions of training samples or, when the training set is unavailable, custom-generated noisy samples, are fed to the classifier. Multiplicative noise, so-called "bubbles", or additive noise are applied to the input representation. Reverse correlation techniques are then adapted to extract either the discriminative information, defined as the parts of the input dataset that have the most weight in the classification decision, and represented information, which correspond to the input features most representative of each category.

RESULTS

The method is illustrated for the classification of written numbers by a convolutional deep neural network; for the classification of speech versus music by a support vector machine; and for the classification of sleep stages from neurophysiological recordings by a random forest classifier. In all cases, the features extracted are readily interpretable.

COMPARISON WITH EXISTING METHODS

Quantitative comparisons show that the present method can match state-of-the art interpretation methods for convolutional neural networks. Moreover, our method uses an intuitive and well-established framework in neuroscience, reverse correlation. It is also generic: it can be applied to any kind of classifier and any kind of input data.

CONCLUSIONS

We suggest that the method could provide an intuitive and versatile interface between neuroscientists and machine-learning tools.

The role of spatial frequencies for facial pain categorization

Studies on low-level visual information underlying pain categorization have led to inconsistent findings. Some show an advantage for low spatial frequency information (SFs) and others a preponderance of mid SFs. This study aims to clarify this gap in knowledge since these results have different theoretical and practical implications, such as how far away an observer can be in order to categorize pain. This study addresses this question by using two complementary methods: a data-driven method without a priori expectations about the most useful SFs for pain recognition and a more ecological method that simulates the distance of stimuli presentation. We reveal a broad range of important SFs for pain recognition starting from low to relatively high SFs and showed that performance is optimal in a short to medium distance (1.2-4.8 m) but declines significantly when mid SFs are no longer available. This study reconciles previous results that show an advantage of LSFs over HSFs when using arbitrary cutoffs, but above all reveal the prominent role of mid-SFs for pain recognition across two complementary experimental tasks.

Flexible time course of spatial frequency use during scene categorization

Human observers can quickly and accurately categorize scenes. This remarkable ability is related to the usage of information at different spatial frequencies (SFs) following a coarse-to-fine pattern: Low SFs, conveying coarse layout information, are thought to be used earlier than high SFs, representing more fine-grained information. Alternatives to this pattern have rarely been considered. Here, we probed all possible SF usage strategies randomly with high resolution in both the SF and time dimensions at two categorization levels. We show that correct basic-level categorizations of indoor scenes are linked to the sampling of relatively high SFs, whereas correct outdoor scene categorizations are predicted by an early use of high SFs and a later use of low SFs (fine-to-coarse pattern of SF usage). Superordinate-level categorizations (indoor vs. outdoor scenes) rely on lower SFs early on, followed by a shift to higher SFs and a subsequent shift back to lower SFs in late stages. In summary, our results show no consistent pattern of SF usage across tasks and only partially replicate the diagnostic SFs found in previous studies. We therefore propose that SF sampling strategies of observers differ with varying stimulus and task characteristics, thus favouring the notion of flexible SF usage.

Object expectations alter information use during visual recognition

Prior expectations influence how we perceive and recognize objects. However, how they do so remains unclear, especially in the case of real-world complex objects. Expectations of objects may affect which features are used to recognize them subsequently. In this study, we used reverse correlation to reveal with high precision how the use of information across time is modulated by real-world object expectations in a visual recognition task. We show that coarse information leads to accurate responses earlier when an object is expected, indicating that observers use diagnostic features earlier in this situation. We also demonstrate an increased variability in the use of coarse information depending on the expected object, indicating that observers adopt a more specialized recognition strategy when they expect a specific object. In summary, our results reveal potential mechanisms underlying the effect of expectations on the recognition of complex objects.

Real-world expectations and their affective value modulate object processing

It is well known that expectations influence how we perceive the world. Yet the neural mechanisms underlying this process remain unclear. Studies about the effects of prior expectations have focused so far on artificial contingencies between simple neutral cues and events. Real-world expectations are however often generated from complex associations between contexts and objects learned over a lifetime. Additionally, these expectations may contain some affective value and recent proposals present conflicting hypotheses about the mechanisms underlying affect in predictions. In this study, we used fMRI to investigate how object processing is influenced by realistic context-based expectations, and how affect impacts these expectations. First, we show that the precuneus, the inferotemporal cortex and the frontal cortex are more active during object recognition when expectations have been elicited a priori, irrespectively of their validity or their affective intensity. This result supports previous hypotheses according to which these brain areas integrate contextual expectations with object sensory information. Notably, these brain areas are different from those responsible for simultaneous context-object interactions, dissociating the two processes. Then, we show that early visual areas, on the contrary, are more active during object recognition when no prior expectation has been elicited by a context. Lastly, BOLD activity was shown to be enhanced in early visual areas when objects are less expected, but only when contexts are neutral; the reverse effect is observed when contexts are affective. This result supports the proposal that affect modulates the weighting of sensory information during predictions. Together, our results help elucidate the neural mechanisms of real-world expectations.

Time Course of Cultural Differences in Spatial Frequency Use for Face Identification

Several previous studies of eye movements have put forward that, during face recognition, Easterners spread their attention across a greater part of their visual field than Westerners. Recently, we found that culture’s effect on the perception of faces reaches mechanisms deeper than eye movements, therefore affecting the very nature of information sampled by the visual system: that is, Westerners globally rely more than Easterners on fine-grained visual information (i.e. high spatial frequencies; SFs), whereas Easterners rely more on coarse-grained visual information (i.e. low SFs). These findings suggest that culture influences basic visual processes; however, the temporal onset and dynamics of these culture-specific perceptual differences are still unknown. Here, we investigate the time course of SF use in Western Caucasian (Canadian) and East Asian (Chinese) observers during a face identification task. Firstly, our results confirm that Easterners use relatively lower SFs than Westerners, while the latter use relatively higher SFs. More importantly, our results indicate that these differences arise as early as 34 ms after stimulus onset, and remain stable through time. Our research supports the hypothesis that Westerners and Easterners initially rely on different types of visual information during face processing.

On the use of spatial frequency to isolate contributions from the magnocellular and parvocellular systems and the dorsal and ventral cortical streams

Many authors have claimed that suprathreshold achromatic stimuli of low and high spatial frequency can be used to separate responses from different entities in the visual system. Most prominently, it has been proposed that such stimuli can differentiate responses from the magnocellular and parvocellular systems. As is reviewed here, investigators who have examined stimulus specificity of neurons in these systems have found little difference between magno- and parvocellular cells. It has also been proposed that spatial frequency can be used to selectively activate the "magnocellular-dorsal stream". The present review indicates that cells in Area MT of the dorsal stream do prefer very low spatial frequencies. However, the review also shows that cells in Area V4 of the ventral stream respond, not only to relatively high spatial frequencies, but also to low frequency stimuli. Thus, low spatial frequencies cannot be relied upon to selectively activate the dorsal stream.