Understanding natural scenes: Contributions of image statistics

How and when social evaluative feedback is processed in the brain: A systematic review on ERP studies

Social evaluative feedback informs the receiver of the other's views, which may contain judgments of personality-related traits and/or the level of likability. Such kinds of social evaluative feedback are of particular importance to humans. Event-related potentials (ERPs) can directly measure where in the processing stream feedback valence, expectancy, or contextual relevance modulate information processing. This review provides an overview and systematization of studies and early, mid-latency, and late ERP effects. Early effects were inconsistently reported for all factors. Feedback valence effects are more consistently reported for specific mid-latency ERPs (Reward Positivity, RewP, and Early Posterior Negativity, EPN) and late positivities (P3 and Late Positive Potential, LPP). Unexpected feedback consistently increased the Feedback Related Negativity (FRN) and, less consistently, decreased P3 amplitudes. Contextual relevance of the sender (e.g., human vs computer sender) or self-relatedness increased mid-latency to late ERPs. Interactions between valence and other factors were less often found, arising during mid-latency stages, where most consistent interactions showed larger EPN and P3 amplitude differences for valent feedback in a more relevant context. The ERP findings highlight that social evaluative feedback is consistently differentiated during mid-latency processing stages. The review discusses the relevance of findings, possible shortcomings of different experimental designs, and open questions. Furthermore, we suggest concrete venues for future research.

Delayed ocular disengagement from arousing scenes

Visual exploration of the world is supported by eye movements which can be speeded up or delayed depending on bottom-up stimulation, top-down goals, and prior associations. Previous studies observed faster initiation of saccades toward emotional than neutral natural scenes; however, less is known concerning saccades which originate from emotional, compared with neutral, scenes. Here, we addressed this issue by examining a task in which participants continuously moved their gaze from and toward pictures (natural scenes), which could be emotional or neutral, and changed position in every trial. Saccades were initiated later when the starting picture was emotional compared to neutral, and this slowing was associated with the arousal value of the picture, suggesting that ocular disengagement does not vary with stimulus valence but is affected by engaging picture contents such as erotica and threat/injuries.

Alpha-band oscillations and emotion: A review of studies on picture perception

Although alpha-band activity has long been a focus of psychophysiological research, its modulation by emotional value during picture perception has only recently been studied systematically. Here, we review these studies and report that the most consistent alpha oscillatory pattern indexing emotional processing is an enhanced desynchronization (ERD) over posterior sensors when viewing emotional compared with neutral pictures. This enhanced alpha ERD is not specific to unpleasant picture content, as previously proposed for other measures of affective response, but has also been observed for pleasant stimuli. Evidence suggests that this effect is not confined to the alpha band but that it also involves a desynchronization of the lower beta frequencies (8-20 Hz). The emotional modulation of alpha ERD occurs even after massive stimulus repetition and when emotional cues serve as task-irrelevant distractors, consistent with the hypothesis that evaluative processes are mandatory in emotional picture processing. A similar enhanced ERD has been observed for other significant cues (e.g., conditioned aversive stimuli, or in anticipation of a potential threat), suggesting that it reflects cortical excitability associated with the engagement of the motivational systems.

Dissociating different temporal stages of emotional word processing by feature-based attention

Negative emotional content is prioritized across different stages of information processing as reflected by different components of the event-related potential (ERP). In this preregistered study (N = 40), we investigated how varying the attentional focus allows us to dissociate the involvement of specific ERP components in the processing of negative and neutral words. Participants had to discriminate the orientation of lines overlaid onto the words, the word type (adjective/noun), or the emotional content (negative/neutral). Thus, attention was either not focused on words (distraction task), non-emotional aspects, or the emotional relevance of words. Regardless of the task, there were no significant differences between negative and neutral words for the P1, N1, or P2 components. In contrast, interactions between emotion and task were observed for the early posterior negativity (EPN) and late positive potential (LPP). EPN differences were absent during the distraction task but were present in the other two tasks. LPP emotion differences were found only when attention was directed to the emotional content of words. Our study adds to the evidence that early ERP components do not reliably separate negative and neutral words. However, results show that mid-latency and late stages of emotion processing are separable by different attention tasks. The EPN represents a stage of attentional enhancement of negative words given sufficient attentional resources. Differential activations during the LPP stage are associated with more elaborative processing of the emotional meaning of words.

Ramp-shaped neural tuning supports graded population-level representation of the object-to-scene continuum

We can easily perceive the spatial scale depicted in a picture, regardless of whether it is a small space (e.g., a close-up view of a chair) or a much larger space (e.g., an entire class room). How does the human visual system encode this continuous dimension? Here, we investigated the underlying neural coding of depicted spatial scale, by examining the voxel tuning and topographic organization of brain responses. We created naturalistic yet carefully-controlled stimuli by constructing virtual indoor environments, and rendered a series of snapshots to smoothly sample between a close-up view of the central object and far-scale view of the full environment (object-to-scene continuum). Human brain responses were measured to each position using functional magnetic resonance imaging. We did not find evidence for a smooth topographic mapping for the object-to-scene continuum on the cortex. Instead, we observed large swaths of cortex with opposing ramp-shaped profiles, with highest responses to one end of the object-to-scene continuum or the other, and a small region showing a weak tuning to intermediate scale views. However, when we considered the population code of the entire ventral occipito-temporal cortex, we found smooth and linear representation of the object-to-scene continuum. Our results together suggest that depicted spatial scale information is encoded parametrically in large-scale population codes across the entire ventral occipito-temporal cortex.

Luminance and timing control during visual presentation of natural scenes

In the study of visual cognition, accurate control of stimulus presentation is of primary importance yet is complicated by hardware malfunctioning, software variability, and visual materials used. Here, we describe VISTO 2.0, a low-cost and open-source device which is capable to measure the timing and temporal luminance profile of visual stimuli. This device represents a major improvement over VISTO (De Cesarei, Marzocchi, & Loftus, 2021), as it is only sensitive to a light spectrum in the visible range, is easier to assemble, and has a modular design that can be extended to other sensory modalities.

Inter- and Transcultural Learning in Social Virtual Reality: A Proposal for an Inter- and Transcultural Virtual Object Database to be Used in the Implementation, Reflection, and Evaluation of Virtual Encounters

Visual stimuli are frequently used to improve memory, language learning or perception, and understanding of metacognitive processes. However, in virtual reality (VR), there are few systematically and empirically derived databases. This paper proposes the first collection of virtual objects based on empirical evaluation for inter-and transcultural encounters between English- and German-speaking learners. We used explicit and implicit measurement methods to identify cultural associations and the degree of stereotypical perception for each virtual stimuli (n = 293) through two online studies, including native German and English-speaking participants. The analysis resulted in a final well-describable database of 128 objects (called InteractionSuitcase). In future applications, the objects can be used as a great interaction or conversation asset and behavioral measurement tool in social VR applications, especially in the field of foreign language education. For example, encounters can use the objects to describe their culture, or teachers can intuitively assess stereotyped attitudes of the encounters.

Nature in motion: The tuning of the visual system to the spatiotemporal properties of natural scenes

Natural scenes contain several statistical regularities despite their superficially diverse appearances (e.g., mountains, rainforests, deserts). First, they exhibit a unique distribution of luminance intensities decreasing across spatial frequency, known as the 1/f^α amplitude spectrum (α ≈ 1). Additionally, natural scenes share consistent geometric properties, comprising similar densities of structure across multiple scales—a property classifying them as fractal (e.g., how the branching patterns of rivers and trees appear similar irrespective of scale). These two properties are intimately related and correlate strongly in natural scenes. However, research using thresholded noise images suggests that spatially, the human visual system is preferentially tuned to natural scene structure more so than 1/f^α spectra. It is currently unclear whether this dependency on natural geometry extends to the temporal domain. We used a psychophysics task to measure discrimination sensitivity toward two types of synthetic noise movies: gray scale and thresholded (N = 60). Each movie type shared the same geometric properties (measured fractal D), but substantially differing spectral properties (measured α). In both space and time, we observe a characteristic dependency on stimulus structure across movie types, with sensitivity peaking for stimuli with natural geometry despite having altered 1/f^α spectra. Although only measured behaviorally, our findings may imply that the neural processes underlying this tuning have developed to be sensitive to the most stable signal in our natural environment—structure (e.g., the structural properties of a tree are consistent from morning to night despite illumination changes across time points).

The influence of stimuli valence, extraversion, and emotion regulation on visual search within real-world scenes

Affective traits, including extraversion and emotion regulation, are important considerations in clinical psychology due to their associations with the occurrence of affective disorders. Previously, emotional real-world scenes have been shown to influence visual search. However, it is currently unknown whether extraversion and emotion regulation can influence visual search towards neutral targets embedded within real-world scenes, or whether these traits can impact the effect of emotional stimuli on visual search. An opportunity sample of healthy individuals had trait levels of extraversion and emotion regulation recorded before completing a visual search task. Participants more accurately identified search targets in neutral images compared to positive images, whilst response times were slower in negative images. Importantly, individuals with higher trait levels of expressive suppression displayed faster identification of search targets regardless of the emotional valence of the stimuli. Extraversion and cognitive reappraisal did not influence visual search. These findings add to our understanding regarding the influence of extraversion, cognitive reappraisal, and expressive suppression on our ability to allocate attention during visual search when viewing real-world scenes.

Analysis and Synthesis of Natural Texture Perception From Visual Evoked Potentials

The primate visual system analyzes statistical information in natural images and uses it for the immediate perception of scenes, objects, and surface materials. To investigate the dynamical encoding of image statistics in the human brain, we measured visual evoked potentials (VEPs) for 166 natural textures and their synthetic versions, and performed a reverse-correlation analysis of the VEPs and representative texture statistics of the image. The analysis revealed occipital VEP components strongly correlated with particular texture statistics. VEPs correlated with low-level statistics, such as subband SDs, emerged rapidly from 100 to 250 ms in a spatial frequency dependent manner. VEPs correlated with higher-order statistics, such as subband kurtosis and cross-band correlations, were observed at slightly later times. Moreover, these robust correlations enabled us to inversely estimate texture statistics from VEP signals via linear regression and to reconstruct texture images that appear similar to those synthesized with the original statistics. Additionally, we found significant differences in VEPs at 200-300 ms between some natural textures and their Portilla-Simoncelli (PS) synthesized versions, even though they shared almost identical texture statistics. This differential VEP was related to the perceptual "unnaturalness" of PS-synthesized textures. These results suggest that the visual cortex rapidly encodes image statistics hidden in natural textures specifically enough to predict the visual appearance of a texture, while it also represents high-level information beyond image statistics, and that electroencephalography can be used to decode these cortical signals.

Nonlinear Effects of Linearly Increasing Perceptual Load on ERPs to Emotional Pictures

The prioritized processing of emotional as compared to neutral stimuli is reflected in enlarged event-related potentials (ERPs). However, perceptual load theory proposes that under conditions of high perceptual load, information processing is attenuated or abolished. The parametrical effects of load on ERPs to emotional pictures are unknown. To shed light on this question, the current preregistered ERP study (N = 30) systematically investigated the effects of load on ERPs to task-irrelevant negative, neutral, and positive pictures. Crucially, while perceptual input was held constant, perceptual load was systematically manipulated so that it increased linearly across 4 load levels, which was evident in behavioral data. In contrast, load effects on ERP differences between emotional and neutral stimuli did not follow a linear function. For the N1, early posterior negativity and late positive potential, a nonlinear function with reversed emotion effects at the third load level provided the best fit. These findings do not only show that perceptual load attenuates emotional picture processing but also suggest that active processes are initiated to reduce distraction by emotional information. Moreover, these effects of perceptual load on emotional ERP components appear to deviate from theoretically expected functions.

Do Humans and Deep Convolutional Neural Networks Use Visual Information Similarly for the Categorization of Natural Scenes?

The investigation of visual categorization has recently been aided by the introduction of deep convolutional neural networks (CNNs), which achieve unprecedented accuracy in picture classification after extensive training. Even if the architecture of CNNs is inspired by the organization of the visual brain, the similarity between CNN and human visual processing remains unclear. Here, we investigated this issue by engaging humans and CNNs in a two-class visual categorization task. To this end, pictures containing animals or vehicles were modified to contain only low/high spatial frequency (HSF) information, or were scrambled in the phase of the spatial frequency spectrum. For all types of degradation, accuracy increased as degradation was reduced for both humans and CNNs; however, the thresholds for accurate categorization varied between humans and CNNs. More remarkable differences were observed for HSF information compared to the other two types of degradation, both in terms of overall accuracy and image-level agreement between humans and CNNs. The difficulty with which the CNNs were shown to categorize high-passed natural scenes was reduced by picture whitening, a procedure which is inspired by how visual systems process natural images. The results are discussed concerning the adaptation to regularities in the visual environment (scene statistics); if the visual characteristics of the environment are not learned by CNNs, their visual categorization may depend only on a subset of the visual information on which humans rely, for example, on low spatial frequency information.

Functional Imaging of Visuospatial Attention in Complex and Naturalistic Conditions

One of the ultimate goals of cognitive neuroscience is to understand how the brain works in the real world. Functional imaging with naturalistic stimuli provides us with the opportunity to study the brain in situations similar to the everyday life. This includes the processing of complex stimuli that can trigger many types of signals related both to the physical characteristics of the external input and to the internal knowledge that we have about natural objects and environments. In this chapter, I will first outline different types of stimuli that have been used in naturalistic imaging studies. These include static pictures, short video clips, full-length movies, and virtual reality, each comprising specific advantages and disadvantages. Next, I will turn to the main issue of visual-spatial orienting in naturalistic conditions and its neural substrates. I will discuss different classes of internal signals, related to objects, scene structure, and long-term memory. All of these, together with external signals about stimulus salience, have been found to modulate the activity and the connectivity of the frontoparietal attention networks. I will conclude by pointing out some promising future directions for functional imaging with naturalistic stimuli. Despite this field of research is still in its early days, I consider that it will play a major role in bridging the gap between standard laboratory paradigms and mechanisms of brain functioning in the real world.

E-MOVIE - Experimental MOVies for Induction of Emotions in neuroscience: An innovative film database with normative data and sex differences

The need for a validated set of emotional clips to elicit emotions in more ecological experiments is increasing. Here we present the validation of a new database of emotional films, named E-MOVIE, which includes, in this first validation phase, 39 excerpts arranged in six categories, three negative (Fear, Sadness and Compassion), two positive (Erotic and Scenery) and a Neutral category. Notably, Compassion and Scenery are new in the field as they were not included in other databases. The clips in E-MOVIE are characterized by homogenous durations of approximately two minutes, which make them suitable for psychophysiological research. In order to study the affective profile prompted by each category 174 participants (112 women) rated the movies on multiple dimensions, namely valence and arousal, intensity and discreteness of the induction of one of the six basic emotions and, finally, intensity of the experience of the emotional states defined by a series of emotional adjectives. Erotic clips were effective in the elicitation of a positive emotional state, characterized by high levels of arousal and excitement. On the other hand, Fear clips (selected without blood to avoid disgust reaction) prompted an affect characterized by high arousal, low valence and high levels of reported fear and anxiety. Women reported greater unpleasantness, distress, anxiety and jittery than men to the three negative categories. Compassion clips, characterized by the depiction of crying characters, were able to induce an affective state dominated by sadness and feeling touched, consistent with an empathic reaction to emotional sufferance. Sadness clips, instead, elicited an affective state characterized by sadness together with distress and angst. We also demonstrated that clips depicting natural environments (i.e. Scenery) prompted in the viewer a surprised, inspired affective state, characterized by high valence and arousal (especially in males), a result which suggests that their past categorization as neutral stimuli was inaccurate and problematic.

The role of low-level image features in the affective categorization of rapidly presented scenes

It remains unclear how the visual system is able to extract affective content from complex scenes even with extremely brief (< 100 millisecond) exposures. One possibility, suggested by findings in machine vision, is that low-level features such as unlocalized, two-dimensional (2-D) Fourier spectra can be diagnostic of scene content. To determine whether Fourier image amplitude carries any information about the affective quality of scenes, we first validated the existence of image category differences through a support vector machine (SVM) model that was able to discriminate our intact aversive and neutral images with ~ 70% accuracy using amplitude-only features as inputs. This model allowed us to confirm that scenes belonging to different affective categories could be mathematically distinguished on the basis of amplitude spectra alone. The next question is whether these same features are also exploited by the human visual system. Subsequently, we tested observers' rapid classification of affective and neutral naturalistic scenes, presented briefly (~33.3 ms) and backward masked with synthetic textures. We tested categorization accuracy across three distinct experimental conditions, using: (i) original images, (ii) images having their amplitude spectra swapped within a single affective image category (e.g., an aversive image whose amplitude spectrum has been swapped with another aversive image) or (iii) images having their amplitude spectra swapped between affective categories (e.g., an aversive image containing the amplitude spectrum of a neutral image). Despite its discriminative potential, the human visual system does not seem to use Fourier amplitude differences as the chief strategy for affectively categorizing scenes at a glance. The contribution of image amplitude to affective categorization is largely dependent on interactions with the phase spectrum, although it is impossible to completely rule out a residual role for unlocalized 2-D amplitude measures.

Categorization Goals Modulate the Use of Natural Scene Statistics

Understanding natural scenes involves the contribution of bottom–up analysis and top–down modulatory processes. However, the interaction of these processes during the categorization of natural scenes is not well understood. In the current study, we approached this issue using ERPs and behavioral and computational data. We presented pictures of natural scenes and asked participants to categorize them in response to different questions (Is it an animal/vehicle? Is it indoors/outdoors? Are there one/two foreground elements?). ERPs for target scenes requiring a “yes” response began to differ from those of nontarget scenes, beginning at 250 msec from picture onset, and this ERP difference was unmodulated by the categorization questions. Earlier ERPs showed category-specific differences (e.g., between animals and vehicles), which were associated with the processing of scene statistics. From 180 msec after scene onset, these category-specific ERP differences were modulated by the categorization question that was asked. Categorization goals do not modulate only later stages associated with target/nontarget decision but also earlier perceptual stages, which are involved in the processing of scene statistics.

Scene complexity modulates degree of feedback activity during object detection in natural scenes

Selective brain responses to objects arise within a few hundreds of milliseconds of neural processing, suggesting that visual object recognition is mediated by rapid feed-forward activations. Yet disruption of neural responses in early visual cortex beyond feed-forward processing stages affects object recognition performance. Here, we unite these discrepant findings by reporting that object recognition involves enhanced feedback activity (recurrent processing within early visual cortex) when target objects are embedded in natural scenes that are characterized by high complexity. Human participants performed an animal target detection task on natural scenes with low, medium or high complexity as determined by a computational model of low-level contrast statistics. Three converging lines of evidence indicate that feedback was selectively enhanced for high complexity scenes. First, functional magnetic resonance imaging (fMRI) activity in early visual cortex (V1) was enhanced for target objects in scenes with high, but not low or medium complexity. Second, event-related potentials (ERPs) evoked by target objects were selectively enhanced at feedback stages of visual processing (from ~220 ms onwards) for high complexity scenes only. Third, behavioral performance for high complexity scenes deteriorated when participants were pressed for time and thus less able to incorporate the feedback activity. Modeling of the reaction time distributions using drift diffusion revealed that object information accumulated more slowly for high complexity scenes, with evidence accumulation being coupled to trial-to-trial variation in the EEG feedback response. Together, these results suggest that while feed-forward activity may suffice to recognize isolated objects, the brain employs recurrent processing more adaptively in naturalistic settings, using minimal feedback for simple scenes and increasing feedback for complex scenes.

Toward a Computer Vision Perspective on the Visual Impact of Vegetation in Symmetries of Urban Environments

Rapid urbanization is a worldwide critical environmental challenge. With this urban migration soaring, we need to live far more efficiently than we currently do by incorporating the natural world in new and innovative ways. There are a lot of researches on ecological, architectural or aesthetic points of view to address this issue. We present a novel approach to assess the visual impact of vegetation in urban street pedestrian view with the assistance of computer vision metrics. We statistically evaluate the correlations of the amount of vegetation with objective computer vision traits such as Fourier domain, color histogram, and estimated depth from monocular view. We show that increasing vegetation in urban street views breaks the orthogonal symmetries of urban blocks, enriches the color space with fractal-like symmetries and decreases the cues of projective geometry in depth. These uncovered statistical facts are applied to predict the requested amount of vegetation to make urban street views appear like natural images. Interestingly, these amounts are found in accordance with the ecosystemic approach for urban planning. Also, the study opens new questions for the understanding of the link between geometry and depth perception.

Effects of Directed Attention on Subsequent Processing of Emotions: Increased Attention to Unpleasant Pictures Occurs in the Late Positive Potential

Directed attention is a fundamental mental resource for voluntarily managing the focus and direction of cognitive resources. The present study investigated how processing of unpleasant and neutral images is affected by emotion and previous directed attention. The results showed that there was enhanced early posterior negativity, anterior N2, and parietal late positive potential (LPP) in response to unpleasant pictures compared to neutral pictures. Furthermore, attention history (i.e., whether stimuli were previously attended to) modulated the amplitudes of the anterior N2 and parietal LPP. Most notably, an interaction between attention history and emotion was found in the LPP: pictures with an 'attended history' evoked larger LPP amplitudes than pictures with an 'unattended history,' but this effect was only significant for unpleasant pictures (not for neutral pictures). These results suggest that directed attention to affective pictures facilitates subsequent neural processing of these pictures, and that this effect was amplified by unpleasant emotions experienced in the LPP. The current findings provide further empirical evidence of a two-stage model of emotion-attention interaction.

Entrainment of visual steady-state responses is modulated by global spatial statistics

The rhythmic delivery of visual stimuli evokes large-scale neuronal entrainment in the form of steady-state oscillatory field potentials. The spatiotemporal properties of stimulus drive appear to constrain the relative degrees of neuronal entrainment. Specific frequency ranges, for example, are uniquely suited for enhancing the strength of stimulus-driven brain oscillations. When it comes to the nature of the visual stimulus itself, studies have used a plethora of inputs ranging from spatially unstructured empty fields to simple contrast patterns (checkerboards, gratings, stripes) and complex arrays (human faces, houses, natural scenes). At present, little is known about how the global spatial statistics of the input stimulus influence entrainment of scalp-recorded electrophysiological signals. In this study, we used rhythmic entrainment source separation of scalp EEG to compare stimulus-driven phase alignment for distinct classes of visual inputs, including broadband spatial noise ensembles with varying second-order statistics, natural scenes, and narrowband sine-wave gratings delivered at a constant flicker frequency. The relative magnitude of visual entrainment was modulated by the global properties of the driving stimulus. Entrainment was strongest for pseudo-naturalistic broadband visual noise patterns in which luminance contrast is greatest at low spatial frequencies (a power spectrum slope characterized by 1/ƒ-2).NEW & NOTEWORTHY Rhythmically modulated visual stimuli entrain the activity of neuronal populations, but the effect of global stimulus statistics on this entrainment is unknown. We assessed entrainment evoked by 1) visual noise ensembles with different spectral slopes, 2) complex natural scenes, and 3) narrowband sinusoidal gratings. Entrainment was most effective for broadband noise with naturalistic luminance contrast. This reveals some global properties shaping stimulus-driven brain oscillations in the human visual system.