Eye movement related brain responses to emotional scenes during free viewing

How emotional contexts modulate item memory in individuals with high and low negative affect and worry

Emotional stimuli are usually remembered with high confidence. Yet, it remains unknown whether-in addition to memory for the emotional stimulus itself-memory for a neutral stimulus encountered just after an emotional one can be enhanced. Further, little is known about the interplay between emotion elicited by a stimulus and emotion relating to affective dispositions. To address these questions, we examined (1) how emotional valence and arousal of a context image preceding a neutral item image affect memory of the item, and (2) how such memory modulation is affected by two hallmark features of emotional disorders: trait negative affect and tendency to worry. In two experiments, participants encoded a series of trials in which an emotional (negative, neutral, or positive) context image was followed by a neutral item image. In experiment 1 (n = 42), items presented seconds after negative context images were remembered better and with greater confidence compared to those presented after neutral and positive ones. Arousal ratings of negative context images were higher compared to neutral and positive ones and the likelihood of correctly recognizing an item image was related to higher arousal of the context image. In experiment 2 (n = 59), better item memory was related to lower trait negative affect. Participants with lower trait negative affect or tendency to worry displayed higher confidence compared to those with high negative affect or tendency to worry. Our findings describe an emotional "carry-over" effect elicited by a context image that enhances subsequent item memory on a trial-by-trial basis, however, not in individuals with high trait negative affect who seem to have a general memory disadvantage.

(Micro)saccade-related potentials during face recognition: A study combining EEG, eye-tracking, and deconvolution modeling

Under natural viewing conditions, complex stimuli such as human faces are typically looked at several times in succession, implying that their recognition may unfold across multiple eye fixations. Although electrophysiological (EEG) experiments on face recognition typically prohibit eye movements, participants still execute frequent (micro)saccades on the face, each of which generates its own visuocortical response. This finding raises the question of whether the fixation-related potentials (FRPs) evoked by these tiny gaze shifts also contain psychologically valuable information about face processing. Here, we investigated this question by corecording EEG and eye movements in an experiment with emotional faces (happy, angry, neutral). Deconvolution modeling was used to separate the stimulus ERPs to face onset from the FRPs generated by subsequent microsaccades-induced refixations on the face. As expected, stimulus ERPs exhibited typical emotion effects, with a larger early posterior negativity (EPN) for happy/angry compared with neutral faces. Eye tracking confirmed that participants made small saccades in 98% of the trials, which were often aimed at the left eye of the stimulus face. However, while each saccade produced a strong response over visual areas, this response was unaffected by the face's emotional expression, both for the first and for subsequent (micro)saccades. This finding suggests that the face's affective content is rapidly evaluated after stimulus onset, leading to only a short-lived sensory enhancement by arousing stimuli that does not repeat itself during immediate refixations. Methodologically, our work demonstrates how eye tracking and deconvolution modeling can be used to extract several brain responses from each EEG trial, providing insights into neural processing at different latencies after stimulus onset.

Targeting Histamine and Histamine Receptors for Memory Regulation: An Emotional Perspective

Histamine has long been accepted as a pro-cognitive agent. However, lines of evidence have suggested that the roles of histamine in learning and memory processes are much more complex than previously thought. When explained by the spatial perspectives, there are many contradictory results. However, using emotional memory perspectives, we suspect that the histaminergic system may interplay with stress, reward inhibition, and attention to modulate emotional memory formation. The functional diversity of histamine makes it a viable target for clinical management of neuropsychiatric disorders. Here, we update the current knowledge about the functions of histamine in emotional memory and summarize the underlying molecular and neural circuit mechanisms. Finally, we review the main clinical studies about the impacts of histamine-related compounds on memory and discuss insights into future research on the roles of histamine in emotional memory. Despite the recent progress in histamine research, the histaminergic emotional memory circuits are poorly understood, and it is also worth verifying the functions of histamine receptors in a more spatiotemporally specific manner.

In Search of the Most Attractive Lip-Proportions and Lip-Volume -An Eye Tracking- and Survey-based Investigation

BACKGROUND

Despite various aesthetic trends, the ideal lip proportion and lip volume still remains elusive. Thus, the aim of this study was to investigate the aesthetic perception of various lip shapes to identify the most attractive lips.

METHODS

59 Caucasian study participants with a mean age of 32.73 (9.4) years were asked to assess lips of various proportions and of various volumes. Gaze assessment and aesthetic rating of the same set of modified lip images were performed.

RESULTS

The results revealed that the lip proportion which was rated to be most attractive was the 1:1.6 ratio (upper:lower lip, 4.21 out of a possible 5). This lip proportion, however, was not the first one to be viewed upon initial image display (1.20 sec) and had the shortest duration of a stable eye fixation within the 6 sec image display interval (2.18 out of 6 sec). The lip volume which received the highest aesthetic rating was the 100% volume (original lip size) with 4.56 out of a possible 5. This lip volume however, had the third longest interval between initial image exposure and the first stable eye fixation (0.81 sec) and had the shortest duration of stable eye fixations during the 6 sec of image display (2.22 out of 6 sec).

CONCLUSION

The results presented could be potentially reflective of the internal cognitive processes during the involuntary and voluntary assessment of lip attractiveness.

CAPSULE SUMMARY

The results revealed that the lip proportion and lip volume which was rated to be most attractive was the 1:1.6 ratio and the non-augmented, natural volume. Eye tracking showed that attractive lips captured the observer's attention less quickly and were viewed for a shorter time.

Emotional memories are (usually) harder to forget: A meta-analysis of the item-method directed forgetting literature

The current meta-analysis explored whether emotional memories are less susceptible to item-method directed forgetting than neutral memories. Basic analyses revealed superior memory for remember (R) than forget (F) items in both the neutral, M = 19.6%, CI_95% [16.1, 23.1], and the emotional, M = 15.1%, CI_95% [12.4, 17.7], conditions. Directed forgetting in either valence condition was larger for (a) words than for other stimuli; (b) recall than recognition tests; (c) studies that used recall prior to recognition testing; (d) shorter lists; and (e) studies that included buffer items. Direct comparison of the magnitude of the directed forgetting effect across neutral and emotional conditions within studies revealed relatively diminished directed forgetting of emotional items compared to neutral items, with an average difference of 4.2%, CI_95% [2.0, 6.4]. However, the nature of this finding varied broadly across studies, meaning that whether - and to what degree - emotional memories are more resilient than neutral memories likely depends on the methodological features of the study in question. Moderator analyses revealed larger differences (a) in studies for which the emotional items were more arousing than the neutral items, and (b) when buffer items were included. Together, these findings suggest that emotional memories are often more resilient to intentional forgetting than neutral memories, although further research is necessary to characterize the circumstances under which these differences emerge.

The influence of negative emotional intensity on dual-processing recognition

Dual-processing theory assumes recognition memory involves two distinct processes: familiarity and recollection. Although the influence of emotional intensity on memory has been investigated, it remains unclear whether the influence of negative stimuli depends on familiarity or recollection. This study recorded event-related potentials as participants performed a modified remember/know procedure with highly negative, mildly negative, and neutral stimuli. The results showed that, relative to highly negative stimuli, mildly negative and neutral stimuli showed increased mean discrimination for responses of 'know' in the following pattern: highly negative < mildly negative < neutral. Neutral stimuli enhanced the frontal old/new effect. Relative to mildly negative and neutral stimuli, highly negative stimuli showed increased mean discrimination for responses of 'remember', and enhanced the parietal old/new effect. These results suggested negative emotional intensity influences recollection and familiarity differently, as recognition of highly negative stimuli depends on recollection, and recognition of neutral stimuli depends on familiarity.

The relationship between dispositional attention to feelings and visual attention to emotion

Attention to feelings is a core dimension of individual differences in the perception of one's emotions. It concerns the frequency with which own emotions are attended to. The aim of the present eye-tracking investigation was to examine the relationship of dispositional attention to feelings with early and late attentional processes in the visual perception of emotional information. Attentional orientation was assessed in a sample of healthy women (N = 91) using eye-tracking during a free viewing task in which images with positive, negative and neutral content were shown simultaneously. Pictures were taken from the Besançon Affective Picture Set. State and trait affect, depression, and intelligence of participants were controlled. In our sample, attention to feelings was not related to positive affect, negative affect, depression or intelligence. Attention to feelings was negatively correlated with entry times for all emotional picture types. Moreover, attention to feelings was positively correlated with dwell time on positive images and negatively correlated with dwell time on neutral images. Our data indicate that devoting habitually attention to one's feelings is linked to an enhanced initial orientation of attention towards emotional scenes, irrespective of affective valence. Dispositional attention to feelings might also be associated with an attentional preference for emotional over neutral visual stimuli. The present findings on general individual differences in attention to feelings could have important implications for future clinical research on attentional biases.

From Abstract Symbols to Emotional (In-)Sights: An Eye Tracking Study on the Effects of Emotional Vignettes and Pictures

Reading is known to be a highly complex, emotion-inducing process, usually involving connected and cohesive sequences of sentences and paragraphs. However, most empirical results, especially from studies using eye tracking, are either restricted to simple linguistic materials (e.g., isolated words, single sentences) or disregard valence-driven effects. The present study addressed the need for ecologically valid stimuli by examining the emotion potential of and reading behavior in emotional vignettes, often used in applied psychological contexts and discourse comprehension. To allow for a cross-domain comparison in the area of emotion induction, negatively and positively valenced vignettes were constructed based on pre-selected emotional pictures from the Nencki Affective Picture System (NAPS; Marchewka et al., 2014). We collected ratings of perceived valence and arousal for both material groups and recorded eye movements of 42 participants during reading and picture viewing. Linear mixed-effects models were performed to analyze effects of valence (i.e., valence category, valence rating) and stimulus domain (i.e., textual, pictorial) on ratings of perceived valence and arousal, eye movements in reading, and eye movements in picture viewing. Results supported the success of our experimental manipulation: emotionally positive stimuli (i.e., vignettes, pictures) were perceived more positively and less arousing than emotionally negative ones. The cross-domain comparison indicated that vignettes are able to induce stronger valence effects than their pictorial counterparts, no differences between vignettes and pictures regarding effects on perceived arousal were found. Analyses of eye movements in reading replicated results from experiments using isolated words and sentences: perceived positive text valence attracted shorter reading times than perceived negative valence at both the supralexical and lexical level. In line with previous findings, no emotion effects on eye movements in picture viewing were found. This is the first eye tracking study reporting superior valence effects for vignettes compared to pictures and valence-specific effects on eye movements in reading at the supralexical level.

Neural dynamics of sentiment processing during naturalistic sentence reading

When we read, our eyes move through the text in a series of fixations and high-velocity saccades to extract visual information. This process allows the brain to obtain meaning, e.g., about sentiment, or the emotional valence, expressed in the written text. How exactly the brain extracts the sentiment of single words during naturalistic reading is largely unknown. This is due to the challenges of naturalistic imaging, which has previously led researchers to employ highly controlled, timed word-by-word presentations of custom reading materials that lack ecological validity. Here, we aimed to assess the electrical neural correlates of word sentiment processing during naturalistic reading of English sentences. We used a publicly available dataset of simultaneous electroencephalography (EEG), eye-tracking recordings, and word-level semantic annotations from 7129 words in 400 sentences (Zurich Cognitive Language Processing Corpus; Hollenstein et al., 2018). We computed fixation-related potentials (FRPs), which are evoked electrical responses time-locked to the onset of fixations. A general linear mixed model analysis of FRPs cleaned from visual- and motor-evoked activity showed a topographical difference between the positive and negative sentiment condition in the 224-304 ​ms interval after fixation onset in left-central and right-posterior electrode clusters. An additional analysis that included word-, phrase-, and sentence-level sentiment predictors showed the same FRP differences for the word-level sentiment, but no additional FRP differences for phrase- and sentence-level sentiment. Furthermore, decoding analysis that classified word sentiment (positive or negative) from sentiment-matched 40-trial average FRPs showed a 0.60 average accuracy (95% confidence interval: [0.58, 0.61]). Control analyses ruled out that these results were based on differences in eye movements or linguistic features other than word sentiment. Our results extend previous research by showing that the emotional valence of lexico-semantic stimuli evoke a fast electrical neural response upon word fixation during naturalistic reading. These results provide an important step to identify the neural processes of lexico-semantic processing in ecologically valid conditions and can serve to improve computer algorithms for natural language processing.

Optimizing the ICA-based removal of ocular EEG artifacts from free viewing experiments

Combining EEG with eye-tracking is a promising approach to study neural correlates of natural vision, but the resulting recordings are also heavily contaminated by activity of the eye balls, eye lids, and extraocular muscles. While Independent Component Analysis (ICA) is commonly used to suppress these ocular artifacts, its performance under free viewing conditions has not been systematically evaluated and many published reports contain residual artifacts. Here I evaluated and optimized ICA-based correction for two tasks with unconstrained eye movements: visual search in images and sentence reading. In a first step, four parameters of the ICA pipeline were varied orthogonally: the (1) high-pass and (2) low-pass filter applied to the training data, (3) the proportion of training data containing myogenic saccadic spike potentials (SP), and (4) the threshold for eye tracker-based component rejection. In a second step, the eye-tracker was used to objectively quantify the correction quality of each ICA solution, both in terms of undercorrection (residual artifacts) and overcorrection (removal of neurogenic activity). As a benchmark, results were compared to those obtained with an alternative spatial filter, Multiple Source Eye Correction (MSEC). With commonly used settings, Infomax ICA not only left artifacts in the data, but also distorted neurogenic activity during eye movement-free intervals. However, correction results could be strongly improved by training the ICA on optimally filtered data in which SPs were massively overweighted. With optimized procedures, ICA removed virtually all artifacts, including the SP and its associated spectral broadband artifact from both viewing paradigms, with little distortion of neural activity. It also outperformed MSEC in terms of SP correction. Matlab code is provided.

Misleading emotions: judgments of learning overestimate recognition of negative and positive emotional images

Research has shown that memory predictions are higher for emotional words, pictures, and facial expressions, relative to neutral stimuli, with recognition memory performance often not following the same pattern as predictions. Memory predictions for negative emotional images have not yet been examined. The current study examined how memory predictions and recognition memory for negative and positive emotional images differed from neutral images. Participants studied a mixed list of positive, negative, and neutral images and predicted future recognition by providing judgements of learning (JOLs). JOLs were highest for negative images, followed by positive images and then neutral images. However, recognition accuracy showed the opposite pattern: neutral images were recognised most accurately and negative images were the most poorly recognised. Participants incorporate beliefs and subjective experience in predicting recognition of emotional images, but fail to account for the influences of study and test conditions.

NEVER forget: negative emotional valence enhances recapitulation

A hallmark feature of episodic memory is that of "mental time travel," whereby an individual feels they have returned to a prior moment in time. Cognitive and behavioral neuroscience methods have revealed a neurobiological counterpart: Successful retrieval often is associated with reactivation of a prior brain state. We review the emerging literature on memory reactivation and recapitulation, and we describe evidence for the effects of emotion on these processes. Based on this review, we propose a new model: Negative Emotional Valence Enhances Recapitulation (NEVER). This model diverges from existing models of emotional memory in three key ways. First, it underscores the effects of emotion during retrieval. Second, it stresses the importance of sensory processing to emotional memory. Third, it emphasizes how emotional valence - whether an event is negative or positive - affects the way that information is remembered. The model specifically proposes that, as compared to positive events, negative events both trigger increased encoding of sensory detail and elicit a closer resemblance between the sensory encoding signature and the sensory retrieval signature. The model also proposes that negative valence enhances the reactivation and storage of sensory details over offline periods, leading to a greater divergence between the sensory recapitulation of negative and positive memories over time. Importantly, the model proposes that these valence-based differences occur even when events are equated for arousal, thus rendering an exclusively arousal-based theory of emotional memory insufficient. We conclude by discussing implications of the model and suggesting directions for future research to test the tenets of the model.

Improving free-viewing fixation-related EEG potentials with continuous-time regression

BACKGROUND

In the analysis of combined ET-EEG data, there are several issues with estimating FRPs by averaging. Neural responses associated with fixations will likely overlap with one another in the EEG recording and neural responses change as a function of eye movement characteristics. Especially in tasks that do not constrain eye movements in any way, these issues can become confounds.

NEW METHOD

Here, we propose the use of regression based estimates as an alternative to averaging. Multiple regression can disentangle different influences on the EEG and correct for overlap. It thereby accounts for potential confounds in a way that averaging cannot. Specifically, we test the applicability of the rERP framework, as proposed by Smith and Kutas (2015b), (2017), or Sassenhagen (2018) to combined eye tracking and EEG data from a visual search and a scene memorization task.

RESULTS

Results show that the method successfully estimates eye movement related confounds in real experimental data, so that these potential confounds can be accounted for when estimating experimental effects.

COMPARISON WITH EXISTING METHODS

The rERP method successfully corrects for overlapping neural responses in instances where averaging does not. As a consequence, baselining can be applied without risking distortions. By estimating a known experimental effect, we show that rERPs provide an estimate with less variance and more accuracy than averaged FRPs. The method therefore provides a practically feasible and favorable alternative to averaging.

CONCLUSIONS

We conclude that regression based ERPs provide novel opportunities for estimating fixation related EEG in free-viewing experiments.

Target probability modulates fixation-related potentials in visual search

This study investigated the influence of target probability on the neural response to target detection in free viewing visual search. Participants were asked to indicate the number of targets (one or two) among distractors in a visual search task while EEG and eye movements were co-registered. Target probability was manipulated by varying the set size of the displays between 10, 22, and 30 items. Fixation-related potentials time-locked to first target fixations revealed a pronounced P300 at the centro-parietal cortex with larger amplitudes for set sizes 22 and 30 than for set size 10. With increasing set size, more distractor fixations preceded the detection of the target, resulting in a decreased target probability and, consequently, a larger P300. For distractors, no increase of P300 amplitude with set size was observed. The findings suggest that set size specifically affects target but not distractor processing in overt serial visual search.

Temporal Dynamics of Natural Static Emotional Facial Expressions Decoding: A Study Using Event- and Eye Fixation-Related Potentials

This study aims at examining the precise temporal dynamics of the emotional facial decoding as it unfolds in the brain, according to the emotions displayed. To characterize this processing as it occurs in ecological settings, we focused on unconstrained visual explorations of natural emotional faces (i.e., free eye movements). The General Linear Model (GLM; Smith and Kutas, 2015a,b; Kristensen et al., 2017a) enables such a depiction. It allows deconvolving adjacent overlapping responses of the eye fixation-related potentials (EFRPs) elicited by the subsequent fixations and the event-related potentials (ERPs) elicited at the stimuli onset. Nineteen participants were displayed with spontaneous static facial expressions of emotions (Neutral, Disgust, Surprise, and Happiness) from the DynEmo database (Tcherkassof et al., 2013). Behavioral results on participants' eye movements show that the usual diagnostic features in emotional decoding (eyes for negative facial displays and mouth for positive ones) are consistent with the literature. The impact of emotional category on both the ERPs and the EFRPs elicited by the free exploration of the emotional faces is observed upon the temporal dynamics of the emotional facial expression processing. Regarding the ERP at stimulus onset, there is a significant emotion-dependent modulation of the P2-P3 complex and LPP components' amplitude at the left frontal site for the ERPs computed by averaging. Yet, the GLM reveals the impact of subsequent fixations on the ERPs time-locked on stimulus onset. Results are also in line with the valence hypothesis. The observed differences between the two estimation methods (Average vs. GLM) suggest the predominance of the right hemisphere at the stimulus onset and the implication of the left hemisphere in the processing of the information encoded by subsequent fixations. Concerning the first EFRP, the Lambda response and the P2 component are modulated by the emotion of surprise compared to the neutral emotion, suggesting an impact of high-level factors, in parieto-occipital sites. Moreover, no difference is observed on the second and subsequent EFRP. Taken together, the results stress the significant gain obtained in analyzing the EFRPs using the GLM method and pave the way toward efficient ecological emotional dynamic stimuli analyses.

Visual Complexity and Affect: Ratings Reflect More Than Meets the Eye

Pictorial stimuli can vary on many dimensions, several aspects of which are captured by the term 'visual complexity.' Visual complexity can be described as, "a picture of a few objects, colors, or structures would be less complex than a very colorful picture of many objects that is composed of several components." Prior studies have reported a relationship between affect and visual complexity, where complex pictures are rated as more pleasant and arousing. However, a relationship in the opposite direction, an effect of affect on visual complexity, is also possible; emotional arousal and valence are known to influence selective attention and visual processing. In a series of experiments, we found that ratings of visual complexity correlated with affective ratings, and independently also with computational measures of visual complexity. These computational measures did not correlate with affect, suggesting that complexity ratings are separately related to distinct factors. We investigated the relationship between affect and ratings of visual complexity, finding an 'arousal-complexity bias' to be a robust phenomenon. Moreover, we found this bias could be attenuated when explicitly indicated but did not correlate with inter-individual difference measures of affective processing, and was largely unrelated to cognitive and eyetracking measures. Taken together, the arousal-complexity bias seems to be caused by a relationship between arousal and visual processing as it has been described for the greater vividness of arousing pictures. The described arousal-complexity bias is also of relevance from an experimental perspective because visual complexity is often considered a variable to control for when using pictorial stimuli.

Attention in natural scenes: Affective-motivational factors guide gaze independently of visual salience

In addition to low-level stimulus characteristics and current goals, our previous experience with stimuli can also guide attentional deployment. It remains unclear, however, if such effects act independently or whether they interact in guiding attention. In the current study, we presented natural scenes including every-day objects that differed in affective-motivational impact. In the first free-viewing experiment, we presented visually-matched triads of scenes in which one critical object was replaced that varied mainly in terms of motivational value, but also in terms of valence and arousal, as confirmed by ratings by a large set of observers. Treating motivation as a categorical factor, we found that it affected gaze. A linear-effect model showed that arousal, valence, and motivation predicted fixations above and beyond visual characteristics, like object size, eccentricity, or visual salience. In a second experiment, we experimentally investigated whether the effects of emotion and motivation could be modulated by visual salience. In a medium-salience condition, we presented the same unmodified scenes as in the first experiment. In a high-salience condition, we retained the saturation of the critical object in the scene, and decreased the saturation of the background, and in a low-salience condition, we desaturated the critical object while retaining the original saturation of the background. We found that highly salient objects guided gaze, but still found additional additive effects of arousal, valence and motivation, confirming that higher-level factors can also guide attention, as measured by fixations towards objects in natural scenes.

Emotional Modulation of the Late Positive Potential during Picture Free Viewing in Older and Young Adults

Hedonic bias during free viewing of novel emotional and neutral scenes was investigated in older adults and college students. A neurophysiological index of emotional picture processing–the amplitude of the centroparietal late positive potential (LPP)–was recorded from the scalp using a dense sensor array while participants (29 older adults; 21 college students) viewed emotionally engaging or mundane natural scenes that varied in specific content. Both students and older adults showed LPP enhancement when viewing affective, compared to neutral, scenes, and there was no difference in LPP amplitude between older individuals and college students when viewing neutral everyday scenes. However, compared to the college students, older individuals showed attenuated LPP amplitude when viewing emotional scenes, regardless of hedonic valence or specific content. Age related differences could be mediated by a reduction in reactive emotional arousal with age, possible mediated by repeated life exposure to emotional stimuli.