Independent effects of motivation and spatial attention in the human visual cortex

Distinct Patterns of Connectivity between Brain Regions Underlie the Intra-Modal and Cross-Modal Value-Driven Modulations of the Visual Cortex

Past reward associations may be signaled from different sensory modalities; however, it remains unclear how different types of reward-associated stimuli modulate sensory perception. In this human fMRI study (female and male participants), a visual target was simultaneously presented with either an intra- (visual) or a cross-modal (auditory) cue that was previously associated with rewards. We hypothesized that, depending on the sensory modality of the cues, distinct neural mechanisms underlie the value-driven modulation of visual processing. Using a multivariate approach, we confirmed that reward-associated cues enhanced the target representation in early visual areas and identified the brain valuation regions. Then, using an effective connectivity analysis, we tested three possible patterns of connectivity that could underlie the modulation of the visual cortex: a direct pathway from the frontal valuation areas to the visual areas, a mediated pathway through the attention-related areas, and a mediated pathway that additionally involved sensory association areas. We found evidence for the third model demonstrating that the reward-related information in both sensory modalities is communicated across the valuation and attention-related brain regions. Additionally, the superior temporal areas were recruited when reward was cued cross-modally. The strongest dissociation between the intra- and cross-modal reward-driven effects was observed at the level of the feedforward and feedback connections of the visual cortex estimated from the winning model. These results suggest that, in the presence of previously rewarded stimuli from different sensory modalities, a combination of domain-general and domain-specific mechanisms are recruited across the brain to adjust the visual perception.SIGNIFICANCE STATEMENT Reward has a profound effect on perception, but it is not known whether shared or disparate mechanisms underlie the reward-driven effects across sensory modalities. In this human fMRI study, we examined the reward-driven modulation of the visual cortex by visual (intra-modal) and auditory (cross-modal) reward-associated cues. Using a model-based approach to identify the most plausible pattern of inter-regional effective connectivity, we found that higher-order areas involved in the valuation and attentional processing were recruited by both types of rewards. However, the pattern of connectivity between these areas and the early visual cortex was distinct between the intra- and cross-modal rewards. This evidence suggests that, to effectively adapt to the environment, reward signals may recruit both domain-general and domain-specific mechanisms.

Differential effects of intra-modal and cross-modal reward value on perception: ERP evidence

In natural environments objects comprise multiple features from the same or different sensory modalities but it is not known how perception of an object is affected by the value associations of its constituent parts. The present study compares intra- and cross-modal value-driven effects on behavioral and electrophysiological correlates of perception. Human participants first learned the reward associations of visual and auditory cues. Subsequently, they performed a visual discrimination task in the presence of previously rewarded, task-irrelevant visual or auditory cues (intra- and cross-modal cues, respectively). During the conditioning phase, when reward associations were learned and reward cues were the target of the task, high value stimuli of both modalities enhanced the electrophysiological correlates of sensory processing in posterior electrodes. During the post-conditioning phase, when reward delivery was halted and previously rewarded stimuli were task-irrelevant, cross-modal value significantly enhanced the behavioral measures of visual sensitivity, whereas intra-modal value produced only an insignificant decrement. Analysis of the simultaneously recorded event-related potentials (ERPs) of posterior electrodes revealed similar findings. We found an early (90-120 ms) suppression of ERPs evoked by high-value, intra-modal stimuli. Cross-modal stimuli led to a later value-driven modulation, with an enhancement of response positivity for high- compared to low-value stimuli starting at the N1 window (180-250 ms) and extending to the P3 (300-600 ms) responses. These results indicate that sensory processing of a compound stimulus comprising a visual target and task-irrelevant visual or auditory cues is modulated by the reward value of both sensory modalities, but such modulations rely on distinct underlying mechanisms.

Estimation Strategy Utilization Is Modulated by Implicit Emotion Regulation: Evidence from Behavioral and Event-Related Potentials Studies

A large number of studies have studied the influence of emotional experience on an individual's estimation performance, but the influence of implicit emotion regulation is still unknown. Participants were asked to complete the following tasks in order: idiom matching task, multiplication computational estimation task (MCE task), gender judgment task (GJ task), and emotional experience intensity assessment task. The words matching task was adopted to achieve the purpose of implicit emotion regulation (implicit reappraisal and implicit suppression). Behavioral results showed that implicit reappraisal and implicit suppression equally contributed to improving an individual's estimation speed (but not ACC (accuracy)). The MCE task related ERP (event-related potential) results showed that the influence of implicit emotion regulation on estimation consisted of two phases. In the first phase (encoding phase), implicit reappraisal both enhanced (larger P1 amplitudes) and weakened (smaller N170 amplitudes) an individual's encoding sensitivity, while implicit suppression enhanced an individual's encoding sensitivity (larger P1 amplitudes). In the second phase (estimation strategies retrieval phase), implicit reappraisal (but not implicit suppression) cost more attention resources (larger LPC2 and LPC3 amplitudes). The present study suggested that both implicit reappraisal and implicit suppression contributed to improving an individual's estimation performance, and the regulation effect of implicit suppression (vs. implicit reappraisal) was better.

Implicit happy and fear experience contributes to computational estimation strategy execution: Behavioral and neurophysiological evidence

Previous study have explored the influence of explicit emotion priming on computational estimation strategy execution, but the corresponding influence of implicit emotion priming still remains unknown. The present study aimed to solve this problem. Participants were asked to complete two-digit multiplication computational estimation task, under different implicit emotion priming conditions (gender judgment task). In the two-digit multiplication computational estimation task, the computational estimation question was presented in the middle of the screen, meanwhile, two alternative answers were presented side by side at the bottom of the screen, participants were required to select which answer is correct, by using the down-up strategy (e.g., doing 30 × 50 = 1500 for 34 × 46). Behavioral results showed that the response speed under implicit happy and fear (vs. neutral and angry) priming conditions was quicker, and the accuracy under different priming conditions showed no significant difference. The ERP results showed that the influence of implicit emotion priming on computational estimation strategy execution consisted of two phases: in the first phase, the N1 amplitudes elicited by completing the multiplication computational estimation task were smaller under implicit fear (vs. angry) priming condition; in the second phase, the corresponding P2 amplitudes under implicit happy (vs. fear) priming condition were smaller. The present study indicated that implicit happy and fear experience contributed to complete computational estimation tasks, suggesting that implicit negative emotional (e.g., fear) experience was not always detrimental to computational estimation strategy execution.

Emotional valence modulates arithmetic strategy execution in priming paradigm: an event-related potential study

Combined with the prime paradigm, the present study aimed to explore the influence of emotion (anger, fear, happiness, and neutral) on performing multiplication estimation. Participants were asked to complete a two-digit multiplication estimation task using the down-up strategy (e.g., doing 20 × 80 = 1600 for 24 × 79). Behavioral results showed that the reaction time for completing multiplication estimation tasks under happy conditions was shorter than for those under anger and fear, and it was shorter under neutral than under fearful conditions. The ERP results showed that about 100 ms after multiplication estimation task onset, multiplication estimation execution in the context of happiness (vs. neutral) elicited smaller P1 amplitudes; about 170 ms after multiplication estimation task onset, the N170 amplitudes elicited by multiplication estimation strategy execution under different emotional priming conditions showed no significant differences. The above results showed that the impact of emotion priming demonstrates the occurrence of a dynamic process when participants use a specified strategy to complete the multiplication estimation task. The present study revealed that emotional valence modulated arithmetic strategy execution, suggesting the role of different emotions should be fully considered in similar study.

No Evidence that Predictions and Attention Modulate the First Feedforward Sweep of Cortical Information Processing

Predictive coding models propose that predictions (stimulus likelihood) reduce sensory signals as early as primary visual cortex (V1), and that attention (stimulus relevance) can modulate these effects. Indeed, both prediction and attention have been shown to modulate V1 activity, albeit with fMRI, which has low temporal resolution. This leaves it unclear whether these effects reflect a modulation of the first feedforward sweep of visual information processing and/or later, feedback-related activity. In two experiments, we used electroencephalography and orthogonally manipulated spatial predictions and attention to address this issue. Although clear top-down biases were found, as reflected in pre-stimulus alpha-band activity, we found no evidence for top-down effects on the earliest visual cortical processing stage (<80 ms post-stimulus), as indexed by the amplitude of the C1 event-related potential component and multivariate pattern analyses. These findings indicate that initial visual afferent activity may be impenetrable to top-down influences by spatial prediction and attention.

Spatiotemporal pattern of appraising social and emotional relevance: Evidence from event-related brain potentials

Social information is particularly relevant for the human species because of its direct link to guiding physiological responses and behavior. Accordingly, extant functional magnetic resonance imaging (fMRI) data suggest that social content may form a unique stimulus dimension. It remains largely unknown, however, how neural activity underlying social (versus nonsocial) information processing temporally unfolds, and how such social information appraisal may interact with the processing of other stimulus characteristics, particularly emotional meaning. Here, we presented complex visual scenes differing in both social (vs. nonsocial) and emotional relevance (positive, negative, neutral) intermixed with scrambled versions of these pictures to N = 24 healthy young adults. Event-related brain potentials (ERPs) to intact pictures were examined for gaining insight to the dynamics of appraisal of both dimensions, implemented within the brain. Our main finding is an early interaction between social and emotional relevance due to enhanced amplitudes of early ERP components to emotionally positive and neutral pictures of social compared to nonsocial content, presumably reflecting rapid allocation of attention and counteracting an overall negativity bias. Importantly, our ERP data show high similarity with previously observed fMRI data using the same stimuli, and source estimations located the ERP effects in overlapping occipitotemporal brain areas. Our novel data suggest that relevance detection may occur already as early as around 100 ms after stimulus onset and may combine relevance checks not only examining intrinsic pleasantness/emotional valence but also social content as a unique, highly relevant stimulus dimension.

The Puzzling Relationship between Attention and Motivation: Do Motor Biases Matter?

The relationship between attention and incentive motivation has been mostly examined by administering Posner style cueing tasks in humans and varying monetary stakes. These studies found that higher incentives improved performance independently of spatial attention. However, the ability of the cueing task to measure actual attentional orienting has been debated by several groups that have highlighted the function of the motor system in affecting the behavioral features that are commonly attributed to spatial attention. To determine the impact of motor influences on the interplay between attention and motivation, we administered 2 reaching versions of a cueing task to monkeys in various motor scenarios. In both tasks, a central stimulus indicated the reward stake and predicted the stimulus target location in 80% of trials. In Experiment 1, subjects were requested to report the detection of a target stimulus in each trial. In Experiment 2, the task was modified to fit a paradigm of Go/NoGo target identification. We found that attention and motivation interacted exclusively in Experiment 2, wherein anticipated motor activation was discouraged and more demanding visual processing was imposed. Consequently, we suggest a protocol that provides novel insights into the study of the relationship between spatial attention and motivation and highlights the influence of the arm motor system in the estimation of the deployment of spatial attention.

Punishment-related memory-guided attention: Neural dynamics of perceptual modulation

Remembering the outcomes of past experiences allows us to generate future expectations and shape selection in the long-term. A growing number of studies has shown that learned positive reward values impact spatial memory-based attentional biases on perception. However, whether memory-driven attentional biases extend to punishment-related values has received comparatively less attention. Here, we manipulated whether recent spatial contextual memories became associated with successful avoidance of punishment (potential monetary loss). Behavioral and electrophysiological measures were collected from 27 participants during a subsequent memory-based attention task, in which we tested for the effect of punishment avoidance associations. Punishment avoidance significantly amplified effects of spatial contextual memories on visual search processes within natural scenes. Compared to non-associated scenes, contextual memories paired with punishment avoidance lead to faster responses to targets presented at remembered locations. Event-related potentials elicited by target stimuli revealed that acquired motivational value of specific spatial locations, by virtue of their association with past avoidance of punishment, dynamically affected neural signatures of early visual processing (indexed by larger P1 and earlier N1 potentials) and target selection (as indicated by reduced N2pc potentials). The present results extend our understanding of how memory, attention, and punishment-related mechanisms interact to optimize perceptual decision in real world environments.

Single-trial EEG dissociates motivation and conflict processes during decision-making under risk

In making decisions under risk (i.e., choosing whether to gamble when the outcome probabilities are known), two aspects of decision are of particular concern. The first, if gambling, is how large are potential gains compared to losses? The subjectively larger, the more rewarding to gamble. Thus, this aspect of decision-making, quantified through expected utility (EU), is motivation-related. The second concern is how easy is it to reach the decision? When subjective desirability between gambling and not-gambling is clearly different from each other (regardless of the direction), it is easier to decide. This aspect, quantified through utility distance (UD), is conflict-related. It is unclear how the brain simultaneously processes these two aspects of decision-making. Forty-five participants decided whether to gamble during electroencephalogram (EEG) recording. To compute trial-by-trial variability in EU and UD, we fit participants' choices to models inspired by Expected-Utility and Prospect theories using hierarchical-Bayesian modeling. To examine unique influences of EU and UD, we conducted model-based single-trial EEG analyses with EU and UD as simultaneous regressors. While both EU and UD were positively associated with P3-like activity and delta-band power, the contribution of EU was around 200 ms earlier. Thus, during decision-making under risk, people may allocate their attention to motivation-related aspects before conflict-related aspects. Next, following learning the options and before reporting their decision, higher EU was associated with stronger alpha and beta suppression, while higher UD was associated with a stronger contingent-negativity-variation-like activity. This suggests distinct roles of EU and UD on anticipation-related processes. Overall, we identified time and frequency characteristics of EEG signals that differentially traced motivation-related and conflict-related information during decision-making under risk.

Money or smiles: Independent ERP effects of associated monetary reward and happy faces

In comparison to neutral faces, facial expressions of emotion are known to gain attentional prioritization, mainly demonstrated by means of event-related potentials (ERPs). Recent evidence indicated that such a preferential processing can also be elicited by neutral faces when associated with increased motivational salience via reward. It remains, however, an open question whether impacts of inherent emotional salience and associated motivational salience might be integrated. To this aim, expressions and monetary outcomes were orthogonally combined. Participants (N = 42) learned to explicitly categorize happy and neutral faces as either reward- or zero-outcome-related via an associative learning paradigm. ERP components (P1, N170, EPN, and LPC) were measured throughout the experiment, and separately analyzed before (learning phase) and after (consolidation phase) reaching a pre-defined learning criterion. Happy facial expressions boosted early processing stages, as reflected in enhanced amplitudes of the N170 and EPN, both during learning and consolidation. In contrast, effects of monetary reward became evident only after successful learning and in form of enlarged amplitudes of the LPC, a component linked to higher-order evaluations. Interactions between expressions and associated outcome were absent in all ERP components of interest. The present study provides novel evidence that acquired salience impacts stimulus processing but independent of the effects driven by happy facial expressions.

The P1 visual-evoked potential, red light, and transdiagnostic psychiatric symptoms

A reduced P1 visual-evoked potential amplitude has been reported across several psychiatric disorders, including schizophrenia-spectrum, bipolar, and depressive disorders. In addition, a difference in P1 amplitude change to a red background compared to its opponent color, green, has been found in schizophrenia-spectrum samples. The current study examined whether specific psychiatric symptoms that related to these P1 abnormalities in earlier studies would be replicated when using a broad transdiagnostic sample. The final sample consisted of 135 participants: 26 with bipolar disorders, 25 with schizophrenia-spectrum disorders, 19 with unipolar depression, 62 with no current psychiatric disorder, and 3 with disorders in other categories. Low (8%) and high (64%) contrast check arrays were presented on gray, green, and red background conditions during electroencephalogram, while an eye tracker monitored visual fixation on the stimuli. Linear regressions across the entire sample (N = 135) found that greater severity of both clinician-rated and self-reported delusions/magical thinking correlated with a reduced P1 amplitude on the low contrast gray (neutral) background condition. In addition, across the entire sample, higher self-reported constricted affect was associated with a larger decrease in P1 amplitude (averaged across contrast conditions) to the red, compared to green, background. All relationships remained statistically significant after covarying for diagnostic class, suggesting that they are relatively transdiagnostic in nature. These findings indicate that early visual processing abnormalities may be more directly related to specific transdiagnostic symptoms such as delusions and constricted affect rather than specific psychiatric diagnoses or broad symptom factor scales.

Hot Speech and Exploding Bombs: Autonomic Arousal During Emotion Classification of Prosodic Utterances and Affective Sounds

Emotional expressions provide strong signals in social interactions and can function as emotion inducers in a perceiver. Although speech provides one of the most important channels for human communication, its physiological correlates, such as activations of the autonomous nervous system (ANS) while listening to spoken utterances, have received far less attention than in other domains of emotion processing. Our study aimed at filling this gap by investigating autonomic activation in response to spoken utterances that were embedded into larger semantic contexts. Emotional salience was manipulated by providing information on alleged speaker similarity. We compared these autonomic responses to activations triggered by affective sounds, such as exploding bombs, and applause. These sounds had been rated and validated as being either positive, negative, or neutral. As physiological markers of ANS activity, we recorded skin conductance responses (SCRs) and changes of pupil size while participants classified both prosodic and sound stimuli according to their hedonic valence. As expected, affective sounds elicited increased arousal in the receiver, as reflected in increased SCR and pupil size. In contrast, SCRs to angry and joyful prosodic expressions did not differ from responses to neutral ones. Pupil size, however, was modulated by affective prosodic utterances, with increased dilations for angry and joyful compared to neutral prosody, although the similarity manipulation had no effect. These results indicate that cues provided by emotional prosody in spoken semantically neutral utterances might be too subtle to trigger SCR, although variation in pupil size indicated the salience of stimulus variation. Our findings further demonstrate a functional dissociation between pupil dilation and skin conductance that presumably origins from their differential innervation.

Simultaneous EEG-fMRI reveals attention-dependent coupling of early face processing with a distributed cortical network

•

Distributed cortical activation to faces covaries with ERPs as early as 100 ms.

•

Covariations depend on both sustained attention and trial-by-trial cognitive conflict.

•

Top-down gating might apply to earlier visual processing stages than previously known.

The speed of visual processing is central to our understanding of face perception. Yet the extent to which early visual processing influences later processing in distributed face processing networks, and the top-down modulation of such bottom-up effects, remains unclear. We used simultaneous EEG-fMRI to investigate cortical activity that showed unique covariation with ERP components of face processing (C1, P1, N170, P3), while manipulating sustained attention and transient cognitive conflict employing an emotional face-word Stroop task.

ERP markers of visual processing within 100 ms after stimulus onset showed covariation with brain activation in precuneous, posterior cingulate gyrus, left middle temporal gyrus, left inferior frontal gyrus and frontal operculum, and a left lateral parietal-occipital cluster, illustrating the impact of early stage processing on higher-order mechanisms. Crucially, this covariation depended on sustained attentional focus and was absent for incongruent trials, suggesting flexible top-down gating of bottom-up processing.

How early does attention modulate visual information processing? The importance of experimental protocol and data analysis approach

Whether attention can influence afferent information processing in primary visual cortex (V1) has long been topic of scientific debate. Findings from a recent study by Baumgarter et al. (this issue) add to this debate by providing a null replication of an influential study that reported that spatial attention can enhance feedforward information processing in human V1, as reflected in the amplitude of the C1 ERP component (Kelly, Gomez-Raminez, & Foxe, 2008). Here we discuss several factors, including analytic approach, experimental design, and motivational factors, that, once scientifically tested, may help resolve discrepancies in the current literature.