Reward enhances cross‐modal conflict control in object categorization: Electrophysiological evidence

Proactive reward in conflict tasks: Does it only enhance general performance or also modulate conflict effects?

In the present study, we investigated the influence of performance-contingent reward prospects on task performance across three visual conflict tasks with manual responses (Experiments 1 & 2: Simon and Stroop tasks; Experiment 3: Simon and Eriksen flanker task) using block-wise (Experiment 1) and trial-wise (Experiments 2 & 3) manipulations to signal the possibility of reward. Across all experiments, task performance (in reaction time and/or error rates) generally improved in reward compared with no-reward conditions in each conflict task. However, there was, if any, little evidence that the reward manipulation modulated the size of the mean conflict effects, and there was also no evidence for conflict-specific effects of reward when controlling for time-varying fluctuations in conflict processing via distributional analyses (delta plots). Thus, the results provide no evidence for conflict-specific accounts and instead favor performance-general accounts, where reward anticipation leads to overall performance improvements without affecting conflict effects. We discuss possible implications for how proactive control might modulate the interplay between target- and distractor-processing in conflict tasks.

Reward delays quitting in visual search

Reward motivates goal-directed behaviors, leading to faster reaction time (RT) and lower error rate in searching for a target in the reward condition than in the no-reward condition in target-discrimination tasks. However, it is unclear how reward influences target detection in which participants are required to judge whether a predesignated target is present or absent. Here, we asked participants to complete a target-detection search task in which the color of the search array indicated the reward availability of the current trial. Correct and faster (than a baseline) responses would be rewarded if the search array had the reward-related color. In Experiments 1A and 1B, the target was presented in 50% of the trials. Experiment 1B had the same design as Experiment 1A, except that different baselines were set for the target-present and target-absent conditions. In Experiment 2, the proportion of target presence was manipulated to be high (80%), moderate (50%), or low (20%) in different blocks of stimuli. Results showed that, across all the experiments, participants responded faster and made fewer errors in the reward than in the no-reward condition when the target was present. However, this facilitatory effect was reversed when the target was absent, showcasing a reward-induced interference. The signal detection analysis suggested that reward biased the report criterion to the "yes" response. These findings demonstrate that the impact of reward on goal-directed behavior can be detrimental and reward prolongs the search process by rendering participants reluctant to say "no" in visual search termination.

The dissociation of semantically congruent and incongruent cross-modal effects on the visual attentional blink

Introduction

Recent studies have found that the sound-induced alleviation of visual attentional blink, a well-known phenomenon exemplifying the beneficial influence of multisensory integration on time-based attention, was larger when that sound was semantically congruent relative to incongruent with the second visual target (T2). Although such an audiovisual congruency effect has been attributed mainly to the semantic conflict carried by the incongruent sound restraining that sound from facilitating T2 processing, it is still unclear whether the integrated semantic information carried by the congruent sound benefits T2 processing.

Methods

To dissociate the congruence-induced benefit and incongruence-induced reduction in the alleviation of visual attentional blink at the behavioral and neural levels, the present study combined behavioral measures and event-related potential (ERP) recordings in a visual attentional blink task wherein the T2-accompanying sound, when delivered, could be semantically neutral in addition to congruent or incongruent with respect to T2.

Results

The behavioral data clearly showed that compared to the neutral sound, the congruent sound improved T2 discrimination during the blink to a higher degree while the incongruent sound improved it to a lesser degree. The T2-locked ERP data revealed that the early occipital cross-modal N195 component (192-228 ms after T2 onset) was uniquely larger in the congruent-sound condition than in the neutral-sound and incongruent-sound conditions, whereas the late parietal cross-modal N440 component (400-500 ms) was prominent only in the incongruent-sound condition.

Discussion

These findings provide strong evidence that the modulating effect of audiovisual semantic congruency on the sound-induced alleviation of visual attentional blink contains not only a late incongruence-induced cost but also an early congruence-induced benefit, thereby demonstrating for the first time an unequivocal congruent-sound-induced benefit in alleviating the limitation of time-based visual attention.

Cross-modal enhancement of spatially unpredictable visual target discrimination during the attentional blink

The attentional blink can be substantially reduced by delivering a task-irrelevant sound synchronously with the second target (T2) embedded in a rapid serial visual presentation stream, which is further modulated by the semantic congruency between the sound and T2. The present study extended the cross-modal boost during attentional blink and the modulation of audiovisual semantic congruency in the spatial domain by showing that a spatially uninformative, semantically congruent (but not incongruent) sound could even improve the discrimination of spatially unpredictable T2 during attentional blink. T2-locked event-related potential (ERP) data yielded that the early cross-modal P195 difference component (184-234 ms) over the occipital scalp contralateral to the T2 location was larger preceding accurate than inaccurate discriminations of semantically congruent, but not incongruent, audiovisual T2s. Interestingly, the N2pc component (194-244 ms) associated with visual-spatial attentional allocation was enlarged for incongruent audiovisual T2s relative to congruent audiovisual and unisensory visual T2s only when they were accurately discriminated. These ERP findings suggest that the spatially extended cross-modal boost during attentional blink involves an early cross-modal interaction strengthening the perceptual processing of T2, without any sound-induced enhancement of visual-spatial attentional allocation toward T2. In contrast, the absence of an accuracy decrease in response to semantically incongruent audiovisual T2s may originate from the semantic mismatch capturing extra visual-spatial attentional resources toward T2.

Differential effects of intra-modal and cross-modal reward value on perception: ERP evidence

In natural environments objects comprise multiple features from the same or different sensory modalities but it is not known how perception of an object is affected by the value associations of its constituent parts. The present study compares intra- and cross-modal value-driven effects on behavioral and electrophysiological correlates of perception. Human participants first learned the reward associations of visual and auditory cues. Subsequently, they performed a visual discrimination task in the presence of previously rewarded, task-irrelevant visual or auditory cues (intra- and cross-modal cues, respectively). During the conditioning phase, when reward associations were learned and reward cues were the target of the task, high value stimuli of both modalities enhanced the electrophysiological correlates of sensory processing in posterior electrodes. During the post-conditioning phase, when reward delivery was halted and previously rewarded stimuli were task-irrelevant, cross-modal value significantly enhanced the behavioral measures of visual sensitivity, whereas intra-modal value produced only an insignificant decrement. Analysis of the simultaneously recorded event-related potentials (ERPs) of posterior electrodes revealed similar findings. We found an early (90-120 ms) suppression of ERPs evoked by high-value, intra-modal stimuli. Cross-modal stimuli led to a later value-driven modulation, with an enhancement of response positivity for high- compared to low-value stimuli starting at the N1 window (180-250 ms) and extending to the P3 (300-600 ms) responses. These results indicate that sensory processing of a compound stimulus comprising a visual target and task-irrelevant visual or auditory cues is modulated by the reward value of both sensory modalities, but such modulations rely on distinct underlying mechanisms.

Effect of Audiovisual Cross-Modal Conflict during Working Memory Tasks: A Near-Infrared Spectroscopy Study

Cognitive conflict effects are well characterized within unimodality. However, little is known about cross-modal conflicts and their neural bases. This study characterizes the two types of visual and auditory cross-modal conflicts through working memory tasks and brain activities. The participants consisted of 31 healthy, right-handed, young male adults. The Paced Auditory Serial Addition Test (PASAT) and the Paced Visual Serial Addition Test (PVSAT) were performed under distractor and no distractor conditions. Distractor conditions comprised two conditions in which either the PASAT or PVSAT was the target task, and the other was used as a distractor stimulus. Additionally, oxygenated hemoglobin (Oxy-Hb) concentration changes in the frontoparietal regions were measured during tasks. The results showed significantly lower PASAT performance under distractor conditions than under no distractor conditions, but not in the PVSAT. Oxy-Hb changes in the bilateral ventrolateral prefrontal cortex (VLPFC) and inferior parietal cortex (IPC) significantly increased in the PASAT with distractor compared with no distractor conditions, but not in the PVSAT. Furthermore, there were significant positive correlations between Δtask performance accuracy and ΔOxy-Hb in the bilateral IPC only in the PASAT. Visual cross-modal conflict significantly impairs auditory task performance, and bilateral VLPFC and IPC are key regions in inhibiting visual cross-modal distractors.

The interplay between audiovisual temporal synchrony and semantic congruency in the cross-modal boost of the visual target discrimination during the attentional blink

The visual attentional blink can be substantially reduced by delivering a task-irrelevant sound synchronously with the second visual target (T2), and this effect is further modulated by the semantic congruency between the sound and T2. However, whether the cross-modal benefit originates from audiovisual interactions or sound-induced alertness remains controversial, and whether the semantic congruency effect is contingent on audiovisual temporal synchrony needs further investigation. The current study investigated these questions by recording event-related potentials (ERPs) in a visual attentional blink task wherein a sound could either synchronize with T2, precede T2 by 200 ms, be delayed by 100 ms, or be absent, and could be either semantically congruent or incongruent with T2 when delivered. The behavioral data showed that both the cross-modal boost of T2 discrimination and the further semantic modulation were the largest when the sound synchronized with T2. In parallel, the ERP data yielded that both the early occipital cross-modal P195 component (192-228 ms after T2 onset) and late parietal cross-modal N440 component (424-448 ms) were prominent only when the sound synchronized with T2, with the former being elicited solely when the sound was further semantically congruent whereas the latter occurring only when that sound was incongruent. These findings demonstrate not only that the cross-modal boost of T2 discrimination during the attentional blink stems from early audiovisual interactions and the semantic congruency effect depends on audiovisual temporal synchrony, but also that the semantic modulation can unfold at the early stage of visual discrimination processing.

Reward facilitates response conflict resolution via global motor inhibition: Electromyography evidence

It is crucial for humans to coordinate between behavioural tendencies that can lead to reward but are in conflict with each other. This response conflict can be measured in a reward-modulated Simon task, in which a discriminative response to the identity of a lateral target is required and the target is associated with either high- or low-reward. Critically, the lateral target is presented either congruent or incongruent with the location of the responding hand. It has been shown that relative to the low-reward target, the high-reward target induced a larger response conflict when the target was incongruent with the position of the task-required, reward-obtaining hand. Here we investigated how this response conflict is resolved by acquiring 24 healthy participants' electromyography (EMG) signals from both the task-required responding hand (i.e., goal-directed effector) and the alternative hand (i.e., inappropriate effector). During the coping with the response conflict, motor inhibition (indexed by reduction in EMG signals between conditions) was observed not only at the inappropriate effector but also at the goal-directed effector. Individuals who showed stronger inhibition on the inappropriate effector suffered less from the inhibition on the goal-directed effector, and had more efficient implementation of the reward-obtaining response. Our findings suggest a global motor inhibition that may function to increase the signal-noise ratio in the motor system so as to implement reward-guided behavior.

The interactive effects of reward expectation and emotional interference on cognitive conflict control: An ERP study

The effects of reward expectation and task-irrelevant emotional content on performance and event-related potential (ERP) recordings in a cognitive conflict control task were investigated using the face-word Stroop paradigm. A precue indicating additional monetary rewards for fast and accurate responses during the upcoming trial (incentive condition; relative to a cue indicating no additional reward, i.e., nonincentive condition) was followed by the presentation of target Chinese words (male vs. female) superimposed on background emotional faces (happy vs. fearful). The face's gender was congruent or incongruent with the target Chinese words. ERP results revealed that incentive cues elicited larger P1, P3, and CNV responses compared to nonincentive cues. There was a significant three-way interaction of reward expectation, emotional content, and congruency during the target processing stage such that emotionality and congruency interacted to affect the N170 and N2 component responses during the nonincentive condition but not during the incentive condition. These results indicate that reward-induced motivation reduces the interference effect of task-irrelevant emotional information, leading to better conflict resolution.

Reward makes the rhythmic sampling of spatial attention emerge earlier

A growing body of evidence demonstrates a rhythmic characteristic of spatial attention, with the corresponding behavioral performance fluctuating periodically. Here, we investigate whether and how the rhythmic characteristic of spatial attention is affected by reward-an important factor in attentional selection. We adopted the classic spatial cueing paradigm with a time-resolved stimulus-onset-asynchrony (SOA) between the spatial cue and the target such that responses to the target in different phases could be examined. The color of the spatial cue was associated with either a high or low level of reward. Results showed that in the low-frequency band (<2 Hz) where classic exogenous spatial attention effects (i.e., facilitation and inhibition of return; IOR) appeared, reward enhanced the late IOR effect through facilitating behavioral responses to the target at the uncued location. Recurring lower alpha power (alpha inhibition) which fluctuated in a low-theta frequency (2-3 Hz) was observed at the cued location relative to the uncued location, irrespective of the reward level of the cue. Importantly, the recurring alpha inhibition emerged earlier (~120 ms) in the high-reward condition relative to the low-reward condition. We propose that the recurring alpha inhibition at the cued location implies a recurring attention sampling at the cued location and the expectation of a high reward makes the periodic attention sampling emerge earlier.

Cross-modal nonspatial repetition inhibition: An ERP study

Previous studies have demonstrated that nonspatial repetition inhibition can occur across modalities. However, the underlying mechanism of such cross-modal nonspatial repetition inhibition is unknown. The present experiment adopted a cross-modal prime-neutral cue-target paradigm in which in consecutive trials the prime and the target were matched or mismatched, not only in identity but also in modality. Meanwhile, event-related potentials (ERPs) to visual and auditory targets were recorded. The present study aimed to answer two questions: which ERP components reflect nonspatial repetition inhibition across modalities, and is the ERP component modality specific or supramodal? The results showed that for visual targets, robust nonspatial repetition inhibition occurred similarly for both unimodal (visual-visual) and cross-modal (audio-visual) target pairings, as indexed by an N400 repetition-induced increment in the typical N400 window but null effects during the N2 epoch. For auditory targets, similar modulation of cross-modal nonspatial repetition inhibition on the auditory-evoked N400 repetition-induced increment was observed. These results suggest that the N400 repetition-induced increment occurs during the N400 epoch that underlies cross-modal nonspatial repetition inhibition and that this N400 component is a supramodal component.

Reward expectation modulates multiple stages of auditory conflict control

Although mounting evidence has shown that reward can improve conflict control in the visual domain, little is known about whether and how reward affects conflict processing in the auditory domain. In the present study, we adopted an auditory Stroop task in which the meaning of a sound word ('male' or 'female') could be either congruent or incongruent with the gender of the voice (male or female speaker), and the participants were asked to discriminate the gender of the voice (the phonetic task) or the meaning of the word (the semantic task). Importantly, an auditory cue signalling a potential reward or no-reward for the current trial was presented prior to the sound word. In both tasks, relative to the congruent sound word, response to the incongruent sound word was delayed, i.e., an auditory Stroop effect. However, this auditory Stroop effect was reduced following a reward cue relative to a no-reward cue. Event-related potentials (ERPs) showed a stronger contingent negativity variation (CNV, 1000-1500 ms) for the reward cue than for the no-reward cue. The conflict negativity N_inc (300-400 ms) was more negative-going for the incongruent word than for the congruent word, but this effect was significantly reduced in the reward condition. However, the late positive complex (LPC) showed at most a weak reward modulation. These findings suggest that reward expectation improves auditory conflict control by modulating different stages of conflict processing: promoting better attentional preparation for the upcoming target (CNV), and facilitating conflict detection (N_inc) on the presentation of the target.