The beneficial effects of sounds on attentional blink performance: An ERP study

The dissociation of semantically congruent and incongruent cross-modal effects on the visual attentional blink

Introduction

Recent studies have found that the sound-induced alleviation of visual attentional blink, a well-known phenomenon exemplifying the beneficial influence of multisensory integration on time-based attention, was larger when that sound was semantically congruent relative to incongruent with the second visual target (T2). Although such an audiovisual congruency effect has been attributed mainly to the semantic conflict carried by the incongruent sound restraining that sound from facilitating T2 processing, it is still unclear whether the integrated semantic information carried by the congruent sound benefits T2 processing.

Methods

To dissociate the congruence-induced benefit and incongruence-induced reduction in the alleviation of visual attentional blink at the behavioral and neural levels, the present study combined behavioral measures and event-related potential (ERP) recordings in a visual attentional blink task wherein the T2-accompanying sound, when delivered, could be semantically neutral in addition to congruent or incongruent with respect to T2.

Results

The behavioral data clearly showed that compared to the neutral sound, the congruent sound improved T2 discrimination during the blink to a higher degree while the incongruent sound improved it to a lesser degree. The T2-locked ERP data revealed that the early occipital cross-modal N195 component (192-228 ms after T2 onset) was uniquely larger in the congruent-sound condition than in the neutral-sound and incongruent-sound conditions, whereas the late parietal cross-modal N440 component (400-500 ms) was prominent only in the incongruent-sound condition.

Discussion

These findings provide strong evidence that the modulating effect of audiovisual semantic congruency on the sound-induced alleviation of visual attentional blink contains not only a late incongruence-induced cost but also an early congruence-induced benefit, thereby demonstrating for the first time an unequivocal congruent-sound-induced benefit in alleviating the limitation of time-based visual attention.

Cross-modal enhancement of spatially unpredictable visual target discrimination during the attentional blink

The attentional blink can be substantially reduced by delivering a task-irrelevant sound synchronously with the second target (T2) embedded in a rapid serial visual presentation stream, which is further modulated by the semantic congruency between the sound and T2. The present study extended the cross-modal boost during attentional blink and the modulation of audiovisual semantic congruency in the spatial domain by showing that a spatially uninformative, semantically congruent (but not incongruent) sound could even improve the discrimination of spatially unpredictable T2 during attentional blink. T2-locked event-related potential (ERP) data yielded that the early cross-modal P195 difference component (184-234 ms) over the occipital scalp contralateral to the T2 location was larger preceding accurate than inaccurate discriminations of semantically congruent, but not incongruent, audiovisual T2s. Interestingly, the N2pc component (194-244 ms) associated with visual-spatial attentional allocation was enlarged for incongruent audiovisual T2s relative to congruent audiovisual and unisensory visual T2s only when they were accurately discriminated. These ERP findings suggest that the spatially extended cross-modal boost during attentional blink involves an early cross-modal interaction strengthening the perceptual processing of T2, without any sound-induced enhancement of visual-spatial attentional allocation toward T2. In contrast, the absence of an accuracy decrease in response to semantically incongruent audiovisual T2s may originate from the semantic mismatch capturing extra visual-spatial attentional resources toward T2.

Effect of Target Semantic Consistency in Different Sequence Positions and Processing Modes on T2 Recognition: Integration and Suppression Based on Cross-Modal Processing

In the rapid serial visual presentation (RSVP) paradigm, sound affects participants' recognition of targets. Although many studies have shown that sound improves cross-modal processing, researchers have not yet explored the effects of sound semantic information with respect to different locations and processing modalities after removing sound saliency. In this study, the RSVP paradigm was used to investigate the difference between attention under conditions of consistent and inconsistent semantics with the target (Experiment 1), as well as the difference between top-down (Experiment 2) and bottom-up processing (Experiment 3) for sounds with consistent semantics with target 2 (T2) at different sequence locations after removing sound saliency. The results showed that cross-modal processing significantly improved attentional blink (AB). The early or lagged appearance of sounds consistent with T2 did not affect participants' judgments in the exogenous attentional modality. However, visual target judgments were improved with endogenous attention. The sequential location of sounds consistent with T2 influenced the judgment of auditory and visual congruency. The results illustrate the effects of sound semantic information in different locations and processing modalities.

The interplay between audiovisual temporal synchrony and semantic congruency in the cross-modal boost of the visual target discrimination during the attentional blink

The visual attentional blink can be substantially reduced by delivering a task-irrelevant sound synchronously with the second visual target (T2), and this effect is further modulated by the semantic congruency between the sound and T2. However, whether the cross-modal benefit originates from audiovisual interactions or sound-induced alertness remains controversial, and whether the semantic congruency effect is contingent on audiovisual temporal synchrony needs further investigation. The current study investigated these questions by recording event-related potentials (ERPs) in a visual attentional blink task wherein a sound could either synchronize with T2, precede T2 by 200 ms, be delayed by 100 ms, or be absent, and could be either semantically congruent or incongruent with T2 when delivered. The behavioral data showed that both the cross-modal boost of T2 discrimination and the further semantic modulation were the largest when the sound synchronized with T2. In parallel, the ERP data yielded that both the early occipital cross-modal P195 component (192-228 ms after T2 onset) and late parietal cross-modal N440 component (424-448 ms) were prominent only when the sound synchronized with T2, with the former being elicited solely when the sound was further semantically congruent whereas the latter occurring only when that sound was incongruent. These findings demonstrate not only that the cross-modal boost of T2 discrimination during the attentional blink stems from early audiovisual interactions and the semantic congruency effect depends on audiovisual temporal synchrony, but also that the semantic modulation can unfold at the early stage of visual discrimination processing.

Modal-based attention modulates attentional blink

In the rapid serial visual presentation (RSVP) paradigm, response accuracy for the target decreases when it appears within a short time window (200~500 ms) after the previous target. This phenomenon is termed the attentional blink (AB). Although mechanisms of cross-modal processing that reduce the AB have been documented, researchers have not explored the differences across modal attentional conditions. In the present study, we used the RSVP paradigm to investigate the effect of auditory-driven visual target perceptual enhancement on the AB under modality-specific selective attention (Experiment 1) and bimodal-divided attention (Experiment 2). The results showed that cross-modal attentional enhancement was not moderated by stimulus salience. Moreover, the results also showed that accuracy was higher when the attended sound appeared simultaneously with the target. These results indicated that audiovisual enhancement reduced AB and that stronger attentional enhancement in the bimodal-divided attentional condition led to the disappearance of AB.

Attentional blink suppresses both stimulus-driven and representation-driven cross-modal spread of attention

Previous studies have shown that visual attention effect can spread to the task-irrelevant auditory modality automatically through either the stimulus-driven binding process or the representation-driven priming process. Using an attentional blink paradigm, the present study investigated whether the long-latency stimulus-driven and representation-driven cross-modal spread of attention would be inhibited or facilitated when the attentional resources operating at the post-perceptual stage of processing are inadequate, whereas ensuring all visual stimuli were spatially attended and the representations of visual target object categories were activated, which were previously thought to be the only endogenous prerequisites for triggering cross-modal spread of attention. The results demonstrated that both types of attentional spreading were completely suppressed during the attentional blink interval but were highly prominent outside the attentional blink interval, with the stimulus-driven process being independent of, whereas the representation-driven process being dependent on, audiovisual semantic congruency. These findings provide the first evidence that the occurrences of both stimulus-driven and representation-driven spread of attention are contingent on the amount of post-perceptual attentional resources responsible for the late consolidation processing of visual stimuli, whereas the early detection of visual stimuli and the top-down activation of the visual representations are not the sole endogenous prerequisites for triggering any types of cross-modal attentional spreading.

Neural Basis of Semantically Dependent and Independent Cross-Modal Boosts on the Attentional Blink

The present study recorded event-related potentials (ERPs) in a visual object-recognition task under the attentional blink paradigm to explore the temporal dynamics of the cross-modal boost on attentional blink and whether this auditory benefit would be modulated by semantic congruency between T2 and the simultaneous sound. Behaviorally, the present study showed that not only a semantically congruent but also a semantically incongruent sound improved T2 discrimination during the attentional blink interval, whereas the enhancement was larger for the congruent sound. The ERP results revealed that the behavioral improvements induced by both the semantically congruent and incongruent sounds were closely associated with an early cross-modal interaction on the occipital N195 (192–228 ms). In contrast, the lower T2 accuracy for the incongruent than congruent condition was accompanied by a larger late occurring cento-parietal N440 (424–448 ms). These findings suggest that the cross-modal boost on attentional blink is hierarchical: the task-irrelevant but simultaneous sound, irrespective of its semantic relevance, firstly enables T2 to escape the attentional blink via cross-modally strengthening the early stage of visual object-recognition processing, whereas the semantic conflict of the sound begins to interfere with visual awareness only at a later stage when the representation of visual object is extracted.

No Evidence for an Awareness-Dependent Emotional Modulation of the Attentional Blink

Pictures of faces with emotional expressions presented before a temporal attention task have been reported to affect temporal attention in an awareness-dependent manner: Awareness of a fearful face was linked to an increased deficit in the temporal attention task, while preventing the face from reaching awareness was linked to a decreased deficit, both relative to neutral faces. Here we report the results of two temporal attention experiments which aimed to extend and conceptually replicate this basic finding. The temporal attention task was preceded by an unmasked or a masked fearful face on a trial-by-trial basis. In both experiments the finding of an awareness-dependent emotional modulation of temporal attention through fearful faces could not be replicated, even when data were pooled across experiments. Pooling of experiments indicated however that, independent of awareness level, fearful faces can be associated with slightly worse temporal attention performance than neutral faces, and suggested a lag-specific practice effect in terms of a reduced deficit in temporal attention in the second half of the experiment.

Ictal EEG source imaging in presurgical evaluation: High agreement between analysis methods

•

There was good agreement between different methods of ictal EEG source imaging.

•

Ictal source imaging achieved an accuracy of 73% (for operated patients: 86%).

•

Agreement between all methods did not necessarily imply accuracy of localization.

Purpose

To determine the agreement between five different methods of ictal EEG source imaging, and to assess their accuracy in presurgical evaluation of patients with focal epilepsy. It was hypothesized that high agreement between methods was associated with higher localization-accuracy.

Methods

EEGs were recorded with a 64-electrode array. Thirty-eight seizures from 22 patients were analyzed using five different methods phase mapping, dipole fitting, CLARA, cortical-CLARA and minimum norm. Localization accuracy was determined at sub-lobar level. Reference standard was the final decision of the multidisciplinary epilepsy surgery team, and, for the operated patients, outcome one year after surgery.

Results

Agreement between all methods was obtained in 13 patients (59%) and between all but one methods in additional six patients (27%). There was a trend for minimum norm being less accurate than phase mapping, but none of the comparisons reached significance. Source imaging in cases with agreement between all methods was not more accurate than in the other cases. Ictal source imaging achieved an accuracy of 73% (for operated patients: 86%).

Conclusion

There was good agreement between different methods of ictal source imaging. However, good inter-method agreement did not necessarily imply accurate source localization, since all methods faced the limitations of the inverse solution.