How modality processing differences affect cross-modal nonspatial repetition inhibition

Semantically congruent bimodal presentation modulates cognitive control over attentional guidance by working memory

Although previous studies have well established that audiovisual enhancement has a promoting effect on working memory and selective attention, there remains an open question about the influence of audiovisual enhancement on attentional guidance by working memory. To address this issue, the present study adopted a dual-task paradigm that combines a working memory task and a visual search task, in which the content of working memory was presented in audiovisual or visual modalities. Given the importance of search speed in memory-driven attentional suppression, we divided participants into two groups based on their reaction time (RT) in neutral trials and examined whether audiovisual enhancement in attentional suppression was modulated by search speed. The results showed that the slow search group exhibited a robust memory-driven attentional suppression effect, and the suppression effect started earlier and its magnitude was greater in the audiovisual condition than in the visual-only condition. However, among the faster search group, the suppression effect only occurred in the trials with longer RTs in the visual-only condition, and its temporal dynamics were selectively improved in the audiovisual condition. Furthermore, audiovisual enhancement of memory-driven attention evolved over time. These findings suggest that semantically congruent bimodal presentation can progressively facilitate the strength and temporal dynamics of memory-driven attentional suppression, and that search speed plays an important role in this process. This may be due to a synergistic effect between multisensory working memory representation and top-down suppression mechanism. The present study demonstrates the flexible role of audiovisual enhancement on cognitive control over memory-driven attention.

Non-spatial inhibition of return attenuates audiovisual integration owing to modality disparities

Although previous studies have investigated the relationship between inhibition of return (IOR) and multisensory integration, the influence of non-spatial has not been explored. The present study aimed to investigate the influence of non-spatial IOR on audiovisual integration by using a "prime-neutral cue-target" paradigm. In Experiment 1, which manipulated prime validity and target modality, the targets were positioned centrally, revealing significant non-spatial IOR effects in the visual, auditory, and audiovisual modalities. Analysis of relative multisensory response enhancement (rMRE) indicated substantial audiovisual integration enhancement in both valid and invalid target conditions. Furthermore, the enhancement was weaker for valid targets than for invalid targets. In Experiment 2, the targets were positioned above and below to rule out repetition blindness (RB); this experiment successfully replicated the results observed in Experiment 1. Notably, Experiments 1 and 2 consistently found that the correlation between modality differences and rMRE for valid targets indicated that differences in signal strength between visual and auditory modalities contributed to a reduction in audiovisual integration. However, the absence of correlation with the invalid target suggests that attention, as a key factor, may play a significant role in this process. The present study highlights how non-spatial IOR reduces audiovisual integration and sheds light on the complex interaction between attention and multisensory integration.

Detection cost: A nonnegligible factor contributing to inhibition of return in the discrimination task under the cue-target paradigm

The three-factor model argues that the spatial orienting benefit triggered by the cue, the spatial selection benefit of cue-target matching, and the detection cost of distinguishing the cue from the target contribute to the measured inhibition of return (IOR) effect. According to the three-factor model, the spatial selection benefit dominates the occurrence of the IOR effect in the discrimination task, while the detection cost is negligible. The present study verified the three-factor model in the discrimination task under the cue-target paradigm by manipulating the spatial location and nonspatial feature consistency of the cue and the target as well as the promotion or hindrance of attentional disengagement from the cued location with a central reorienting cue. The results indicated that the three factors of the three-factor model contributed to the measured IOR effect in the discrimination task. Interestingly, the IOR effect was stable when the cue and target were perfectly repeated and attention was maintained at the cued location, implying that detection cost was not a negligible factor. The current study supported the contribution of all three factors in the three-factor model to the measured IOR effect; however, we argue that the role of detection cost in the discrimination task under different paradigms should be further refined.

Rewards weaken cross-modal inhibition of return with visual targets

Previous studies have shown that rewards weaken visual inhibition of return (IOR). However, the specific mechanisms underlying the influence of rewards on cross-modal IOR remain unclear. Based on the Posner exogenous cue-target paradigm, the present study was conducted to investigate the effect of rewards on exogenous spatial cross-modal IOR in both visual cue with auditory target (VA) and auditory cue with visual target (AV) conditions. The results showed the following: in the AV condition, the IOR effect size in the high-reward condition was significantly lower than that in the low-reward condition. However, in the VA condition, there was no significant IOR in either the high- or low-reward condition and there was no significant difference between the two conditions. In other words, the use of rewards modulated exogenous spatial cross-modal IOR with visual targets; specifically, high rewards may have weakened IOR in the AV condition. Taken together, our study extended the effect of rewards on IOR to cross-modal attention conditions and demonstrated for the first time that higher motivation among individuals under high-reward conditions weakened the cross-modal IOR with visual targets. Moreover, the present study provided evidence for future research on the relationship between reward and attention.

Does the construction retrieval account apply to cross‐modal inhibition of return in semantic context?

The traditional attentional reorienting hypothesis is insufficient to explain spatial and nonspatial inhibition of return (IOR). Therefore, a construction retrieval account that includes the influence of top-down attentional sets has been proposed and can explain both spatial and nonspatial IOR. However, it remains unknown whether the construction retrieval account can be applied to non-surface features of stimuli, as well as whether its construction and retrieval mechanisms are supra-modal. The present study manipulated semantic feature congruency and spatial location congruency between the prime and the target in cross-modal audio-visual and visual-audio experimental conditions, respectively, by orthogonally combining spatial and nonspatial IOR paradigms. Our results showed that there was an interaction between semantic feature congruency and spatial location congruency controlled by the attentional sets, and that this interaction was consistent in cross-modal audio-visual and visual-audio conditions. These results suggest that the construction retrieval account can be applied to abstract semantic features and that its construction and retrieval mechanisms are supra-modal. The present study extends the application scope of the construction retrieval account and promotes the interpretation of IOR under a unified theoretical framework.

Levels of neuroticism can predict attentional performance during cross-modal nonspatial repetition inhibition

Inhibition of return (IOR) refers to the slower response to targets presented at previously attended locations, and such repetition-induced inhibition has been found to be differentially associated with personality traits. Although it has been well documented how personality traits affect spatial IOR, a mechanism associated with the attentional orienting network, there is not yet a consensus as to the relationship between personality traits and nonspatial repetition inhibition, a mechanism associated with the attentional executive network. The present study herein examined how the Big Five personality traits relate to cross-modal nonspatial repetition inhibition. Participants completed the NEO-PI-R and performed a cross-modal nonspatial repetition inhibition task built on the prime-neutral cue-target paradigm, in which the relationships of the identities and modalities between the prime and the target were manipulated. The results showed a significant nonspatial inhibitory effect and the effect was larger under the visual-auditory condition than under the auditory-visual condition. More importantly, neuroticism was associated with decreased cross-modal nonspatial inhibitory effect, presumably due to impaired attentional control. However, such a result was only found in the visual-auditory condition. We propose that retrieving previous prime representations under the visual-auditory condition requires a large consumption of cognitive resources, making inhibitory control more difficult for individuals with high neuroticism. These findings provide new insight into the influence of personality traits on attentional performance requiring nonspatial inhibitory control and enrich the relationship between neuroticism and repetition-induced inhibition.

Modal-based attention modulates attentional blink

In the rapid serial visual presentation (RSVP) paradigm, response accuracy for the target decreases when it appears within a short time window (200~500 ms) after the previous target. This phenomenon is termed the attentional blink (AB). Although mechanisms of cross-modal processing that reduce the AB have been documented, researchers have not explored the differences across modal attentional conditions. In the present study, we used the RSVP paradigm to investigate the effect of auditory-driven visual target perceptual enhancement on the AB under modality-specific selective attention (Experiment 1) and bimodal-divided attention (Experiment 2). The results showed that cross-modal attentional enhancement was not moderated by stimulus salience. Moreover, the results also showed that accuracy was higher when the attended sound appeared simultaneously with the target. These results indicated that audiovisual enhancement reduced AB and that stronger attentional enhancement in the bimodal-divided attentional condition led to the disappearance of AB.

Endogenous Spatial Attention Modulates the Magnitude of the Colavita Visual Dominance Effect

The Colavita effect refers to the phenomenon wherein people tend to not respond to an auditory stimulus when a visual stimulus is simultaneously presented. Although previous studies have shown that endogenous modality attention influences the Colavita effect, whether the Colavita effect is influenced by endogenous spatial attention remains unknown. In the present study, we established endogenous spatial cues to investigate whether the size of the Colavita effect changes under visual or auditory cues. We measured three indexes to investigate the effect of endogenous spatial attention on the size of the Colavita effect. These three indexes were developed based on the following observations in bimodal trials: (a) The proportion of the "only vision" response was significantly higher than that of the "only audition" response; (b) the proportion of the "vision precedes audition" response was significantly higher than that of the "audition precedes vision" response; and (c) the reaction time difference of the "vision precedes audition" response was significantly higher than that of the "audition precedes vision" response. Our results showed that the Colavita effect was always influenced by endogenous spatial attention and that its size was larger at the cued location than at the uncued location; the cue modality (visual vs. auditory) had no effect on the size of the Colavita effect. Taken together, the present results shed light on how endogenous spatial attention affects the Colavita effect.

Repetition Suppression in Visual and Auditory Modalities Affects the Sound-Induced Flash Illusion

Sound-induced flash illusion (SiFI) refers to the illusion that the number of visual flashes is equal to the number of auditory sounds when the visual flashes are accompanied by an unequal number of auditory sounds presented within 100 ms. The effect of repetition suppression (RS), an adaptive effect caused by stimulus repetition, upon the SiFI has not been investigated. Based on the classic SiFI paradigm, the present study investigated whether RS would affect the SiFI differently by adding preceding stimuli in visual and auditory modalities prior to the appearance of audiovisual stimuli. The results showed the auditory RS effect on the SiFI varied with the number of preceding auditory stimuli. The hit rate was higher with two preceding auditory stimuli than one preceding auditory stimulus in fission illusion, but it did not affect the size of the fusion illusion. However, the visual RS had no effect on the size of the fission and fusion illusions. The present study suggested that RS could affect the SiFI, indicating that the RS effect in different modalities would differentially affect the magnitude of the SiFI. In the process of multisensory integration, the visual and auditory modalities had asymmetrical RS effects.

Crossmodal Nonspatial Repetition Inhibition Due to Modality Shift

Previous studies have found that processing of a second stimulus is slower when the modality of the first stimulus differs, which is termed the modality shift effect. Moreover, people tend to respond more slowly to the second stimulus when the two stimuli are similar in the semantic dimension, which is termed the nonspatial repetition inhibition effect. This study aimed to explore the modality shift effect on nonspatial repetition inhibition and whether such modulation was influenced by different temporal intervals. A cue-target paradigm was adopted in which modality priming and identity priming were manipulated at three interstimuli intervals. The results showed that the response times under the modality shift condition were slower than those under the modality repeat condition. In trials with modality shift, responses to congruent cues and targets were slower than to incongruent cue-target combinations, indicating crossmodal nonspatial repetition inhibition. The crossmodal nonspatial repetition inhibition effect decreased with increasing interstimuli interval. These results provide evidence that the additional intervening event proposed in previous studies is not necessary for the occurrence of crossmodal nonspatial repetition inhibition.

Visual aperiodic temporal prediction increases perceptual sensitivity and reduces response latencies

As a predictive organ, the brain can predict upcoming events to guide perception and action in the process of adaptive behavior. The classical models of oscillatory entrainment explain the facilitating effects that occur after periodic stimulation in behavior but cannot explain aperiodic facilitating effects. In the present study, by comparing the behavior performance of participants in periodic predictable (PP), aperiodic predictable (AP) and aperiodic unpredictable (AU) stimulus streams, we investigated the effect of an aperiodic predictable stream on the perceptual sensitivity and response latencies in the visual modality. The results showed that there was no difference between PP and AP conditions in sensitivity (d') and reaction times (RTs), both of which were significantly different from those in the AU condition. Moreover, a significant correlation between d' and RTs was observed when predictability existed. These results indicate that the aperiodic predictable stimulus streams increases perceptual sensitivity and reduces response latencies in a top-down manner. Individuals proactively and flexibly predict upcoming events based on the temporal structure of visual stimuli in the service of adaptive behavior.

Cross-modal nonspatial repetition inhibition: An ERP study

Previous studies have demonstrated that nonspatial repetition inhibition can occur across modalities. However, the underlying mechanism of such cross-modal nonspatial repetition inhibition is unknown. The present experiment adopted a cross-modal prime-neutral cue-target paradigm in which in consecutive trials the prime and the target were matched or mismatched, not only in identity but also in modality. Meanwhile, event-related potentials (ERPs) to visual and auditory targets were recorded. The present study aimed to answer two questions: which ERP components reflect nonspatial repetition inhibition across modalities, and is the ERP component modality specific or supramodal? The results showed that for visual targets, robust nonspatial repetition inhibition occurred similarly for both unimodal (visual-visual) and cross-modal (audio-visual) target pairings, as indexed by an N400 repetition-induced increment in the typical N400 window but null effects during the N2 epoch. For auditory targets, similar modulation of cross-modal nonspatial repetition inhibition on the auditory-evoked N400 repetition-induced increment was observed. These results suggest that the N400 repetition-induced increment occurs during the N400 epoch that underlies cross-modal nonspatial repetition inhibition and that this N400 component is a supramodal component.