Auditory and audiovisual inhibition of return

Rewards weaken cross-modal inhibition of return with visual targets

Previous studies have shown that rewards weaken visual inhibition of return (IOR). However, the specific mechanisms underlying the influence of rewards on cross-modal IOR remain unclear. Based on the Posner exogenous cue-target paradigm, the present study was conducted to investigate the effect of rewards on exogenous spatial cross-modal IOR in both visual cue with auditory target (VA) and auditory cue with visual target (AV) conditions. The results showed the following: in the AV condition, the IOR effect size in the high-reward condition was significantly lower than that in the low-reward condition. However, in the VA condition, there was no significant IOR in either the high- or low-reward condition and there was no significant difference between the two conditions. In other words, the use of rewards modulated exogenous spatial cross-modal IOR with visual targets; specifically, high rewards may have weakened IOR in the AV condition. Taken together, our study extended the effect of rewards on IOR to cross-modal attention conditions and demonstrated for the first time that higher motivation among individuals under high-reward conditions weakened the cross-modal IOR with visual targets. Moreover, the present study provided evidence for future research on the relationship between reward and attention.

Same, but different: Binding effects in auditory, but not visual detection performance

Responding to a stimulus leads to the integration of response and stimulus' features into an event file. Upon repetition of any of its features, the previous event file is retrieved, thereby affecting ongoing performance. Such integration-retrieval explanations exist for a number of sequential tasks (that measure these processes as 'binding effects') and are thought to underlie all actions. However, based on attentional orienting literature, Schöpper, Hilchey, et al. (2020) could show that binding effects are absent when participants detect visual targets in a sequence: In visual detection performance, there is simply a benefit for target location changes (inhibition of return). In contrast, Mondor and Leboe (2008) had participants detect auditory targets in a sequence, and found a benefit for frequency repetition - presumably reflecting a binding effect in auditory detection performance. In the current study, we conducted two experiments, that only differed in the modality of the target: Participants signaled the detection of a sound (N = 40) or of a visual target (N = 40). Whereas visual detection performance showed a pattern incongruent with binding assumptions, auditory detection performance revealed a non-spatial feature repetition benefit, suggesting that frequency was bound to the response. Cumulative reaction time distributions indicated that the absence of a binding effect in visual detection performance was not caused by overall faster responding. The current results show a clear limitation to binding accounts in action control: Binding effects are not only limited by task demands, but can entirely depend on target modality.

Does the construction retrieval account apply to cross‐modal inhibition of return in semantic context?

The traditional attentional reorienting hypothesis is insufficient to explain spatial and nonspatial inhibition of return (IOR). Therefore, a construction retrieval account that includes the influence of top-down attentional sets has been proposed and can explain both spatial and nonspatial IOR. However, it remains unknown whether the construction retrieval account can be applied to non-surface features of stimuli, as well as whether its construction and retrieval mechanisms are supra-modal. The present study manipulated semantic feature congruency and spatial location congruency between the prime and the target in cross-modal audio-visual and visual-audio experimental conditions, respectively, by orthogonally combining spatial and nonspatial IOR paradigms. Our results showed that there was an interaction between semantic feature congruency and spatial location congruency controlled by the attentional sets, and that this interaction was consistent in cross-modal audio-visual and visual-audio conditions. These results suggest that the construction retrieval account can be applied to abstract semantic features and that its construction and retrieval mechanisms are supra-modal. The present study extends the application scope of the construction retrieval account and promotes the interpretation of IOR under a unified theoretical framework.

When does “inhibition of return” occur in spatial cueing tasks? Temporally disentangling multiple cue-triggered effects using response history and conditional accuracy analyses

Research on spatial cueing has shown that uninformative cues often facilitate mean response time (RT) performance in valid- compared to invalid-cueing conditions at short cue-target stimulus-onset-asynchronies (SOAs), and robustly generate a reversed or inhibitory cueing effect at longer SOAs that is widely known as inhibition-of-return (IOR). To study the within-trial time course of the IOR and facilitation effects we employ discrete-time hazard and conditional accuracy analyses to analyze the shapes of the RT and accuracy distributions measured in two experimental tasks. Our distributional analyses show that (a) IOR is present only from ~160 ms to ~280 ms after target onset for cue-target SOAs above ~200 ms, (b) facilitation does not precede IOR, but co-occurs with it, (c) the cue-triggered motor response activation is selectively and actively inhibited before target onset, (d) the presence of a central cue causes a temporary negative cueing effect in the conditional accuracy functions, (e) the IOR effect consists of a facilitatory and an inhibitory component when compared to central cueing, and (f) the within-trial time course of IOR is not affected much by the task employed (detection or localization). We conclude that the traditional mean performance measures conceal crucial information on behavioral dynamics in spatial cueing paradigms.

Global and local priming in a multi-modal context

Perceptual information can be processed at many different scales, from featural details to entire scenes. Attentional selection of different scales has been studied using hierarchical stimuli, with research elucidating a variety of biases in local and global attentional selection (due to, e.g., stimulus properties, brain injury, and experience). In this study, the emphasis is on biases produced through recent experience, or level-specific priming effects, which have been demonstrated within both the visual and auditory modalities. Namely, when individuals attend to local information, they are subsequently biased to attend locally (and similarly so with global attention). Here, these level-specific priming effects are investigated in a multi-modal context to determine whether cross-modal interactions occur between visual and auditory modalities during hierarchical processing. Specifically, the study addresses if attentional selection of local or global information in the visual modality subsequently biases auditory attentional selection to that level, and vice versa (i.e., level-priming). Though expected identity priming effects emerged in the study, no cross-modal level-priming effects manifested. Furthermore, the multi-modal context eliminated the well-established within-modality level-specific priming effects. Thus, though the study does reveal a multi-modal effect, it was not a level-based effect. Instead, paradoxically, the multi-modal context eliminated attentional scope biases (i.e., level-priming) within uni-modal transitions. In other words, when visual and auditory information are equally likely require attention, no persistence emerges for processing local or global information over time, even within a single modality.

Effects of Audiovisual Presentations on Visual Localization Errors: One or Several Multisensory Mechanisms?

The present study examines the extent to which temporal and spatial properties of sound modulate visual motion processing in spatial localization tasks. Participants were asked to locate the place at which a moving visual target unexpectedly vanished. Across different tasks, accompanying sounds were factorially varied within subjects as to their onset and offset times and/or positions relative to visual motion. Sound onset had no effect on the localization error. Sound offset was shown to modulate the perceived visual offset location, both for temporal and spatial disparities. This modulation did not conform to attraction toward the timing or location of the sounds but, demonstrably in the case of temporal disparities, to bimodal enhancement instead. Favorable indications to a contextual effect of audiovisual presentations on interspersed visual-only trials were also found. The short sound-leading offset asynchrony had equivalent benefits to audiovisual offset synchrony, suggestive of the involvement of early-level mechanisms, constrained by a temporal window, at these conditions. Yet, we tentatively hypothesize that the whole of the results and how they compare with previous studies requires the contribution of additional mechanisms, including learning-detection of auditory-visual associations and cross-sensory spread of endogenous attention.

Electrophysiological evidence of different neural processing between visual and audiovisual inhibition of return

Inhibition of return (IOR) refers to the slower response to targets appearing on the same side as the cue (valid locations) than to targets appearing on the opposite side as the cue (invalid locations). Previous behaviour studies have found that the visual IOR is larger than the audiovisual IOR when focusing on both visual and auditory modalities. Utilising the high temporal resolution of the event-related potential (ERP) technique we explored the possible neural correlates with the behaviour IOR difference between visual and audiovisual targets. The behavioural results revealed that the visual IOR was larger than the audiovisual IOR. The ERP results showed that the visual IOR effect was generated from the P1 and N2 components, while the audiovisual IOR effect was derived only from the P3 component. Multisensory integration (MSI) of audiovisual targets occurred on the P1, N1 and P3 components, which may offset the reduced perceptual processing due to audiovisual IOR. The results of early and late differences in the neural processing of the visual IOR and audiovisual IOR imply that the two target types may have different inhibitory orientation mechanisms.

Crossmodal spatial distraction across the lifespan

The ability to resist distracting stimuli whilst voluntarily focusing on a task is fundamental to our everyday cognitive functioning. Here, we investigated how this ability develops, and thereafter declines, across the lifespan using a single task/experiment. Young children (5-7 years), older children (10-11 years), young adults (20-27 years), and older adults (62-86 years) were presented with complex visual scenes. Endogenous (voluntary) attention was engaged by having the participants search for a visual target presented on either the left or right side of the display. The onset of the visual scenes was preceded - at stimulus onset asynchronies (SOAs) of 50, 200, or 500 ms - by a task-irrelevant sound (an exogenous crossmodal spatial distractor) delivered either on the same or opposite side as the visual target, or simultaneously on both sides (cued, uncued, or neutral trials, respectively). Age-related differences were revealed, especially in the extreme age-groups, which showed a greater impact of crossmodal spatial distractors. Young children were highly susceptible to exogenous spatial distraction at the shortest SOA (50 ms), whereas older adults were distracted at all SOAs, showing significant exogenous capture effects during the visual search task. By contrast, older children and young adults' search performance was not significantly affected by crossmodal spatial distraction. Overall, these findings present a detailed picture of the developmental trajectory of endogenous resistance to crossmodal spatial distraction from childhood to old age and demonstrate a different efficiency in coping with distraction across the four age-groups studied.

A Biologically Motivated, Proto-Object-Based Audiovisual Saliency Model

The natural environment and our interaction with it are essentially multisensory, where we may deploy visual, tactile and/or auditory senses to perceive, learn and interact with our environment. Our objective in this study is to develop a scene analysis algorithm using multisensory information, specifically vision and audio. We develop a proto-object-based audiovisual saliency map (AVSM) for the analysis of dynamic natural scenes. A specialized audiovisual camera with 360∘ field of view, capable of locating sound direction, is used to collect spatiotemporally aligned audiovisual data. We demonstrate that the performance of a proto-object-based audiovisual saliency map in detecting and localizing salient objects/events is in agreement with human judgment. In addition, the proto-object-based AVSM that we compute as a linear combination of visual and auditory feature conspicuity maps captures a higher number of valid salient events compared to unisensory saliency maps. Such an algorithm can be useful in surveillance, robotic navigation, video compression and related applications.

Cross-modal nonspatial repetition inhibition: An ERP study

Previous studies have demonstrated that nonspatial repetition inhibition can occur across modalities. However, the underlying mechanism of such cross-modal nonspatial repetition inhibition is unknown. The present experiment adopted a cross-modal prime-neutral cue-target paradigm in which in consecutive trials the prime and the target were matched or mismatched, not only in identity but also in modality. Meanwhile, event-related potentials (ERPs) to visual and auditory targets were recorded. The present study aimed to answer two questions: which ERP components reflect nonspatial repetition inhibition across modalities, and is the ERP component modality specific or supramodal? The results showed that for visual targets, robust nonspatial repetition inhibition occurred similarly for both unimodal (visual-visual) and cross-modal (audio-visual) target pairings, as indexed by an N400 repetition-induced increment in the typical N400 window but null effects during the N2 epoch. For auditory targets, similar modulation of cross-modal nonspatial repetition inhibition on the auditory-evoked N400 repetition-induced increment was observed. These results suggest that the N400 repetition-induced increment occurs during the N400 epoch that underlies cross-modal nonspatial repetition inhibition and that this N400 component is a supramodal component.

Basic auditory processing in the children with autistic features

This study aimed inhibition mechanisms of auditory processing in the group with autistic features. Thirty-two children (autistic group = 16, typically developing [TD] group = 16) received neuropsychological tests, IQ test and experimental tasks. Both groups showed similar performances except the processing speed index. The results showed that the group with autistic features had less inhibition of return (IOR) than the TD group. However, we did not get a statistically significant group difference in the auditory Go-NoGo task. These results might be attributed to a ceiling effect due to an adjustment failure of a difficulty level instead of showing that the group with autistic features would have intact inhibitory or pitch discriminative function problems. In conclusion, this study showed that the group with autistic features could have an inhibitory processing difficulty in both auditory and visual IOR tasks even when their general cognitive functions are relatively intact. This study presented a possibility that the group with autistic features might have a basic inhibitory function problem, but these findings should be investigated in the further study with enough samples. In addition, we are going to revise the auditory Go-NoGo task and verify the feasibility as a tool to detect ASD in an early stage in the following study.

Different visual and auditory latencies affect cross-modal non-spatial repetition inhibition

The present study examined the effect of different latencies for processing visual and auditory stimuli in cross-modal non-spatial repetition inhibition. In two experiments, the cue validity of modality and identity between the prime and the target was manipulated in a "prime-neutral cue-target" paradigm. A distinct neutral event was presented after the prime and before the onset of the target. The prime probe was visual in Experiment 1 and auditory in Experiment 2. The results in both experiments showed that RTs for identity-cued trials were significantly slower than RTs for identity-cued trials regardless of whether the modality of the target was visual or auditory. In addition, RTs for visual trials were significantly faster than RTs for auditory trials, indicating different latencies of processing visual and auditory stimuli. This latency difference affects cross-modal non-spatial repetition inhibition in two aspects: 1) creating a new representation (identity uncued) that is delivered via visual modality is easier under audio-visual conditions, and 2) retrieving an inhibited representation (identity cued) that is delivered via auditory modality is more difficult under visual-audio conditions. We propose that cross-modal non-spatial repetition inhibition, which is distinct from unimodal repetition inhibition, can be easily influenced by different latencies of processing visual and auditory stimuli.

Inhibition of return modulates the flash-lag effect

Transient events are known to draw exogenous attention, and visual processing at the attended location is transiently facilitated, but after several hundred milliseconds, attentional processing at the cued location becomes poorer than processing elsewhere, resulting in a slower reaction to a target stimulus that subsequently appears at the cued location. Despite a number of previous studies on this effect, termed inhibition of return (IOR), it is still unclear whether a perceptual process related to the subjective onset time of the target stimulus is disrupted when IOR occurs. In the present study, we used a distinct visual phenomenon termed the flash-lag effect (FLE) as a tool to quantify IOR. The FLE is an illusion in which a flashed stimulus appears to lag behind a moving stimulus, despite being physically aligned. We used an identical stimulus configuration and asked observers to conduct two independent tasks in separate sessions. The first was a simple reaction task to measure the onset reaction time (RT) to an abruptly appearing target. The second was an orientation judgment task to measure the degree of the FLE. Both the RT and the FLE were found to be altered in accordance with IOR, and a significant correlation was demonstrated between the changes in the RT and those in the FLE. These results demonstrate that the perceptual process related to the stimulus onset can be compromised by IOR.

ERP-study on the time course of disgust-motivated spatial avoidance

Recently, we showed that disgusting sound cues direct spatial attention away from the location of their origin to the opposite location indicating spatial disgust avoidance (Zimmer et al., 2015, Psychophysiology; 2016, Neuroimage). However, in these studies, we had solely used an inter-stimulus interval (ISI) of 200-300 ms, leaving unclear how disgust avoidance develops over time. Studies have shown that spatial attraction due to anger persists longer when induced by auditory rather than visual stimuli. In the present ERP-study, one of two laterally presented sounds (neutral/disgust) cued a laterally presented visual target. ISIs varied from short (50-150 ms) over middle (350-450 ms) to long (650-750 ms). For disgust stimuli, response times were longer for invalidly cued targets compared to validly cued targets, reflecting disgust avoidance. There was an increased P3 amplitude for validly versus invalidly cued targets consistent with disgust avoidance. In contrast, in the neutral condition, we found evidence of inhibition of return (IOR), as we observed a reversal of the usual validity effect from short to long ISI in the behavioral data and on the P1-component. These results indicate that spatial avoidance motivated by auditory disgust persists over time, presumably enforced by emotional rather than general attentional processes.

Bimodal-divided attention attenuates visually induced inhibition of return with audiovisual targets

Inhibition of return (IOR) refers to the slower response to a target appearing at a previously attended location in a cue-target paradigm. It has been greatly explored in the visual or auditory modality. This study investigates differences between the IOR of audiovisual targets and the IOR of visual targets under conditions of modality-specific selective attention (Experiment 1) and divided-modalities attention (Experiment 2). We employed an exogenous spatial cueing paradigm and manipulated the modalities of targets, including visual, auditory, or audiovisual modalities. The participants were asked to detect targets in visual modality or both visual and auditory modalities, which were presented on the same (cued) or opposite (uncued) side as the preceding visual peripheral cues. In Experiment 1, we found the comparable IOR with visual and audiovisual targets when participants were asked to selectively focus on visual modality. In Experiment 2, however, there was a smaller magnitude of IOR with audiovisual targets as compared with visual targets when paying attention to both visual and auditory modalities. We also observed a reduced multisensory response enhancement effect and race model inequality violation at cued locations relative to uncued locations. These results provide the first evidence of the IOR with audiovisual targets. Furthermore, IOR with audiovisual targets decreases when paying attention to both modalities. The interaction between exogenous spatial attention and audiovisual integration is discussed.

Cueing listeners to attend to a target talker progressively improves word report as the duration of the cue-target interval lengthens to 2,000 ms

Endogenous attention is typically studied by presenting instructive cues in advance of a target stimulus array. For endogenous visual attention, task performance improves as the duration of the cue-target interval increases up to 800 ms. Less is known about how endogenous auditory attention unfolds over time or the mechanisms by which an instructive cue presented in advance of an auditory array improves performance. The current experiment used five cue-target intervals (0, 250, 500, 1,000, and 2,000 ms) to compare four hypotheses for how preparatory attention develops over time in a multi-talker listening task. Young adults were cued to attend to a target talker who spoke in a mixture of three talkers. Visual cues indicated the target talker's spatial location or their gender. Participants directed attention to location and gender simultaneously ("objects") at all cue-target intervals. Participants were consistently faster and more accurate at reporting words spoken by the target talker when the cue-target interval was 2,000 ms than 0 ms. In addition, the latency of correct responses progressively shortened as the duration of the cue-target interval increased from 0 to 2,000 ms. These findings suggest that the mechanisms involved in preparatory auditory attention develop gradually over time, taking at least 2,000 ms to reach optimal configuration, yet providing cumulative improvements in speech intelligibility as the duration of the cue-target interval increases from 0 to 2,000 ms. These results demonstrate an improvement in performance for cue-target intervals longer than those that have been reported previously in the visual or auditory modalities.

Electrophysiological evidence of an attentional bias in crossmodal inhibition of return

Inhibition of return (IOR) refers to a delay in responding to targets when they appear at recently attended locations, relative to unattended locations. Within the visual modality, this attentional bias has been associated with a reduction in the N2pc event-related potential (ERP) component at previously attended locations. The present study examined whether a similar attentional bias was observed in crossmodal audio-visual IOR. Our results demonstrate that for visual targets, the attentional component of IOR behaves similarly for both unimodal and crossmodal target pairs, as indexed by a reduction in the N2pc component for targets appearing at previously attended locations. Further, similar IOR-related modulations on the auditory-evoked N2ac indicated that an attentional bias can be observed for auditory targets as well. Finally, we identified two additional ERP components - the ACOP and VCAN - that appear to reflect biasing of attention in the currently unattended sensory modality. These results suggest that the inhibitory attentional bias that underlies the IOR effect may be supramodal and bias attention away from previously attended locations regardless of sensory modality.

Cross-modal cueing effects of visuospatial attention on conscious somatosensory perception

OBJECTIVE

The impact of visuospatial attention on perception with supraliminal stimuli and stimuli at the threshold of conscious perception has been previously investigated. In this study, we assess the cross-modal effects of visuospatial attention on conscious perception for near-threshold somatosensory stimuli applied to the face.

METHODS

Fifteen healthy participants completed two sessions of a near-threshold cross-modality cue-target discrimination/conscious detection paradigm. Each trial began with an endogenous visuospatial cue that predicted the location of a weak near-threshold electrical pulse delivered to the right or left cheek with high probability (∼75%). Participants then completed two tasks: first, a forced-choice somatosensory discrimination task (felt once or twice?) and then, a somatosensory conscious detection task (did you feel the stimulus and, if yes, where (left/right)?). Somatosensory discrimination was evaluated with the response reaction times of correctly detected targets, whereas the somatosensory conscious detection was quantified using perceptual sensitivity (d') and response bias (beta). A 2 × 2 repeated measures ANOVA was used for statistical analysis.

RESULTS

In the somatosensory discrimination task (1st task), participants were significantly faster in responding to correctly detected targets (p < 0.001). In the somatosensory conscious detection task (2nd task), a significant effect of visuospatial attention on response bias (p = 0.008) was observed, suggesting that participants had a less strict criterion for stimuli preceded by spatially valid than invalid visuospatial cues.

CONCLUSIONS

We showed that spatial attention has the potential to modulate the discrimination and the conscious detection of near-threshold somatosensory stimuli as measured, respectively, by a reduction of reaction times and a shift in response bias toward less conservative responses when the cue predicted stimulus location. A shift in response bias indicates possible effects of spatial attention on internal decision processes. The lack of significant results in perceptual sensitivity (d') could be due to weaker effects of endogenous attention on perception.

From Behavioral Facilitation to Inhibition: The Neuronal Correlates of the Orienting and Reorienting of Auditory Attention

Successful adaptive behavior relies on the ability to automatically (bottom-up) orient attention to different locations in the environment. This results in a biphasic pattern in which reaction times (RT) are faster for stimuli that occur in the same spatial location (valid) for the first few hundred milliseconds, which is termed facilitation. This is followed by faster RT for stimuli that appear in novel locations (invalid) after longer delays, termed inhibition of return. The neuronal areas and networks involved in the transition between states of facilitation and inhibition remain poorly understood, especially for auditory stimuli. Functional magnetic resonance imaging (fMRI) data were therefore collected in a large sample of healthy volunteers (N = 52) at four separate auditory stimulus onset asynchronies (SOAs; 200, 400, 600, and 800 ms). Behavioral results indicated that facilitation (valid RT < invalid RT) occurred at the 200 ms SOA, with inhibition of return (valid RT > invalid RT) present at the three longer SOAs. fMRI results showed several brain areas varying their activation as a function of SOA, including bilateral superior temporal gyrus, anterior thalamus, cuneus, dorsal anterior cingulate gyrus, and right ventrolateral prefrontal cortex (VLPFC)/anterior insula. Right VLPFC was active during a behavioral state of facilitation, and its activation (invalid – valid trials) further correlated with behavioral reorienting at the 200 ms delay. These results suggest that right VLPFC plays a critical role when auditory attention must be quickly deployed or redeployed, demanding heightened cognitive and inhibitory control. In contrast to previous work, the ventral and dorsal frontoparietal attention networks were both active during valid and invalid trials across SOAs. These results suggest that the dorsal and ventral networks may not be as specialized during bottom-up auditory orienting as has been previously reported during visual orienting.

Streaming and sound localization with a preceding distractor

Localization of a 2-ms click target was previously shown to be influenced by a preceding identical distractor for inter-click-intervals up to 400 ms [Kopčo, Best, and Shinn-Cunningham (2007). J. Acoust. Soc. Am. 121, 420-432]. Here, two experiments examined whether perceptual organization plays a role in this effect. In the experiments, the distractor was designed either to be grouped with the target (a single-click distractor) or to be processed in a separate stream (an 8-click train). The two distractors affected performance differently, both in terms of bias and variance, suggesting that grouping and streaming play a role in localization in multisource environments.

Fear Spreading Across Senses: Visual Emotional Events Alter Cortical Responses to Touch, Audition, and Vision

Attention and perception are potentiated for emotionally significant stimuli, promoting efficient reactivity and survival. But does such enhancement extend to stimuli simultaneously presented across different sensory modalities? We used functional magnetic resonance imaging in humans to examine the effects of visual emotional signals on concomitant sensory inputs in auditory, somatosensory, and visual modalities. First, we identified sensory areas responsive to task-irrelevant tones, touches, or flickers, presented bilaterally while participants attended to either a neutral or a fearful face. Then, we measured whether these responses were modulated by the emotional content of the face. Sensory responses in primary cortices were enhanced for auditory and tactile stimuli when these appeared with fearful faces, compared with neutral, but striate cortex responses to the visual stimuli were reduced in the left hemisphere, plausibly as a consequence of sensory competition. Finally, conjunction and functional connectivity analyses identified 2 distinct networks presumably responsible for these emotional modulatory processes, involving cingulate, insular, and orbitofrontal cortices for the increased sensory responses, and ventrolateral prefrontal cortex for the decreased sensory responses. These results suggest that emotion tunes the excitability of sensory systems across multiple modalities simultaneously, allowing the individual to adaptively process incoming inputs in a potentially threatening environment.

Visually Induced Inhibition of Return Affects the Integration of Auditory and Visual Information

Multisensory integration (MSI) and exogenous spatial attention can both speedup responses to perceptual events. Recently, it has been shown that audiovisual integration at exogenously attended locations is reduced relative to unattended locations. This effect was observed at short cue-target intervals (200–250 ms). At longer intervals, however, the initial benefits of exogenous shifts of spatial attention at the cued location are often replaced by response time (RT) costs (also known as Inhibition of Return, IOR). Given these opposing cueing effects at shorter versus longer intervals, we decided to investigate whether MSI would also be affected by IOR. Uninformative exogenous visual spatial cues were presented between 350 and 450 ms prior to the onset of auditory, visual, and audiovisual targets. As expected, IOR was observed for visual targets (invalid cue RT < valid cue RT). For auditory and audiovisual targets, neither IOR nor any spatial cueing effects were observed. The amount of relative multisensory response enhancement and race model inequality violation was larger for uncued as compared with cued locations indicating that IOR reduces MSI. The results are discussed in the context of changes in unisensory signal strength at cued as compared with uncued locations.

Contextual plasticity, top-down, and non-auditory factors in sound localization with a distractor

Localization of a 2-ms-click target was previously shown to be influenced by interleaved localization trials in which the target was preceded by an identical distractor [Kopčo, Best, and Shinn-Cunningham (2007). J. Acoust. Soc. Am. 121, 420-432]. Here, two experiments were conducted to explore this contextual effect. Results show that context-related bias is not eliminated (1) when the response method is changed so that vision is available or that no hand-pointing is required; or (2) when the distractor-target order is reversed. Additionally, a keyboard-based localization response method is introduced and shown to be more accurate than traditional pointer-based methods.

A computational model of auditory attention for use in soundscape research

Urban soundscape design involves creating outdoor spaces that are pleasing to the ear. One way to achieve this goal is to add or accentuate sounds that are considered to be desired by most users of the space, such that the desired sounds mask undesired sounds, or at least distract attention away from undesired sounds. In view of removing the need for a listening panel to assess the effectiveness of such soundscape measures, the interest for new models and techniques is growing. In this paper, a model of auditory attention to environmental sound is presented, which balances computational complexity and biological plausibility. Once the model is trained for a particular location, it classifies the sounds that are present in the soundscape and simulates how a typical listener would switch attention over time between different sounds. The model provides an acoustic summary, giving the soundscape designer a quick overview of the typical sounds at a particular location, and allows assessment of the perceptual effect of introducing additional sounds.

The influence of direct and indirect speech on mental representations

Language can be viewed as a set of cues that modulate the comprehender's thought processes. It is a very subtle instrument. For example, the literature suggests that people perceive direct speech (e.g., Joanne said: 'I went out for dinner last night') as more vivid and perceptually engaging than indirect speech (e.g., Joanne said that she went out for dinner last night). But how is this alleged vividness evident in comprehenders' mental representations? We sought to address this question in a series of experiments. Our results do not support the idea that, compared to indirect speech, direct speech enhances the accessibility of information from the communicative or the referential situation during comprehension. Neither do our results support the idea that the hypothesized more vivid experience of direct speech is caused by a switch from the visual to the auditory modality. However, our results do show that direct speech leads to a stronger mental representation of the exact wording of a sentence than does indirect speech. These results show that language has a more subtle influence on memory representations than was previously suggested.

Visual cognition during real social interaction

Laboratory studies of social visual cognition often simulate the critical aspects of joint attention by having participants interact with a computer-generated avatar. Recently, there has been a movement toward examining these processes during authentic social interaction. In this review, we will focus on attention to faces, attentional misdirection, and a phenomenon we have termed social inhibition of return (Social IOR), that have revealed aspects of social cognition that were hitherto unknown. We attribute these discoveries to the use of paradigms that allow for more realistic social interactions to take place. We also point to an area that has begun to attract a considerable amount of interest-that of Theory of Mind (ToM) and automatic perspective taking-and suggest that this too might benefit from adopting a similar approach.

Auditory orienting and inhibition of return in schizophrenia: a functional magnetic resonance imaging study

Patients with schizophrenia (SP) exhibit deficits in both attentional reorienting and inhibition of return (IOR) during visual tasks. However, it is currently unknown whether these deficits are supramodal in nature and how these deficits relate to other domains of cognitive dysfunction. In addition, the neuronal correlates of this pathological orienting response have not been investigated in either the visual or auditory modality. Therefore, 30 SP and 30 healthy controls (HC) were evaluated with an extensive clinical protocol and functional magnetic resonance imaging (fMRI) during an auditory cuing paradigm. SP exhibited both increased costs and delayed IOR during auditory orienting, suggesting a prolonged interval for attentional disengagement from cued locations. Moreover, a delay in the development of IOR was associated with cognitive deficits on formal neuropsychological testing in the domains of attention/inhibition and working memory. Event-related fMRI showed the characteristic activation of a frontoparietal network (invalid trials>valid trials), but there were no differences in functional activation between patients and HC during either attentional reorienting or IOR. Current results suggest that orienting deficits are supramodal in nature in SP, and are related to higher-order cognitive deficits that directly interfere with day-to-day functioning.

Spatial shifts of audio-visual interactions by perceptual learning are specific to the trained orientation and eye

A large proportion of the human cortex is devoted to visual processing. Contrary to the traditional belief that multimodal integration takes place in multimodal processing areas separate from visual cortex, several studies have found that sounds may directly alter processing in visual brain areas. Furthermore, recent findings show that perceptual learning can change the perceptual mechanisms that relate auditory and visual senses. However, there is still a debate about the systems involved in cross-modal learning. Here, we investigated the specificity of audio-visual perceptual learning. Audio-visual cuing effects were tested on a Gabor orientation task and an object discrimination task in the presence of lateralised sound cues before and after eight-days of cross-modal task-irrelevant perceptual learning. During training, the sound cues were paired with visual stimuli that were misaligned at a proximal (trained) visual field location relative to the sound. Training was performed with one eye patched and with only one Gabor orientation. Consistent with previous findings we found that cross-modal perceptual training shifted the audio-visual cueing effect towards the trained retinotopic location. However, this shift in audio-visual tuning was only observed for the trained stimulus (Gabors), at the trained orientation, and in the trained eye. This specificity suggests that multimodal interactions resulting from cross-modal (audio-visual) task-irrelevant perceptual learning involves so-called unisensory visual processing areas in humans. Our findings provide further support for recent anatomical and physiological findings that suggest relatively early interactions in cross-modal processing.

Active inhibition and memory promote exploration and search of natural scenes

Active exploration of the visual world depends on sequential shifts of gaze that bring prioritized regions of a scene into central vision. The efficiency of this system is commonly attributed to a mechanism of "inhibition of return" (IOR) that discourages re-examination of previously-visited locations. Such a process is fundamental to computational models of attentional selection and paralleled by neurophysiological observations of inhibition of target-related activity in visuomotor areas. However, studies examining eye movements in naturalistic visual scenes appear to contradict the hypothesis that IOR promotes exploration. Instead, these reports reveal a surprisingly strong tendency to shift gaze back to the previously fixated location, suggesting that refixations might even be facilitated under natural conditions. Here we resolve this apparent contradiction, based on a probabilistic analysis of gaze patterns recorded during both free-viewing and search of naturalistic scenes. By simulating saccadic selection based on instantaneous influences alone, we show that the observed frequency of return saccades is in fact substantially less than predicted for a memoryless system, demonstrating that refixation is actively inhibited under natural viewing conditions. Furthermore, these observations reveal that gaze history significantly influences the way in which natural scenes are explored, contrary to accounts that suggest visual search has no memory.

Testing equivalence with repeated measures: tests of the difference model of two-alternative forced-choice performance

Solving theoretical or empirical issues sometimes involves establishing the equality of two variables with repeated measures. This defies the logic of null hypothesis significance testing, which aims at assessing evidence against the null hypothesis of equality, not for it. In some contexts, equivalence is assessed through regression analysis by testing for zero intercept and unit slope (or simply for unit slope in case that regression is forced through the origin). This paper shows that this approach renders highly inflated Type I error rates under the most common sampling models implied in studies of equivalence. We propose an alternative approach based on omnibus tests of equality of means and variances and in subject-by-subject analyses (where applicable), and we show that these tests have adequate Type I error rates and power. The approach is illustrated with a re-analysis of published data from a signal detection theory experiment with which several hypotheses of equivalence had been tested using only regression analysis. Some further errors and inadequacies of the original analyses are described, and further scrutiny of the data contradict the conclusions raised through inadequate application of regression analyses.

Automatic selective attention as a function of sensory modality in aging

OBJECTIVES

It was recently hypothesized that age-related differences in selective attention depend on sensory modality (Guerreiro, M. J. S., Murphy, D. R., & Van Gerven, P. W. M. (2010). The role of sensory modality in age-related distraction: A critical review and a renewed view. Psychological Bulletin, 136, 975-1022. doi:10.1037/a0020731). So far, this hypothesis has not been tested in automatic selective attention. The current study addressed this issue by investigating age-related differences in automatic spatial cueing effects (i.e., facilitation and inhibition of return [IOR]) across sensory modalities.

METHODS

Thirty younger (mean age = 22.4 years) and 25 older adults (mean age = 68.8 years) performed 4 left-right target localization tasks, involving all combinations of visual and auditory cues and targets. We used stimulus onset asynchronies (SOAs) of 100, 500, 1,000, and 1,500 ms between cue and target.

RESULTS

The results showed facilitation (shorter reaction times with valid relative to invalid cues at shorter SOAs) in the unimodal auditory and in both cross-modal tasks but not in the unimodal visual task. In contrast, there was IOR (longer reaction times with valid relative to invalid cues at longer SOAs) in both unimodal tasks but not in either of the cross-modal tasks. Most important, these spatial cueing effects were independent of age.

DISCUSSION

The results suggest that the modality hypothesis of age-related differences in selective attention does not extend into the realm of automatic selective attention.

Multisensory perceptual learning reshapes both fast and slow mechanisms of crossmodal processing

Previous research has shown that sounds facilitate perception of visual patterns appearing immediately after the sound but impair perception of patterns appearing after some delay. Here we examined the spatial gradient of the fast crossmodal facilitation effect and the slow inhibition effect in order to test whether they reflect separate mechanisms. We found that crossmodal facilitation is only observed at visual field locations overlapping with the sound, whereas crossmodal inhibition affects the whole hemifield. Furthermore, we tested whether multisensory perceptual learning with misaligned audio-visual stimuli reshapes crossmodal facilitation and inhibition. We found that training shifts crossmodal facilitation towards the trained location without changing its range. By contrast, training narrows the range of inhibition without shifting its position. Our results suggest that crossmodal facilitation and inhibition reflect separate mechanisms that can both be reshaped by multisensory experience even in adult humans. Multisensory links seem to be more plastic than previously thought.

An attentional approach to study mental representations of different parts of the hand

The aim of this study is to investigate whether the fingers are represented separately from the palm. An exogenous spatial orientation paradigm was used where participants had to detect a tactile stimulus that could appear on the palm, the middle finger or the ring finger of the left hand. The tactile target was preceded by a non-predictive cue using different stimulus-onset asynchronies (SOA). We observed a Facilitation Effect in the palm and inhibition of return (IOR) for fingers using a short cue-target SOA, whereas the IOR was found in fingers and palm in long cue-target SOA. Also we observed a 'Cue above Target' effect (facilitation effect when the Cue had appeared distal to the target location in a vertical line) at the long SOA. Together, we suggest that the general pattern of results supports the proposed hypothesis about the different mental representation of fingers and palms, but with a considerable and hierarchical interrelation between them.

Social inhibition of return

Responses to a target stimulus can be slower when it appears in the same rather than a different location to a previous event, an effect known as inhibition of return (IOR). Recently, it has been shown that when two people alternate responses to a target, one person's responses are slower when they are directed to the same locations as their partner's previous response. The present study sought to investigate this highly novel effect, which we term social IOR (SIOR), in relation to what is known of IOR in individuals performing alone. We found that only a real conspecific can induce SIOR in another person, whereas an animated conspecific cannot. Additionally, SIOR emerges at locations to which a conspecific has been inferred to respond, even when direct observation of his/her responses is prevented. Finally, SIOR can be induced without the abrupt visual transients normally associated with the generation of IOR. These findings suggest that SIOR is the result of stronger activations of IOR mechanisms, or that it is subserved by entirely separate inhibitory processes.

Context-Sensitive Information Presentation: Integrating Adaptive and Adaptable Approaches to Display Design

University of MichiganOperators in many complex event-driven domains face the challenge of data overload. Two major contributors to this problem are over-reliance in display design on one sensory channel (vision in most domains) and the fact that the presentation of data and information does not vary to account for changing task contexts and operator states. These problems call for the introduction of context-sensitive multimodal displays. There is a substantial and growing body of research on multisensory information processing and presentation. However, little guidance is available for the design of flexible displays that take context into consideration. Two important research questions are: 1) who should be in control of the adaptation of information presentation – the user or the system, or perhaps both? - and b) what factors should drive display adaptation? This article will review two approaches to context-sensitive display design: adaptive and adaptable. The benefits and disadvantages of each approach will be discussed, and a recently developed hybrid adaptive-adaptable multimodal interface will be described. To our knowledge, this is the first display design that combines both approaches to context-sensitivity and employs a wide range of drivers, ranging from environmental conditions to operator states and performance.

Neuronal modulation of auditory attention by informative and uninformative spatial cues

Sounds provide important information about the spatial environment, including the location of approaching objects. Attention to sounds can be directed through automatic or more controlled processes, which have been well studied in the visual modality. However, little is known about the neural underpinnings of attentional control mechanisms for auditory signals. We studied healthy adults who underwent event-related FMRI while performing a task that manipulated automatic and more controlled auditory orienting by varying the probability that cues correctly predicted target location. Specifically, we examined the effects of uninformative (50% validity ratio) and informative (75% validity ratio) auditory cues on reaction time (RT) and neuronal functioning. The stimulus-onset asynchrony (SOA) between the cue and the target was either 100 or 800 ms. At the 100 ms SOA, RT was faster for valid than invalid trials for both cue types, and frontoparietal activation was greater for invalid than valid trials. At the 800 ms SOA, RT and functional activation depended on whether cues were informative or uninformative, and whether cues correctly or incorrectly predicted the target location. Contrary to our prediction, activation in a frontoparietal network was greater for uninformative than informative cues across several different comparisons and at both SOAs. This finding contrasts with similar research of visual orienting, and suggests that the auditory modality may be more biased toward automatic shifts of attention following uninformative cues.

Crossmodal Links in Attention in the Driving Environment: The Roles of Cueing Modality, Signal Timing, and Workload

Multimodal information presentation has been proposed as a means to support timesharing in complex data-rich environments. To ensure the effectiveness of this approach, it is necessary to consider performance effects of recently discovered crossmodal spatial and temporal links in attention, as well as their interaction with other performance-shaping factors. The main goals of this research were to confirm that performance effects of crossmodal links in spatial attention scale to complex environments and to examine how these effects vary as a function of cue modality, signal timing, and workload. In the present study, set in a driving simulation, spatially valid and invalid auditory and tactile cues preceded the presentation of visual targets at various stimulus-onset asynchronies and under different levels of workload induced by simulated wind gusts of varied intensity. The findings from this experiment confirm that visual target identification accuracies and response times are, overall, more accurate and faster when validly-cued. Significant interactions were found between cue validity, stimulus onset asynchrony (SOA), and cue modality, such that valid tactile cueing is most beneficial at shorter (100–200 ms) SOAs, while valid auditory cueing resulted in faster responses than invalid cueing at 500 ms SOAs, but slower responses at 1000 ms SOAs. Tactile error rates were significantly higher than auditory error rates at various interactions of modality and SOA. These findings were robust across all workload conditions. They highlight the need for context-sensitive information presentation and can inform the design of multimodal interfaces for a wide range of application domains.

Investigating the time course of tactile reflexive attention using a non-spatial discrimination task

Peripheral cues are thought to facilitate responses to stimuli presented at the same location because they lead to exogenous attention shifts. Facilitation has been observed in numerous studies of visual and auditory attention, but there have been only four demonstrations of tactile facilitation, all in studies with potential confounds. Three studies used a spatial (finger versus thumb) discrimination task, where the cue could have provided a spatial framework that might have assisted the discrimination of subsequent targets presented on the same side as the cue. The final study circumvented this problem by using a non-spatial discrimination; however, the cues were informative and interspersed with visual cues which may have affected the attentional effects observed. In the current study, therefore, we used a non-spatial tactile frequency discrimination task following a non-informative tactile white noise cue. When the target was presented 150 ms after the cue, we observed faster discrimination responses to targets presented on the same side compared to the opposite side as the cue; by 1000 ms, responses were significantly faster to targets presented on the opposite side to the cue. Thus, we demonstrated that tactile attentional facilitation can be observed in a non-spatial discrimination task, under unimodal conditions and with entirely non-predictive cues. Furthermore, we provide the first demonstration of significant tactile facilitation and tactile inhibition of return within a single experiment.

Disentangling perceptual and motor components in inhibition of return

Following an abrupt onset of a peripheral stimulus (a cue), the response to a visual target is faster when the target appears at the cued position than when it appears at other positions. However, if the stimulus onset asynchrony (SOA) is longer than approximately 300 ms, the response to the target is slower at the cued position than that at other positions. This phenomenon of a longer response time to cued targets is called "inhibition of return" (IOR). Previous hypotheses propose contributions of both response inhibition and attentional inhibition at cued position to IOR, and suggest that responding to the cue can eliminate the component of response inhibition. The current study uses tasks either executing or withholding response to the cue to investigate the relative contributions of response and attention components to IOR. A condition with bilateral display of the cue is also chosen as a control condition, and eight different SOAs between 1,000 and 2,750 ms are tested. Compared to the control condition, response delay to the target at a cued position is eliminated by responding to the cue, and a response advantage to the target at an uncued position is not affected by responding to the cue. Furthermore, both response delay at a cued position and response advantage at an uncued position decrease with SOA in the time window tested in these experiments. The results reported here indicate a dominant response inhibition at a cued position and a primary attentional allocation at an uncued position for IOR. Nonsignificant perceptual/attentional suppression at a cued position is argued to be a benefit for visual detection in a changing world.

Cross-modal links among vision, audition, and touch in complex environments

OBJECTIVES

This study sought to determine whether performance effects of cross-modal spatial links that were observed in earlier laboratory studies scale to more complex environments and need to be considered in multimodal interface design. It also revisits the unresolved issue of cross-modal cuing asymmetries.

BACKGROUND

Previous laboratory studies employing simple cues, tasks, and/or targets have demonstrated that the efficiency of processing visual, auditory, and tactile stimuli is affected by the modality, lateralization, and timing of surrounding cues. Very few studies have investigated these cross-modal constraints in the context of more complex environments to determine whether they scale and how complexity affects the nature of cross-modal cuing asymmetries.

METHOD

Amicroworld simulation of battlefield operations with a complex task set and meaningful visual, auditory, and tactile stimuli was used to investigate cuing effects for all cross-modal pairings.

RESULTS

Significant asymmetric performance effects of cross-modal spatial links were observed. Auditory cues shortened response latencies for collocated visual targets but visual cues did not do the same for collocated auditory targets. Responses to contralateral (rather than ipsilateral) targets were faster for tactually cued auditory targets and each visual-tactile cue-target combination, suggesting an inhibition-of-return effect.

CONCLUSIONS

The spatial relationships between multimodal cues and targets significantly affect target response times in complex environments. The performance effects of cross-modal links and the observed cross-modal cuing asymmetries need to be examined in more detail and considered in future interface design.

APPLICATION

The findings from this study have implications for the design of multimodal and adaptive interfaces and for supporting attention management in complex, data-rich domains.

Inhibition of Return Arises from Inhibition of Response Processes: An Analysis of Oscillatory Beta Activity

In the orienting of attention paradigm, inhibition of return (IOR) refers to slowed responses to targets presented at the same location as a preceding stimulus. No consensus has yet been reached regarding the stages of information processing underlying the inhibition. We report the results of an electro-encephalogram experiment designed to examine the involvement of response inhibition in IOR. Using a cue-target design and a target-target design, we addressed the role of response inhibition in a location discrimination task. Event-related changes in beta power were measured because oscillatory beta activity has been shown to be related to motor activity. Bilaterally located sources in the primary motor cortex showed event-related beta desynchronization (ERD) both at cue and target presentation and a rebound to event-related beta synchronization (ERS) after movement execution. In both designs, IOR arose from an enhancement of beta synchrony. IOR was related to an increase of beta ERS in the target-target design and to a decrease of beta ERD in the cue-target design. These results suggest an important role of response inhibition in IOR.

An event-related fMRI Study of exogenous facilitation and inhibition of return in the auditory modality

The orienting of attention to different locations in space is fundamental to most organisms and occurs in all sensory modalities. Orienting has been extensively studied in vision, but to date, few studies have investigated neuronal networks underlying automatic orienting of attention and inhibition of return to auditory signals. In the current experiment, functional magnetic resonance imaging and behavioral data were collected while healthy volunteers performed an auditory orienting task in which a monaurally presented tone pip (cue) correctly or incorrectly cued the location of a target tone pip. The stimulus onset asynchrony (SOA) between the cue and target was 100 or 800 msec. Behavioral results were consistent with previous studies showing that valid auditory cues produced facilitation at the short SOA and inhibition of return at the long SOA. Functional results indicated that the reorienting of attention (100 msec SOA) and inhibition of return (800 msec SOA) were mediated by both common and distinct neuronal structures. Both attention mechanisms commonly activated a network consisting of fronto-oculomotor areas, the left postcentral gyrus, right premotor area, and bilateral tonsil of the cerebellum. Several distinct areas of frontal and parietal activation were identified for the reorienting condition, whereas the right inferior parietal lobule was the only structure uniquely associated with inhibition of return.

The effect of age on inhibition of return is independent of non-ocular response inhibition

Inhibition of return (IOR) refers to the slowing of a response to a target stimulus presented in the same location as a previous stimulus. Increased IOR has been observed in older adults, despite a reduction in other 'inhibitory' processes. However, cue-target tasks have been used in all previous studies and because of this, IOR may have been overestimated due to non-ocular response inhibition associated with withholding a response from the cue. Could increased levels of response inhibition account for the observations of increased IOR in older adults? This confound can be circumvented by using a target-target paradigm, in which a response is made to all stimuli. We tested three groups of 24 subjects: young (mean 22.5 years), young-old (mean 61.9 years) and old-old (mean 74.8 years). Subjects completed both visual cue-target and target-target tasks with identical inter-stimulus intervals of 1400 and 1800ms. IOR magnitude increased with age in both the cue-target task and the target-target task. Furthermore, the magnitude of visual IOR was found to increase with age even when individual differences in baseline response speed were taken into account. Thus, there appears to be a genuine increase in IOR magnitude with age.

Interaction between location- and frequency-based inhibition of return in human auditory system

Using a cue-target paradigm, this study investigated the interaction between location and frequency information processing in human auditory inhibition of return (IOR). The cue and the target varied in terms of location and frequency and participants were asked to perform a target detection, localization or frequency discrimination task. Results showed that, when neither location nor frequency of auditory stimuli was particularly relevant to the target detection task, there was a location-based IOR only if the cue and the target were identical in frequency and there was a frequency-based IOR only if the cue and the target were presented at the same location. When a particular feature of auditory stimuli, whether location or frequency, was directly relevant to the current task, the IOR effect was evident for this feature only if the cue and the target differed on the task-irrelevant feature, while the IOR effect was eliminated for the task-relevant feature when the cue and the target had the same task-irrelevant feature. Similarly, the IOR effect based on the task-irrelevant feature was evident when the cue and the target differed on the task-relevant feature, and was eliminated or reversed when the cue and the target shared the task-relevant feature. Theoretical implications of these findings for auditory IOR are discussed.

Influence of Endogenous and Exogenous Orientations of Attention on Inhibition of Return in a Cross-Modal Target—Target Aiming Task

The authors conducted 2 experiments in which participants (N = 16 in each) executed successive unimanual aiming movements to target locations that were indicated by the onset of either an auditory or a visual stimulus. In Experiment 1 (exogenous orientation), inhibition of return (IOR) effects were observed, with reliable reaction time (RT) costs associated with movements returning to the same target and a trend toward larger IOR effects in left than in right space. There was no influence of stimulus modality on the magnitude of IOR. IOR was also observed in Experiment 2 (endogenous orientation), except the influence of stimulus modality reliably mediated those effect. In that case, IOR was evident only when the previous modality was visual and the current modality was auditory. Together, the results of those 2 experiments suggest that in situations in which 2 paired movements constitute the response criteria, IOR is both supramodal and lateralized to contralateral space.