Switching attention between modalities: further evidence for visual dominance

Impact of aging on crossmodal attention switching

Previous studies on crossmodal visual-auditory attention switching using a spatial discrimination task showed performance costs when the target modality changed relative to when it repeated. The present study (n = 42 for each age group) examined age-related changes in crossmodal attention switching by asking young (age range 19 to 30 years old) and older (age range 64 to 80 years old) participants to respond to unimodal central cues and bimodal lateralized stimuli. The participants' task was to indicate the location of the target in the relevant modality using button presses. Results showed general attention switch costs. Additionally, we found no specific age-related increase of attention switch costs (no difference in performance between switch and repetition of target modality), but age-related increased mixing costs (decreased performance for repetition in modality-mixed condition compared to single target modality). Moreover, spatial distraction produced a crossmodal congruency effect, which was only slightly larger in older adults. Taken together, age-related increased mixing costs suggest a general difficulty with maintaining more than one task, but no specific age-related crossmodal impairment in crossmodal attention switching.

Exploring behavioral adjustments of proportion congruency manipulations in an Eriksen flanker task with visual and auditory distractor modalities

The present study investigated global behavioral adaptation effects to conflict arising from different distractor modalities. Three experiments were conducted using an Eriksen flanker paradigm with constant visual targets, but randomly varying auditory or visual distractors. In Experiment 1, the proportion of congruent to incongruent trials was varied for both distractor modalities, whereas in Experiments 2A and 2B, this proportion congruency (PC) manipulation was applied to trials with one distractor modality (inducer) to test potential behavioral transfer effects to trials with the other distractor modality (diagnostic). In all experiments, mean proportion congruency effects (PCEs) were present in trials with a PC manipulation, but there was no evidence of transfer to diagnostic trials in Experiments 2A and 2B. Distributional analyses (delta plots) provided further evidence for distractor modality-specific global behavioral adaptations by showing differences in the slope of delta plots with visual but not auditory distractors when increasing the ratio of congruent trials. Thus, it is suggested that distractor modalities constrain global behavioral adaptation effects due to the learning of modality-specific memory traces (e.g., distractor-target associations) and/or the modality-specific cognitive control processes (e.g., suppression of modality-specific distractor-based activation). Moreover, additional analyses revealed partial transfer of the congruency sequence effect across trials with different distractor modalities suggesting that distractor modality may differentially affect local and global behavioral adaptations.

Embodied Processing at Six Linguistic Granularity Levels: A Consensus Paper

Language processing is influenced by sensorimotor experiences. Here, we review behavioral evidence for embodied and grounded influences in language processing across six linguistic levels of granularity. We examine (a) sub-word features, discussing grounded influences on iconicity (systematic associations between word form and meaning); (b) words, discussing boundary conditions and generalizations for the simulation of color, sensory modality, and spatial position; (c) sentences, discussing boundary conditions and applications of action direction simulation; (d) texts, discussing how the teaching of simulation can improve comprehension in beginning readers; (e) conversations, discussing how multi-modal cues improve turn taking and alignment; and (f) text corpora, discussing how distributional semantic models can reveal how grounded and embodied knowledge is encoded in texts. These approaches are converging on a convincing account of the psychology of language, but at the same time, there are important criticisms of the embodied approach and of specific experimental paradigms. The surest way forward requires the adoption of a wide array of scientific methods. By providing complimentary evidence, a combination of multiple methods on various levels of granularity can help us gain a more complete understanding of the role of embodiment and grounding in language processing.

Investigating the influence of visuospatial stimuli on driver's speed perception: a laboratory study

Driving at an inappropriate speed is a major accident cause in the EU. Understanding the underlying sensory mechanisms can help to reduce speed and increase traffic safety. The present study investigated the effect of visuospatial stimuli on speed perception using an adaptive countermeasure to speeding based on a manipulation of optic flow. We added red lights on both sides of a simulated road. We expected speed to be perceived as faster when lights moved toward drivers due to increased optic flow, whereas we expected static light stimuli to not alter the optic flow and thus not influence speed perception. Two experiments applied the method of constant stimuli. To this end, participants encountered several trials of two video sequences on a straight road. A reference sequence showed the same traveling speed while test sequences varied around different traveling speeds. Participants indicated which sequence they perceived as faster, leading to the calculation of the point of subjective equality (PSE). A lower PSE indicates that the speed in this experimental condition is perceived as faster than in another experimental condition. Experiment 1A did not show a difference between PSEs of static and oncoming lights. Because participants had counted reflector posts for speed estimation, we removed these reflector posts in Experiment 1B and found a lower PSE for oncoming lights. Thus, such light stimuli may have an effect only in situations without other competing visual stimuli supporting speed perception. Future research should investigate whether speed perception is indeed a primarily visuospatial control task or whether other sensory information such as auditory factors can have an influence as well.

Dynamics of alpha suppression index both modality specific and general attention processes

EEG alpha power varies under many circumstances requiring visual attention. However, mounting evidence indicates that alpha may not only serve visual processing, but also the processing of stimuli presented in other sensory modalities, including hearing. We previously showed that alpha dynamics during an auditory task vary as a function of competition from the visual modality (Clements et al., 2022) suggesting that alpha may be engaged in multimodal processing. Here we assessed the impact of allocating attention to the visual or auditory modality on alpha dynamics at parietal and occipital electrodes, during the preparatory period of a cued-conflict task. In this task, bimodal precues indicated the modality (vision, hearing) relevant to a subsequent reaction stimulus, allowing us to assess alpha during modality-specific preparation and while switching between modalities. Alpha suppression following the precue occurred in all conditions, indicating that it may reflect general preparatory mechanisms. However, we observed a switch effect when preparing to attend to the auditory modality, in which greater alpha suppression was elicited when switching to the auditory modality compared to repeating. No switch effect was evident when preparing to attend to visual information (although robust suppression did occur in both conditions). In addition, waning alpha suppression preceded error trials, irrespective of sensory modality. These findings indicate that alpha can be used to monitor the level of preparatory attention to process both visual and auditory information, and support the emerging view that alpha band activity may index a general attention control mechanism used across modalities.

Processing symmetry between visual and auditory spatial representations in updating working memory

Updating spatial representations in visual and auditory working memory relies on common processes, and the modalities should compete for attentional resources. If competition occurs, one type of spatial information is presumably weighted over the other, irrespective of sensory modality. This study used incompatible spatial information conveyed from two different cue modalities to examine relative dominance in memory updating. Participants mentally manoeuvred a designated target in a matrix according to visual or auditory stimuli that were presented simultaneously, to identify a terminal location. Prior to the navigation task, the relative perceptual saliences of the visual cues were manipulated to be equal, superior, or inferior to the auditory cues. The results demonstrate that visual and auditory information competed for attentional resources, such that visual/auditory guidance was impaired by incongruent cues delivered from the other modality. Although visual bias was generally observed in working-memory navigation, stimuli of relatively high salience interfered with or facilitated other stimuli regardless of modality, demonstrating the processing symmetry of spatial updating in visual and auditory spatial working memory. Furthermore, this processing symmetry can be identified during the encoding of sensory inputs into working-memory representations. The results imply that auditory spatial updating is comparable to visual spatial updating in that salient stimuli receive a high priority when selecting inputs and are used when tracking spatial representations.

Are some effector systems harder to switch to? In search of cost asymmetries when switching between manual, vocal, and oculomotor tasks

In task-switching studies, performance is typically worse in task-switch trials than in task-repetition trials. These switch costs are often asymmetrical, a phenomenon that has been explained by referring to a dominance of one task over the other. Previous studies also indicated that response modalities associated with two tasks may be considered as integral components for defining a task set. However, a systematic assessment of the role of response modalities in task switching is still lacking: Are some response modalities harder to switch to than others? The present study systematically examined switch costs when combining tasks that differ only with respect to their associated effector systems. In Experiment 1, 16 participants switched (in unpredictable sequence) between oculomotor and vocal tasks. In Experiment 2, 72 participants switched (in pairwise combinations) between oculomotor, vocal, and manual tasks. We observed systematic performance costs when switching between response modalities under otherwise constant task features and could thereby replicate previous observations of response modality switch costs. However, we did not observe any substantial switch-cost asymmetries. As previous studies using temporally overlapping dual-task paradigms found substantial prioritization effects (in terms of asymmetric costs) especially for oculomotor tasks, the present results suggest different underlying processes in sequential task switching than in simultaneous multitasking. While more research is needed to further substantiate a lack of response modality switch-cost asymmetries in a broader range of task switching situations, we suggest that task-set representations related to specific response modalities may exhibit rapid decay.

Self-prioritization with unisensory and multisensory stimuli in a matching task

A shape-label matching task is commonly used to examine the self-advantage in motor reaction-time responses (the Self-Prioritization Effect; SPE). In the present study, auditory labels were introduced, and, for the first time, responses to unisensory auditory, unisensory visual, and multisensory object-label stimuli were compared across block-type (i.e., trials blocked by sensory modality type, and intermixed trials of unisensory and multisensory stimuli). Auditory stimulus intensity was presented at either 50 dB (Group 1) or 70 dB (Group 2). The participants in Group 2 also completed a multisensory detection task, making simple speeded motor responses to the shape and sound stimuli and their multisensory combinations. In the matching task, the SPE was diminished in intermixed trials, and in responses to the unisensory auditory stimuli as compared with the multisensory (visual shape+auditory label) stimuli. In contrast, the SPE did not differ in responses to the unisensory visual and multisensory (auditory object+visual label) stimuli. The matching task was associated with multisensory ‘costs’ rather than gains, but response times to self- versus stranger-associated stimuli were differentially affected by the type of multisensory stimulus (auditory object+visual label or visual shape+auditory label). The SPE was thus modulated both by block-type and the combination of object and label stimulus modalities. There was no SPE in the detection task. Taken together, these findings suggest that the SPE with unisensory and multisensory stimuli is modulated by both stimulus- and task-related parameters within the matching task. The SPE does not transfer to a significant motor speed gain when the self-associations are not task-relevant.

Auditory perception dominates in motor rhythm reproduction

It is commonly agreed that vision is more sensitive to spatial information, while audition is more sensitive to temporal information. When both visual and auditory information are available simultaneously, the modality appropriateness hypothesis predicts that, depending on the task, the most appropriate (i.e., reliable) modality dominates perception. While previous research mainly focused on discrepant information from different sensory inputs to scrutinize the modality appropriateness hypothesis, the current study aimed at investigating the modality appropriateness hypothesis when multimodal information was provided in a nondiscrepant and simultaneous manner. To this end, participants performed a temporal rhythm reproduction task for which the auditory modality is known to be the most appropriate. The experiment comprised an auditory (i.e., beeps), a visual (i.e., flashing dots), and an audiovisual condition (i.e., beeps and dots simultaneously). Moreover, constant as well as variable interstimulus intervals were implemented. Results revealed higher accuracy and lower variability in the auditory condition for both interstimulus interval types when compared to the visual condition. More importantly, there were no differences between the auditory and the audiovisual condition across both interstimulus interval types. This indicates that the auditory modality dominated multimodal perception in the task, whereas the visual modality was disregarded and hence did not add to reproduction performance.

Contextual Features of the Cue Enter Episodic Bindings in Task Switching

Evidence suggests that the features of a stimulus and the actions performed on it are bound together into a coherent mental representation of the episode, which is retrieved from memory upon reencountering at least one of these features. Effects of such binding and retrieval processes emerge in action control, such as in multitasking situations like task switching. In the task-switching paradigm, response-repetition benefits are observed in task repetitions, but response-repetition costs in task switches. This interaction of task repetition (vs. switch) with response repetition (vs. switch) may be explained in terms of task-response binding. In two experiments, we included a task-irrelevant contextual feature in a cued task-switching paradigm using word identification tasks. In Experiment 1, the cue modality could vary between visual and auditory; in Experiment 2, the cue language could vary between English and Spanish, while the target stimulus was always presented visually and in German. We predicted that repeating the contextual feature in the subsequent trial would retrieve the features of the previous trial, even though cue modality or cue language did not afford any response and were not associated with either task. The results showed that response repetition-benefits in task repetitions were observable when the context (i.e., the modality or the language of the cue) repeated but disappeared when the context switched from the previous trial. These results are consistent with context-specific binding and retrieval processes in task switching.

Cross-modal perceptual enhancement of unisensory targets is uni-directional and does not affect temporal expectations

Temporal structures in the environment can shape temporal expectations (TE); and previous studies demonstrated that TEs interact with multisensory interplay (MSI) when multisensory stimuli are presented synchronously. Here, we tested whether other types of MSI - evoked by asynchronous yet temporally flanking irrelevant stimuli - result in similar performance patterns. To this end, we presented sequences of 12 stimuli (10 Hz) which consisted of auditory (A), visual (V) or alternating auditory-visual stimuli (e.g. A-V-A-V-…) with either auditory or visual targets (Exp. 1). Participants discriminated target frequencies (auditory pitch or visual spatial frequency) embedded in these sequences. To test effects of TE, the proportion of early and late temporal target positions was manipulated run-wise. Performance for unisensory targets was affected by temporally flanking distractors, with auditory temporal flankers selectively improving visual target perception (Exp. 1). However, no effect of temporal expectation was observed. Control experiments (Exp. 2-3) tested whether this lack of TE effect was due to the higher presentation frequency in Exp. 1 relative to previous experiments. Importantly, even at higher stimulation frequencies redundant multisensory targets (Exp. 2-3) reliably modulated TEs. Together, our results indicate that visual target detection was enhanced by MSI. However, this cross-modal enhancement - in contrast to the redundant target effect - was still insufficient to generate TEs. We posit that unisensory target representations were either instable or insufficient for the generation of TEs while less demanding MSI still occurred; highlighting the need for robust stimulus representations when generating temporal expectations.

How Task Set and Task Switching Modulate Perceptual Processes: Is Recognition of Facial Emotion an Exception?

In Part 1 we review task-switching and other studies showing that, even with time for preparation, participants' ability to shift attention to a relevant attribute or object before the stimulus onset is limited: there is a 'residual cost'. In particular, several brain potential markers of perceptual encoding are delayed on task-switch trials, compared to task-repeat trials that require attention to the same attribute as before. Such effects have been documented even for a process often considered 'automatic' - visual word recognition: ERP markers of word frequency and word/nonword status are (1) delayed when the word recognition task follows a judgement of a perceptual property compared to repeating the lexical task, and (2) strongly attenuated during the perceptual judgements. Thus, even lexical access seems influenced by the task/attentional set. In Part 2, we report in detail a demonstration of what seems to be a special case, where task-set and a task switch have no such effect on perceptual encoding. Participants saw an outline letter superimposed on a face expressing neutral or negative emotion, and were auditorily cued to categorise the letter as vowel/consonant, or the face as emotional/neutral. ERPs exhibited a robust emotional-neutral difference (Emotional Expression Effect) no smaller or later when switching to the face task than when repeating it; in the first half of its time-course it did not vary with the task at all. The initial encoding of the valence of a fixated facial emotional expression appears to be involuntary and invariant, whatever the endogenous task/attentional set.

Cross-Modal Transfer Following Auditory Task-Switching Training in Old Adults

Maintaining and coordinating multiple task-sets is difficult and leads to costs, however task-switching training can reduce these deficits. A recent study in young adults demonstrated that this training effect occurs at an amodal processing level. Old age is associated with reduced cognitive plasticity and further increases the performance costs when mixing multiple tasks. Thus, cognitive aging might be a limiting factor for inducing cross-modal training effects in a task-switching environment. We trained participants, aged 62-83 years, with an auditory task-switching paradigm over four sessions (2880 total trials), to investigate whether training-related reductions in task-switching costs would also manifest in an untrained visual modality version of the task. Two control groups trained with single tasks (active control) or not trained (passive control) allowed us to identify improvements specific to task-switching training. To make statistical evaluations of any age differences in training and cross-modal transfer, the data from the Kattner cohort were incorporated into the present analysis. Despite the tendency for older adults to respond more cautiously, task-switching training specifically led to a mixing cost reduction in both trained and untrained modalities, the magnitude of which was statistically similar regardless of age. In line with a growing body of research, we failed to observe any far transfer effects in measures of inhibition, working memory or fluid intelligence. Overall, we conclude that any apparent cognitive limitations associated with aging do not prevent cognitive control processes which support set-shifting from improving at an amodal level.

Top-down task-specific determinants of multisensory motor reaction time enhancements and sensory switch costs

This study was designed to investigate the complex interplay between multisensory processing, top–down processes related to the task relevance of sensory signals, and sensory switching. Thirty-five adults completed either a speeded detection or a discrimination task using the same auditory and visual stimuli and experimental setup. The stimuli consisted of unisensory and multisensory presentations of the letters ‘b’ and ‘d’. The multisensory stimuli were either congruent (e.g., the grapheme ‘b’ with the phoneme /b/) or incongruent (e.g., the grapheme ‘b’ with the phoneme /d/). In the detection task, the participants had to respond to all of the stimuli as rapidly as possible while, in the discrimination task, they only responded on those trials where one prespecified letter (either ‘b’ or ‘d’) was present. Incongruent multisensory stimuli resulted in faster responses as compared to unisensory stimuli in the detection task. In the discrimination task, only the dual-target congruent stimuli resulted in faster RTs, while the incongruent multisensory stimuli led to slower RTs than to unisensory stimuli; RTs were the slowest when the visual (rather than the auditory) signal was irrelevant, thus suggesting visual dominance. Switch costs were also observed when switching between unisensory target stimuli, while dual-target multisensory stimuli were less likely to be affected by sensory switching. Taken together, these findings suggest that multisensory motor enhancements and sensory switch costs are influenced by top–down modulations determined by task instructions, which can override the influence of prior learnt associations.

An Experimenter's Influence on Motor Enhancements: The Effects of Letter Congruency and Sensory Switch-Costs on Multisensory Integration

Multisensory integration can alter information processing, and previous research has shown that such processes are modulated by sensory switch costs and prior experience (e.g., semantic or letter congruence). Here we report an incidental finding demonstrating, for the first time, the interplay between these processes and experimental factors, specifically the presence (vs. absence) of the experimenter in the testing room. Experiment 1 demonstrates that multisensory motor facilitation in response to audiovisual stimuli (circle and tone with no prior learnt associations) is higher in those trials in which the sensory modality switches than when it repeats. Those participants who completed the study while alone exhibited increased RT variability. Experiment 2 replicated these findings using the letters “b” and “d” presented as unisensory stimuli or congruent and incongruent multisensory stimuli (i.e., grapheme-phoneme pairs). Multisensory enhancements were inflated following a sensory switch; that is, congruent and incongruent multisensory stimuli resulted in significant gains following a sensory switch in the monitored condition. However, when the participants were left alone, multisensory enhancements were only observed for repeating incongruent multisensory stimuli. These incidental findings therefore suggest that the effects of letter congruence and sensory switching on multisensory integration are partly modulated by the presence of an experimenter.

Cognitive control in number processing: new evidence from task switching

Recently, it was demonstrated that even basic numerical cognition such as the processing of number magnitude is under cognitive control. However, evidence so far primarily came from adaptation effects to stimulus characteristics (e.g., relative frequency of specific stimulus categories). Expanding this approach, we evaluated a possible influence of more active exertion of cognitive control on basic number processing in task switching. Participants had to perform a magnitude comparison task while we manipulated the order of compatible and incompatible input–output modalities (i.e., auditory/vocal input–visual/manual output vs. auditory/visual input–manual/vocal output, respectively) on the trial level, differentiating repeat vs. switch trials. Results indicated that the numerical distance effect but not the problem size effect was increased after a switch in input–output modality compatibility. In sum, these findings substantiate that basic number processing is under cognitive control by providing first evidence that it is influenced by the active exertion of cognitive control as required in task switching.

Modality compatibility in task switching depends on processing codes and task demands

Modality compatibility denotes the match between sensory stimulus modality and the sensory modality of the anticipated response effect (for example, vocal responses usually lead to auditory effects, so that auditory-vocal stimulus-response mappings are modality-compatible, whereas visual-vocal mappings are modality incompatible). In task switching studies, it has been found that switching between two modality-incompatible mappings (auditory-manual and visual-vocal) resulted in higher switch costs than switching between two modality-compatible mappings (auditory-vocal and visual-manual). This finding suggests that with modality-incompatible mappings, the anticipation of the effect of each response primes the stimulus modality linked to the competing task, creating task confusion. In Experiment 1, we examined whether modality-compatibility effects in task switching are increased by strengthening the auditory-vocal coupling using spatial-verbal stimuli relative to spatial-location stimuli. In Experiment 2, we aimed at achieving the same goal by requiring temporal stimulus discrimination relative to spatial stimulus localisation. Results suggest that both spatial-verbal stimuli and temporal discrimination can increase modality-specific task interference through a variation of the strength of anticipation in the response-effect coupling. This provides further support for modality specificity of cognitive control processes in task switching.

Probability-driven and stimulus-driven orienting of attention to time and sensory modality

The timing and the sensory modality of behaviorally relevant events often vary predictably, so that it is beneficial to adapt the sensory system to their statistical regularities. Indeed, statistical information about target timing and/or sensory modality modulates behavioral responses-called expectation effects. Responses are also facilitated by short-term repetitions of target timing and/or sensory modality-called priming effects. We examined how the expectation and priming effects on target timing (short vs. long cue-to-target interval) and target modality (auditory vs. visual) interacted. Temporal expectation was manipulated across blocks, while modality expectation was manipulated across participants. Responses were faster when targets were presented at the expected timing and/or in the expected modality in an additive manner, suggesting that temporal and modality expectation operate relatively independently. Similarly, responses were faster when the timing and/or modality of targets was repeated across trials in an additive manner, suggesting that temporal and modality priming operate relatively independently. Importantly, the interactions between expectation and priming were domain specific. In the temporal domain, temporal-expectation effects were observed only when temporal-priming effects were absent. In the modality domain, modality-priming effects predominated for auditory targets whereas modality-expectation effects predominated for visual targets. Thus, the interactions between probability-driven expectation and stimulus-driven priming processes appear to be controlled separately for the mechanisms that direct attention to specific temporal intervals and for the mechanisms that direct attention to specific sensory modalities. These results may suggest that the sensory system concurrently optimizes attentional priorities within temporal and sensory-modality domains.

Response repetitions in auditory task switching: The influence of spatial response distance and of the response-stimulus interval

In task switching studies, response repetition effects are typically obtained: When the task repeats, response repetitions are faster than response switches (response repetition benefit), but when the task switches, the opposite is found (response repetition cost). Previously, it was found that spatial response distance [RD] affected the response repetitions: separated response keys led to longer reaction times [RT] for response repetitions (in both task repetitions and task switches) than adjacent response keys. The goal of the present study was to replicate this RD effect in a modified setup with auditory stimuli (in Experiments 1 and 2). As we were interested in the temporal dynamics of the RD effect, we also introduced a block-wise manipulation of response-stimulus interval (RSI) in Experiment 2. RD modulated responding, replicating the results of a prior study that used visual stimuli, but only when the RSI was long. With short RSI, the RD effect was not obtained. At the same time, a long RSI led to more pronounced response repetition effects in the error rates. These results imply that response inhibition from the previous trial, which is assumed to contribute to the response repetition effect and to the modulation of responding by response distance, builds up over time.

Limits of Perceived Audio-Visual Spatial Coherence as Defined by Reaction Time Measurements

The ventriloquism effect describes the phenomenon of audio and visual signals with common features, such as a voice and a talking face merging perceptually into one percept even if they are spatially misaligned. The boundaries of the fusion of spatially misaligned stimuli are of interest for the design of multimedia products to ensure a perceptually satisfactory product. They have mainly been studied using continuous judgment scales and forced-choice measurement methods. These results vary greatly between different studies. The current experiment aims to evaluate audio-visual fusion using reaction time (RT) measurements as an indirect method of measurement to overcome these great variances. A two-alternative forced-choice (2AFC) word recognition test was designed and tested with noise and multi-talker speech background distractors. Visual signals were presented centrally and audio signals were presented between 0° and 31° audio-visual offset in azimuth. RT data were analyzed separately for the underlying Simon effect and attentional effects. In the case of the attentional effects, three models were identified but no single model could explain the observed RTs for all participants so data were grouped and analyzed accordingly. The results show that significant differences in RTs are measured from 5° to 10° onwards for the Simon effect. The attentional effect varied at the same audio-visual offset for two out of the three defined participant groups. In contrast with the prior research, these results suggest that, even for speech signals, small audio-visual offsets influence spatial integration subconsciously.

Visual Dominance Effect upon Passing the Central Bottleneck of Information Processing

BACKGROUND

In the classical psychological refractory period (PRP) paradigm, two stimuli are presented in brief succession, and participants are asked to make separate speeded responses to both stimuli. Due to a central cognitive bottleneck, responses to the second stimulus are delayed, especially at short stimulus-onset asynchrony (SOA) between the two stimuli. Although the mechanisms of dual-task interference in the classical PRP paradigm have been extensively investigated, specific mechanisms underlying the cross-modal PRP paradigm are not well understood. In particular, it remains unknown whether the dominance of vision over audition manifests in the cross-modal PRP tasks. The present study aimed to investigate whether the visual dominance effect manifests in the cross-modal PRP paradigm.

METHODS

We adapted the classical PRP paradigm by manipulating the order of a visual and an auditory task: the visual task could either precede the auditory task or vice versa, at either short or long SOAs. Twenty-five healthy participants took part in Experiment 1, and thirty-three new participants took part in Experiment 2. Reaction time and accuracy data were calculated and further analyzed by repeated-measures analysis of variance.

RESULTS

The results showed that visual precedence in the Visual-Auditory condition caused larger impairments to the subsequent auditory processing than vice versa in the Auditory-Visual condition: a larger delay of second response was revealed in the Visual-Auditory condition (135 ± 10 ms) than the Auditory-Visual condition (88 ± 9 ms). This effect was found only at the short SOAs under the existence of the central bottleneck, but not at the long SOAs. Moreover, this effect occurred both when the single visual and the single auditory task were of equal difficulty in Experiment 1 and when the single auditory task was more difficult than the single visual task in Experiment 2.

CONCLUSION

Results of the two experiments suggested that the visual dominance effect occurred under the central bottleneck of cognitive processing.

How conceptual overlap and modality pairings affect task-switching and mixing costs

Manipulating the pairings of stimulus and response modalities has been shown to affect how response selection processes for distinct tasks interact. For example, Stephan and Koch (Psychol Res 75(6):491-498, 2011) found smaller performance costs when participants switched between visual-manual (VM) and auditory-vocal (AV) tasks (modality compatible; MC) compared to between visual-vocal (VV) and auditory-manual (AM) tasks (modality incompatible; MI). However, in the Stephan and Koch study, there was conceptual overlap between one set of stimuli and one set of responses. For the MC pair, these stimuli and responses belonged to the same task, whereas for the MI pair, they belonged to different tasks. To examine how conceptual overlap affected switch and mixing costs, we conducted two experiments. Experiment 1a was a near replication of Stephan and Koch in which conceptual overlap was present in the MC AV task. In contrast, Experiment 1b reduced conceptual overlap within the MC AV task and increased it in the MI VV task. In Experiment 1a, we replicated Stephan and Koch's findings: larger switch costs were observed for the MI pair; in Experiment 1b, we found numerically greater switch costs in the MC condition. In Experiment 2, we reduced conceptual overlap in both tasks and found no effect of modality compatibility on switch costs. However, mixing costs were primarily driven by modality compatibility, regardless of conceptual overlap. These results highlight the different roles that conceptual overlap and modality pairings have on switch and mixing costs.

Emerging features of modality mappings in task switching: modality compatibility requires variability at the level of both stimulus and response modality

The term modality compatibility refers to the similarity between the stimulus modality and the modality of response-related sensory consequences. Previous research showed evidence for modality compatibility benefits in task switching, when participants switch either between two modality compatible tasks (auditory-vocal and visual-manual) or between two modality incompatible tasks (auditory-manual and visual-vocal). However, it remained unclear whether there is also a modality compatibility benefit when participants switch between a modality compatible and an incompatible task. To this end, in Experiment 1, we kept the same design as in earlier studies, so participants had to switch either between modality compatible or modality incompatible spatial discrimination tasks, but in Experiment 2A, participants switched at the response level (manual/vocal) while we kept the stimulus modality constant across tasks, and in Experiment 2B, they switched at the stimulus level (visual/auditory) while we kept the response modality constant across tasks. We found increased switch costs in modality incompatible tasks in Experiment 1, but no such a difference between modality compatible and incompatible tasks in Experiment 2A and 2B, supporting the idea that modality incompatible tasks increase crosstalk, due to the response-based priming of the competing task, but this crosstalk is reduced if the competing task involves either the same stimulus modality or the same response modality. We conclude that a significant impact of modality compatibility in task switching requires variability at the level of both stimulus and response modality.

The variability of multisensory processes of natural stimuli in human and non-human primates in a detection task

Background

Behavioral studies in both human and animals generally converge to the dogma that multisensory integration improves reaction times (RTs) in comparison to unimodal stimulation. These multisensory effects depend on diverse conditions among which the most studied were the spatial and temporal congruences. Further, most of the studies are using relatively simple stimuli while in everyday life, we are confronted to a large variety of complex stimulations constantly changing our attentional focus over time, a modality switch that can impact on stimuli detection. In the present study, we examined the potential sources of the variability in reaction times and multisensory gains with respect to the intrinsic features of a large set of natural stimuli.

Methodology/Principle findings

Rhesus macaque monkeys and human subjects performed a simple audio-visual stimulus detection task in which a large collection of unimodal and bimodal natural stimuli with semantic specificities was presented at different saliencies. Although we were able to reproduce the well-established redundant signal effect, we failed to reveal a systematic violation of the race model which is considered to demonstrate multisensory integration. In both monkeys and human species, our study revealed a large range of multisensory gains, with negative and positive values. While modality switch has clear effects on reaction times, one of the main causes of the variability of multisensory gains appeared to be linked to the intrinsic physical parameters of the stimuli.

Conclusion/Significance

Based on the variability of multisensory benefits, our results suggest that the neuronal mechanisms responsible of the redundant effect (interactions vs. integration) are highly dependent on the stimulus complexity suggesting different implications of uni- and multisensory brain regions. Further, in a simple detection task, the semantic values of individual stimuli tend to have no significant impact on task performances, an effect which is probably present in more cognitive tasks.

Modality-specific effects on crosstalk in task switching: evidence from modality compatibility using bimodal stimulation

The present study was aimed at examining modality-specific influences in task switching. To this end, participants switched either between modality compatible tasks (auditory-vocal and visual-manual) or incompatible spatial discrimination tasks (auditory-manual and visual-vocal). In addition, auditory and visual stimuli were presented simultaneously (i.e., bimodally) in each trial, so that selective attention was required to process the task-relevant stimulus. The inclusion of bimodal stimuli enabled us to assess congruence effects as a converging measure of increased between-task interference. The tasks followed a pre-instructed sequence of double alternations (AABB), so that no explicit task cues were required. The results show that switching between two modality incompatible tasks increases both switch costs and congruence effects compared to switching between two modality compatible tasks. The finding of increased congruence effects in modality incompatible tasks supports our explanation in terms of ideomotor "backward" linkages between anticipated response effects and the stimuli that called for this response in the first place. According to this generalized ideomotor idea, the modality match between response effects and stimuli would prime selection of a response in the compatible modality. This priming would cause increased difficulties to ignore the competing stimulus and hence increases the congruence effect. Moreover, performance would be hindered when switching between modality incompatible tasks and facilitated when switching between modality compatible tasks.

The role of sensory-motor modality compatibility in language processing

Language processing requires the combination of compatible (auditory-vocal and visual-manual) or incompatible (auditory-manual and visual-vocal) sensory-motor modalities, and switching between these sensory-motor modality combinations is very common in every-day life. Sensory-motor modality compatibility is defined as the similarity of stimulus modality and the modality of response-related sensory consequences. We investigated the influence of sensory-motor modality compatibility during performing language-related cognitive operations on different linguistic levels. More specifically, we used a variant of the task-switching paradigm, in which participants had to switch between compatible or between incompatible sensory-motor modality combinations during a verbal semantic categorization (Experiment 1) or during a word-form decision (Experiment 2). The data show higher switch costs (i.e., higher reaction times and error rates in switch trials compared to repetition trials) in incompatible sensory-motor modality combinations than in compatible sensory-motor modality combinations. This was true for every language-related cognitive operation, regardless of the individual linguistic level. Taken together, the present study demonstrates that sensory-motor modality compatibility plays an important role in modality switching during language processing.

The role of crossmodal competition and dimensional overlap in crossmodal attention switching

Crossmodal selective attention was investigated in a cued task switching paradigm using bimodal visual and auditory stimulation. A cue indicated the imperative modality. Three levels of spatial S-R associations were established following perceptual (location), structural (numerical), and conceptual (verbal) set-level compatibility. In Experiment 1, participants switched attention between the auditory and visual modality either with a spatial-location or spatial-numerical stimulus set. In the spatial-location set, participants performed a localization judgment on left vs. right presented stimuli, whereas the spatial-numerical set required a magnitude judgment about a visually or auditorily presented number word. Single-modality blocks with unimodal stimuli were included as a control condition. In Experiment 2, the spatial-numerical stimulus set was replaced by a spatial-verbal stimulus set using direction words (e.g., "left"). RT data showed modality switch costs, which were asymmetric across modalities in the spatial-numerical and spatial-verbal stimulus set (i.e., larger for auditory than for visual stimuli), and congruency effects, which were asymmetric primarily in the spatial-location stimulus set (i.e., larger for auditory than for visual stimuli). This pattern of effects suggests task-dependent visual dominance.

Crossmodal attention switching: auditory dominance in temporal discrimination tasks

Visual stimuli are often processed more efficiently than accompanying stimuli in another modality. In line with this "visual dominance", earlier studies on attentional switching showed a clear benefit for visual stimuli in a bimodal visual-auditory modality-switch paradigm that required spatial stimulus localization in the relevant modality. The present study aimed to examine the generality of this visual dominance effect. The modality appropriateness hypothesis proposes that stimuli in different modalities are differentially effectively processed depending on the task dimension, so that processing of visual stimuli is favored in the dimension of space, whereas processing auditory stimuli is favored in the dimension of time. In the present study, we examined this proposition by using a temporal duration judgment in a bimodal visual-auditory switching paradigm. Two experiments demonstrated that crossmodal interference (i.e., temporal stimulus congruence) was larger for visual stimuli than for auditory stimuli, suggesting auditory dominance when performing temporal judgment tasks. However, attention switch costs were larger for the auditory modality than for visual modality, indicating a dissociation of the mechanisms underlying crossmodal competition in stimulus processing and modality-specific biasing of attentional set.

Efficiency of lexical access in children with autism spectrum disorders: does modality matter?

The provision of visual support to individuals with an autism spectrum disorder (ASD) is widely recommended. We explored one mechanism underlying the use of visual supports: efficiency of language processing. Two groups of children, one with and one without an ASD, participated. The groups had comparable oral and written language skills and nonverbal cognitive abilities. In two semantic priming experiments, prime modality and prime-target relatedness were manipulated. Response time and accuracy of lexical decisions on the spoken word targets were measured. In the first uni-modal experiment, both groups demonstrated significant priming effects. In the second experiment which was cross-modal, no effect for relatedness or group was found. This result is considered in the light of the attentional capacity required for access to the lexicon via written stimuli within the developing semantic system. These preliminary findings are also considered with respect to the use of visual support for children with ASD.

Aging and response interference across sensory modalities

Advancing age is associated with decrements in selective attention. It was recently hypothesized that age-related differences in selective attention depend on sensory modality. The goal of the present study was to investigate the role of sensory modality in age-related vulnerability to distraction, using a response interference task. To this end, 16 younger (mean age = 23.1 years) and 24 older (mean age = 65.3 years) adults performed four response interference tasks, involving all combinations of visual and auditory targets and distractors. The results showed that response interference effects differ across sensory modalities, but not across age groups. These results indicate that sensory modality plays an important role in vulnerability to distraction, but not in age-related distractibility by irrelevant spatial information.

Task Switching, Modality Compatibility, and the Supra-Modal Function of Eye Movements

Previous research suggested that specific pairings of stimulus and response modalities (visual-manual and auditory-vocal tasks) lead to better dual-task performance than other pairings (visual-vocal and auditory-manual tasks). In the present task-switching study, we further examined this modality compatibility effect and investigated the role of response modality by additionally studying oculomotor responses as an alternative to manual responses. Interestingly, the switch cost pattern revealed a much stronger modality compatibility effect for groups in which vocal and manual responses were combined as compared to a group involving vocal and oculomotor responses, where the modality compatibility effect was largely abolished. We suggest that in the vocal-manual response groups the modality compatibility effect is based on cross-talk of central processing codes due to preferred stimulus-response modality processing pathways, whereas the oculomotor response modality may be shielded against cross-talk due to the supra-modal functional importance of visual orientation.

Do bodily expressions compete with facial expressions? Time course of integration of emotional signals from the face and the body

The decoding of social signals from nonverbal cues plays a vital role in the social interactions of socially gregarious animals such as humans. Because nonverbal emotional signals from the face and body are normally seen together, it is important to investigate the mechanism underlying the integration of emotional signals from these two sources. We conducted a study in which the time course of the integration of facial and bodily expressions was examined via analysis of event-related potentials (ERPs) while the focus of attention was manipulated. Distinctive integrating features were found during multiple stages of processing. In the first stage, threatening information from the body was extracted automatically and rapidly, as evidenced by enhanced P1 amplitudes when the subjects viewed compound face-body images with fearful bodies compared with happy bodies. In the second stage, incongruency between emotional information from the face and the body was detected and captured by N2. Incongruent compound images elicited larger N2s than did congruent compound images. The focus of attention modulated the third stage of integration. When the subjects' attention was focused on the face, images with congruent emotional signals elicited larger P3s than did images with incongruent signals, suggesting more sustained attention and elaboration of congruent emotional information extracted from the face and body. On the other hand, when the subjects' attention was focused on the body, images with fearful bodies elicited larger P3s than did images with happy bodies, indicating more sustained attention and elaboration of threatening information from the body during evaluative processes.

Does visual information influence infants' movement to music?

Humans are often exposed to music beginning at birth (or even before birth), yet the study of the development of musical abilities during infancy has only recently gained momentum. The goals of the present study were to determine whether young infants (ages four to seven months) spontaneously moved rhythmically in the presence of music, and whether the presence of visual information in addition to music would increase or decrease infants' movement. While nearly all infants moved in the presence of music, very few infants demonstrated rhythmic movement. Results revealed that, when visual information was present, and particularly when infants appeared to show focused attention toward the visual information, infants moved less than when only auditory information was present. The latter result is in agreement with most studies of sensory dominance in adults in which visual stimuli are dominant over auditory stimuli.

Re-evaluating visual and auditory dominance through modality switching costs and congruency analyses

Competition between the senses can lead to modality dominance, where one sense influences multi-modal processing to a greater degree than another. Modality dominance can be influenced by task demands, speeds of processing, contextual influence and practice. To resolve previous discrepancies in these factors, we assessed modality dominance in an audio-visual paradigm controlling for the first three factors while manipulating the fourth. Following a uni-modal task in which auditory and visual processing were equated, participants completed a pre-practice selective attention bimodal task in which the congruency relationship and task-relevant modality changed across trials. Participants were given practice in one modality prior to completing a post-practice selective attention bimodal task similar to the first. The effects of practice were non-specific as participants were speeded post-practice relative to pre-practice. Congruent stimuli relative to incongruent stimuli, also led to increased processing efficiency. RT data tended to reveal symmetric modality switching costs whereas the error rate data tended to reveal asymmetric modality switching costs in which switching from auditory to visual processing was particularly costly. The data suggest that when a number of safeguards are put in place to equate auditory and visual responding as far as possible, evidence for an auditory advantage can arise.

The role of input-output modality compatibility in task switching

Input-output (I-O) modality compatibility refers to the similarity of stimulus modality and modality of response-related sensory consequences. A previous study found higher switch costs in task switching in I-O modality incompatible tasks (auditory-manual and visual-vocal) than in I-O modality compatible tasks (auditory-vocal and visual-manual). However, these tasks had spatially compatible S-R mappings, which implied dimensional overlap (DO). DO may have led to automatic activation of the corresponding compatible responses in the incorrect response modality, thus increasing interference effects. The present study was aimed to examine the influence of DO on I-O modality compatibility effects. In two experiments, we found that I-O modality compatibility affects task switching even in tasks without DO, which even tended to result in further increased modality influences. This finding suggests that I-O modality mappings affect response selection by affecting between-task cross-talk not on the level of specific response codes but on the level of modality-specific processing pathways.

Are there control processes, and (if so) can they be studied?

Generally, so-called control processes are thought to be necessary when we must perform one out of several competing actions. Some examples include performance of a less well-practiced action instead of a well-practiced one (prepotency); learning a new action (novelty); and rapidly switching from one action to another (task-switching). While it certainly is difficult to perform the desired action in these circumstances, it is less clear that a separate set of processes (e.g., control processes) are necessary to explain the observed behavior. Another way to approach the study of control processes is to investigate physiological dependent measures (e.g., electrophysiological or neuroimaging measures). Although these offer another avenue of inquiry into control processes, they have yet to furnish unambiguous evidence that control processes exist. While this might suggest that there are no control processes, it is also possible that our methods are insufficiently sensitive to measure control processes. We have investigated this latter possibility using tasks that are neuroanatomically distinct, though within the same modality (vision). This approach did not yield evidence for a separable set of control processes. However, recent works using a task-switching paradigm in which subjects switch between a visual and an auditory task suggest that switching both task and modality may be importantly different than switching task within a given modality. This may represent a way forward in the study of control processes.

The role of preparation and cue-modality in crossmodal task switching

The present study examined the effects of cue-based preparation and cue-target modality mapping in crossmodal task switching. In two experiments, we randomly presented lateralized visual and auditory stimuli simultaneously. Subjects were asked to make a left/right judgment for a stimulus in only one of the modalities. Prior to each trial, the relevant stimulus modality was indicated by a visual or auditory cue. The cueing interval was manipulated to examine preparation. In Experiment 1, we used a corresponding mapping of cue-modality and stimulus modality, whereas in Experiment 2 the mapping of cue and stimulus modalities was reversed. We found reduced modality-switch costs with a long cueing interval, showing that attention shifts to stimulus modalities can be prepared, irrespective of cue-target modality mapping. We conclude that perceptual processing in crossmodal switching can be biased in a preparatory way towards task-relevant stimulus modalities.