Modality-specific effects on crosstalk in task switching: evidence from modality compatibility using bimodal stimulation

Voluntary task switching is affected by modality compatibility and preparation

Cognitive task control can be examined in task-switching studies. Performance costs in task switches are usually smaller with compatible stimulus-response modality mappings (visual-manual and auditory-vocal) than with incompatible mappings (visual-vocal and auditory-manual). Modality compatibility describes the modality match of sensory input and of the anticipated response effect (e.g., vocal responses produce auditory effects, so that auditory stimuli are modality-compatible with vocal responses). Fintor et al. (Psychological Research, 84(2), 380-388, 2020) found that modality compatibility also biased task choice rates in voluntary task switching (VTS). In that study, in each trial participants were presented with a visual or auditory spatial stimulus and were free to choose the response modality (manual vs. vocal). In this free-choice task, participants showed a bias to create more modality-compatible than -incompatible mappings. In the present study, we assessed the generality of Fintor et al.'s (2020) findings, using verbal rather than spatial stimuli, and more complex tasks, featuring an increased number of stimulus-response alternatives. Experiment 1 replicated the task-choice bias to preferentially create modality-compatible mappings. We also found a bias to repeat the response modality just performed, and a bias to repeat entire stimulus-response modality mappings. In Experiment 2, we manipulated the response-stimulus interval (RSI) to examine whether more time for proactive cognitive control would help resolve modality-specific crosstalk in this free-choice paradigm. Long RSIs led to a decreased response-modality repetition bias and mapping repetition bias, but the modality-compatibility bias was unaffected. Together, the findings suggest that modality-specific priming of response modality influences task choice.

Exploring behavioral adjustments of proportion congruency manipulations in an Eriksen flanker task with visual and auditory distractor modalities

The present study investigated global behavioral adaptation effects to conflict arising from different distractor modalities. Three experiments were conducted using an Eriksen flanker paradigm with constant visual targets, but randomly varying auditory or visual distractors. In Experiment 1, the proportion of congruent to incongruent trials was varied for both distractor modalities, whereas in Experiments 2A and 2B, this proportion congruency (PC) manipulation was applied to trials with one distractor modality (inducer) to test potential behavioral transfer effects to trials with the other distractor modality (diagnostic). In all experiments, mean proportion congruency effects (PCEs) were present in trials with a PC manipulation, but there was no evidence of transfer to diagnostic trials in Experiments 2A and 2B. Distributional analyses (delta plots) provided further evidence for distractor modality-specific global behavioral adaptations by showing differences in the slope of delta plots with visual but not auditory distractors when increasing the ratio of congruent trials. Thus, it is suggested that distractor modalities constrain global behavioral adaptation effects due to the learning of modality-specific memory traces (e.g., distractor-target associations) and/or the modality-specific cognitive control processes (e.g., suppression of modality-specific distractor-based activation). Moreover, additional analyses revealed partial transfer of the congruency sequence effect across trials with different distractor modalities suggesting that distractor modality may differentially affect local and global behavioral adaptations.

Dynamics of alpha suppression index both modality specific and general attention processes

EEG alpha power varies under many circumstances requiring visual attention. However, mounting evidence indicates that alpha may not only serve visual processing, but also the processing of stimuli presented in other sensory modalities, including hearing. We previously showed that alpha dynamics during an auditory task vary as a function of competition from the visual modality (Clements et al., 2022) suggesting that alpha may be engaged in multimodal processing. Here we assessed the impact of allocating attention to the visual or auditory modality on alpha dynamics at parietal and occipital electrodes, during the preparatory period of a cued-conflict task. In this task, bimodal precues indicated the modality (vision, hearing) relevant to a subsequent reaction stimulus, allowing us to assess alpha during modality-specific preparation and while switching between modalities. Alpha suppression following the precue occurred in all conditions, indicating that it may reflect general preparatory mechanisms. However, we observed a switch effect when preparing to attend to the auditory modality, in which greater alpha suppression was elicited when switching to the auditory modality compared to repeating. No switch effect was evident when preparing to attend to visual information (although robust suppression did occur in both conditions). In addition, waning alpha suppression preceded error trials, irrespective of sensory modality. These findings indicate that alpha can be used to monitor the level of preparatory attention to process both visual and auditory information, and support the emerging view that alpha band activity may index a general attention control mechanism used across modalities.

Short-term pre-exposure to modality mappings: Modality-incompatible single-task exposure reduces modality-specific between-task crosstalk in task-switching

Modality compatibility refers to the similarity of the stimulus modality and the modality of the sensory-response effect that the response produces (i.e., vocal responses produce auditory effects). In this study, we investigated the effect of short-term pre-exposure of modality compatibility in task-switching. To this end, participants were exposed to either modality-compatible (visual-manual and auditory-vocal) or modality-incompatible (visual-vocal and auditory-manual) single-tasks. After a short-term single-task pre-exposure (with either both modality-compatible tasks, 2 × 80 trials each, or both modality-incompatible tasks, 2 × 80 trials each), participants were transferred to a task-switching situation, where they switched between tasks in both a modality-compatible and an incompatible condition. We found that after pre-exposure to modality-compatible single-tasks the typical effect of modality compatibility was found (i.e., larger switch costs with modality-incompatible tasks compared to modality-compatible tasks). In contrast, after pre-exposed to modality-incompatible single-tasks, modality compatibility no longer influenced switch costs. We assume that long-term modality-compatible associations could be overridden by short-term, task-specific associations to reduce between-task crosstalk.

Dissociating stimulus-response compatibility and modality compatibility in task switching

Modality compatibility (MC) describes the similarity between the modality of the stimulus and the modality of the anticipated response effect (e.g., auditory effects when speaking). Switching between two incompatible modality mappings (visual-vocal and auditory-manual) typically leads to larger costs than switching between two compatible modality mappings (visual-manual and auditory-vocal). However, it is unclear whether the influence of MC arises before or after task selection or response selection, or affects both processes. We investigated this issue by introducing a factor known to influence response selection, stimulus-response (S-R) compatibility, examining possible interactions with MC. In Experiment 1, stimulus location was task-irrelevant; participants responded manually or vocally to the meaning of visual and auditory colour words presented left or right (Simon task). In Experiment 2, stimulus location was task-relevant; participants responded manually or vocally, indicating the location (left or right) of visual or auditory stimuli, using a spatially compatible versus incompatible mapping rule (“element-level” S-R compatibility). Results revealed independent effects of S-R and modality compatibility in both experiments (n = 40 per experiment). Bayes factors suggested moderate but consistent evidence for the absence of an interaction. Independent effects suggest MC effects arise either before or after response selection, or possibly both. We propose that motor response initiation is associated with anticipatory activation of modality-specific sensory effects (e.g., auditory effects when speaking), which in turn facilitates the correct response in case of modality-compatible mappings (e.g., auditory-vocal) or reactivates, at the task-selection level, the incorrect task in case of modality-incompatible mappings (e.g., visual-vocal).

Crossmodal Effects in Task Switching: Modality Compatibility with Vocal and Pedal Responses

Modality compatibility refers to the similarity between the stimulus modality and the modality of response-related sensory consequences (e.g., vocal output produces audible effects). While previous studies found higher costs of task switching with stimulus-response modality-incompatible tasks (auditory-manual and visual-vocal), the present study was aimed to explore the generality of modality compatibility by examining a new response modality (pedal responses). Experiment 1 showed that the effect of modality compatibility generalizes to pedal responses when these replaced manual responses used in previous studies (i.e., higher switch costs when switching between auditory-pedal and visual-vocal tasks compared to switching between auditory-vocal and visual-pedal tasks). However, in single-task conditions there was no influence of modality compatibility. Experiment 2 was designed to examine whether modality compatibility depends on the frequency of task switches. To this end, one task occurred very frequently, overall decreasing the task switching frequency. Importantly, the results showed a robust task-switching benefit of modality-compatible mappings even for a highly frequent task, suggesting that the sustained representation of potentially competing response modalities affects task-switching performance independent from the actual frequency of the tasks. Together, the data suggest that modality compatibility is an emergent phenomenon arising in task-switching situations based on the necessity to maintain but at the same time separate competing modality mappings, which are characterized by ideomotor ‘‘backward’’ linkages between anticipated response effects and the stimuli that called for this response in the first place.

Modality compatibility in task switching depends on processing codes and task demands

Modality compatibility denotes the match between sensory stimulus modality and the sensory modality of the anticipated response effect (for example, vocal responses usually lead to auditory effects, so that auditory-vocal stimulus-response mappings are modality-compatible, whereas visual-vocal mappings are modality incompatible). In task switching studies, it has been found that switching between two modality-incompatible mappings (auditory-manual and visual-vocal) resulted in higher switch costs than switching between two modality-compatible mappings (auditory-vocal and visual-manual). This finding suggests that with modality-incompatible mappings, the anticipation of the effect of each response primes the stimulus modality linked to the competing task, creating task confusion. In Experiment 1, we examined whether modality-compatibility effects in task switching are increased by strengthening the auditory-vocal coupling using spatial-verbal stimuli relative to spatial-location stimuli. In Experiment 2, we aimed at achieving the same goal by requiring temporal stimulus discrimination relative to spatial stimulus localisation. Results suggest that both spatial-verbal stimuli and temporal discrimination can increase modality-specific task interference through a variation of the strength of anticipation in the response-effect coupling. This provides further support for modality specificity of cognitive control processes in task switching.

Cue the effects: Stimulus-action effect modality compatibility and dual-task costs

The pairings of tasks' stimulus and response modalities affect the magnitude of dual-task costs. For example, dual-task costs are larger when a visual-vocal task is paired with an auditory-manual task compared with when a visual-manual task is paired with an auditory-vocal task. These results are often interpreted as reflecting increased crosstalk between central codes for each task. Here we examine a potential source: modality-based crosstalk between the stimuli and the response-induced sensory consequences (i.e., action effects). In five experiments, we manipulated experimentally induced action effects so that they were either modality-compatible or -incompatible with the stimuli. Action effects that were modality-compatible (e.g., visual stimulus, visual action effect) produced smaller dual-task costs than those that were modality-incompatible (e.g., visual stimulus, auditory action effect). Thus, the relationship between stimuli and action effects contributes to dual-task costs. Moreover, modality-compatible pairs showed an advantage compared with when no action effects were experimentally induced. These results add to a growing body of work demonstrating that postresponse sensory events affect response selection processes. (PsycInfo Database Record (c) 2020 APA, all rights reserved).

Modality compatibility biases voluntary choice of response modality in task switching

The term modality compatibility refers to the similarity between stimulus modality and the modality of response-related sensory consequences (e.g., vocal responses produce auditory effects). The previous results showed smaller task-switching costs when participants switched between modality compatible tasks (auditory-vocal and visual-manual) compared to switching between modality incompatible tasks (auditory-manual and visual-vocal). In the present study using a voluntary task-switching paradigm (VTS), participants chose the response modality (vocal or manual) to indicate the location of either a visual or an auditory stimulus. We examined whether free task choices were biased by modality compatibility, so that modality compatible tasks are preferred in VTS. The choice probability analysis indicated that participants tended to choose the response modality that is compatible to the stimulus modality. However, participants did not show a preference to repeat a stimulus-response (S-R) modality mapping, but to switch between modality compatibility (i.e., from S-R modality compatible mapping to S-R modality incompatible mapping and vice versa). More interestingly, even though participants freely chose the response modality, modality compatibility still influenced task-switching costs, showing larger costs with modality incompatible mappings. The finding that modality compatibility influenced choice behaviour suggests components of both top-down control and bottom-up effects of selecting a response modality for different stimulus modalities.

Sensory-motor modality compatibility in multitasking: The influence of processing codes

Sensory-motor modality compatibility is defined as the similarity between the sensory modality and the modality of response-related effects. Previous dual-task and task-switching studies have shown higher performance costs for coordinating relatively incompatible sensory-motor modality mappings (i.e., auditory-manual and visual-vocal) compared to more compatible mappings (i.e., auditory-vocal and visual-manual). Until now, however, little attention has been paid to potential variability in effects of modality compatibility depending on different processing codes. In the present study, we independently varied the processing codes of input and output (nonverbal-spatial, nonverbal-nominal, verbal-spatial, verbal-nominal) while participants switched between incompatible and compatible sensory-motor modality mappings. Beside higher switch costs for switching between incompatible sensory-motor modality mappings than for switching between compatible mappings, the results revealed stronger effects of modality compatibility on switch costs for verbal input than for nonverbal input codes. This suggests that priming mechanisms between sensory input and compatible motor output are modulated by the processing code of the sensory input. As possible explanations, we assume a higher degree of concordance with output processing codes as well as stronger associations with potential response effects for verbal than for nonverbal input.

The interplay of crossmodal attentional preparation and modality compatibility in cued task switching

Two experiments examined the influence of preparation on modality compatibility effects in task switching. The term modality compatibility refers to the similarity between the stimulus modality and the modality of response-related sensory consequences. Previous research showed evidence for modality compatibility benefits in task switching when participants switched either between two modality compatible tasks (auditory-vocal and visual-manual) or between two modality incompatible tasks (auditory-manual and visual-vocal). In this study, we investigated the influence of active preparation on modality compatibility effects in task switching. To this end, in Experiment 1, we introduced unimodal modality cues, whereas in Experiment 2, bimodal abstract cues were used. In both experiments, the cue-stimulus interval (CSI) was manipulated while holding the response-stimulus interval (RSI) constant. In both experiments, we found not only decreased switch costs with long CSI but also the elimination of the residual switch costs. More importantly, this preparation effect did not modulate the modality compatibility effect in task switching. To account for this data pattern, we assume that cue-based preparation of switches by modality mappings was highly effective and produced no residual reaction time (RT) costs with long CSI.

How conceptual overlap and modality pairings affect task-switching and mixing costs

Manipulating the pairings of stimulus and response modalities has been shown to affect how response selection processes for distinct tasks interact. For example, Stephan and Koch (Psychol Res 75(6):491-498, 2011) found smaller performance costs when participants switched between visual-manual (VM) and auditory-vocal (AV) tasks (modality compatible; MC) compared to between visual-vocal (VV) and auditory-manual (AM) tasks (modality incompatible; MI). However, in the Stephan and Koch study, there was conceptual overlap between one set of stimuli and one set of responses. For the MC pair, these stimuli and responses belonged to the same task, whereas for the MI pair, they belonged to different tasks. To examine how conceptual overlap affected switch and mixing costs, we conducted two experiments. Experiment 1a was a near replication of Stephan and Koch in which conceptual overlap was present in the MC AV task. In contrast, Experiment 1b reduced conceptual overlap within the MC AV task and increased it in the MI VV task. In Experiment 1a, we replicated Stephan and Koch's findings: larger switch costs were observed for the MI pair; in Experiment 1b, we found numerically greater switch costs in the MC condition. In Experiment 2, we reduced conceptual overlap in both tasks and found no effect of modality compatibility on switch costs. However, mixing costs were primarily driven by modality compatibility, regardless of conceptual overlap. These results highlight the different roles that conceptual overlap and modality pairings have on switch and mixing costs.

Emerging features of modality mappings in task switching: modality compatibility requires variability at the level of both stimulus and response modality

The term modality compatibility refers to the similarity between the stimulus modality and the modality of response-related sensory consequences. Previous research showed evidence for modality compatibility benefits in task switching, when participants switch either between two modality compatible tasks (auditory-vocal and visual-manual) or between two modality incompatible tasks (auditory-manual and visual-vocal). However, it remained unclear whether there is also a modality compatibility benefit when participants switch between a modality compatible and an incompatible task. To this end, in Experiment 1, we kept the same design as in earlier studies, so participants had to switch either between modality compatible or modality incompatible spatial discrimination tasks, but in Experiment 2A, participants switched at the response level (manual/vocal) while we kept the stimulus modality constant across tasks, and in Experiment 2B, they switched at the stimulus level (visual/auditory) while we kept the response modality constant across tasks. We found increased switch costs in modality incompatible tasks in Experiment 1, but no such a difference between modality compatible and incompatible tasks in Experiment 2A and 2B, supporting the idea that modality incompatible tasks increase crosstalk, due to the response-based priming of the competing task, but this crosstalk is reduced if the competing task involves either the same stimulus modality or the same response modality. We conclude that a significant impact of modality compatibility in task switching requires variability at the level of both stimulus and response modality.

Exploring Modality Compatibility in the Response-Effect Compatibility Paradigm

According to ideomotor theory, action planning is based on anticipatory perceptual representations of action-effects. This aspect of action control has been investigated in studies using the response-effect compatibility (REC) paradigm, in which responses have been shown to be facilitated if ensuing perceptual effects share codes with the response based on dimensional overlap (i.e., REC). Additionally, according to the notion of ideomotor compatibility, certain response-effect (R-E) mappings will be stronger than others because some response features resemble the anticipated sensory response effects more strongly than others (e.g., since vocal responses usually produce auditory effects, an auditory stimulus should be anticipated in a stronger manner following vocal responses rather than following manual responses). Yet, systematic research on this matter is lacking. In the present study, two REC experiments aimed to explore the influence of R-E modality mappings. In Experiment 1, vocal number word responses produced visual effects on the screen (digits vs. number words; i.e., visual-symbolic vs. visual-verbal effect codes). The REC effect was only marginally larger for visual-verbal than for visual-symbolic effects. Using verbal effect codes in Experiment 2, we found that the REC effect was larger with auditory-verbal R-E mapping than with visual-verbal R-E mapping. Overall, the findings support the hypothesis of a role of R-E modality mappings in REC effects, suggesting both further evidence for ideomotor accounts as well as code-specific and modality-specific contributions to effect anticipation.

Spatial attention across perception and action

We hypothesize that a shared spatial attention mechanism is used for both perception and action. To this end we created a new dual-task version of the classical Simon task. In one task, the spatial-input task, associated with input spatial attention, participants named one shape out of two bilaterally presented colored shapes. In a second task, the spatial-output task, associated with output spatial attention, participants discriminated between high and low pitch tones by pressing either a left or a right key. In Experiment 1, input for both tasks appeared simultaneously, and participants were instructed not to prioritize either task. A between tasks Simon-like effect was found for responses to both tasks. Reaction times were shorter when the side of the relevant shape in the spatial-input task and the side of the correct response in the spatial-output task were congruent. In Experiment 2, we manipulated the stimulus-onset asynchrony (SOA) between the inputs for the two tasks and showed that the Simon-like effect remained intact at all SOAs. Experiment 3 was similar to Experiment 1 except that the vocal response for the spatial-input task was not speeded. A Simon-like effect was still observed. Experiment 4 was the same as Experiment 3 except that the non-speeded response for the spatial-input task was manual rather than vocal. No Simon-like effect was observed in this experiment. Our results support a shared spatial attention mechanism involved in the Simon effect and indicate that this spatial attention mechanism is shared by perception and action.

Feature Integration and Task Switching: Diminished Switch Costs after Controlling for Stimulus, Response, and Cue Repetitions

This report presents data from two versions of the task switching procedure in which the separate influence of stimulus repetitions, response key repetitions, conceptual response repetitions, cue repetitions, task repetitions, and congruency are considered. Experiment 1 used a simple alternating runs procedure with parity judgments of digits and consonant/vowel decisions of letters as the two tasks. Results revealed sizable effects of stimulus and response repetitions, and controlling for these effects reduced the switch cost. Experiment 2 was a cued version of the task switch paradigm with parity and magnitude judgments of digits as the two tasks. Results again revealed large effects of stimulus and response repetitions, in addition to cue repetition effects. Controlling for these effects again reduced the switch cost. Congruency did not interact with our novel “unbiased” measure of switch costs. We discuss how the task switch paradigm might be thought of as a more complex version of the feature integration paradigm and propose an episodic learning account of the effect. We further consider to what extent appeals to higher-order control processes might be unnecessary and propose that controls for feature integration biases should be standard practice in task switching experiments.