Crossmodal action selection: Evidence from dual-task compatibility

Cognitive control and meta-control in dual-task coordination

When two tasks are presented simultaneously or in close succession, such as in the overlapping task paradigm of the psychological refractory period, dual-task performance on those tasks is usually impaired compared with separate single-task performance. Numerous theories explain these emerging dual-task costs in terms of the existence of capacity limitations in the constituent component tasks. The current paper proposes active dual-task coordination processes that work on the scheduling of these capacity-limited processes. Further, there are recent findings that point to a meta-cognitive control level in addition to these active coordination processes. This additional level's responsibility is to adjust the dual-task coordination of capacity-limited stages (i.e., coordination adjustment). I review evidence focusing on the existence of dual-task coordination processes and processes of coordination adjustment. The remainder of the paper elaborates on preliminary findings and points to the separability of these sets of processes, which is a key assumption of the framework of dual-task coordination adjustment.

Dual-action benefits: global (action-inherent) and local (transient) sources of action prepotency underlying inhibition failures in multiple action control

Previous research has shown that the simultaneous execution of two actions (instead of only one) is not necessarily more difficult but can actually be easier (less error-prone), in particular when executing one action requires the simultaneous inhibition of another action. Corresponding inhibitory demands are particularly challenging when the to-be-inhibited action is highly prepotent (i.e., characterized by a strong urge to be executed). Here, we study a range of important potential sources of such prepotency. Building on a previously established paradigm to elicit dual-action benefits, participants responded to stimuli with single actions (either manual button press or saccade) or dual actions (button press and saccade). Crucially, we compared blocks in which these response demands were randomly intermixed (mixed blocks) with pure blocks involving only one type of response demand. The results highlight the impact of global (action-inherent) sources of action prepotency, as reflected in more pronounced inhibitory failures in saccade vs. manual control, but also more local (transient) sources of influence, as reflected in a greater probability of inhibition failures following trials that required the to-be-inhibited type of action. In addition, sequential analyses revealed that inhibitory control (including its failure) is exerted at the level of response modality representations, not at the level of fully specified response representations. In sum, the study highlights important preconditions and mechanisms underlying the observation of dual-action benefits.

Response-code conflict in dual-task interference and its modulation by age

Difficulties in performing two tasks at once can arise from several sources and usually increase in advanced age. Tasks with concurrent bimodal (e.g., manual and oculomotor) responding to single stimuli consistently revealed crosstalk between conflicting response codes as a relevant source. However, how this finding translates to unimodal (i.e., manual only) response settings and how it is affected by age remains open. To address this issue, we had young and older adults respond to high- or low-pitched tones with one (single task) or both hands concurrently (dual task). Responses were either compatible or incompatible with the pitch. When responses with the same level of compatibility were combined in dual-task conditions, their response codes were congruent to each other, whereas combining a compatible and an incompatible response created mutually incongruent (i.e., conflicting) response codes, potentially inducing detrimental crosstalk. Across age groups, dual-task costs indeed were overall highest with response-code incongruency. In these trials, compatible responses exhibited higher costs than incompatible ones, even after removing trials with strongly synchronized responses. This underadditive cost asymmetry argues against mutual crosstalk as the sole source of interference and corroborates notions of strategic prioritization of limited processing capacity based on mapping-selection difficulty. As expected, the effects of incongruent response codes were found to be especially deleterious in older adults, supporting assumptions of age-related deficits in multiple-action control at the level of task-shielding. Overall, our results suggest that aging is linked to higher response confusability and less efficient flexibility for capacity sharing in dual-task settings.

Response modalities and the cognitive architecture underlying action control: Intra-modal trumps cross-modal action coordination

Performing two actions at the same time usually hampers performance. Previous studies have demonstrated a strong impact of the particular effector systems on performance in multiple action control situations. However, an open question is whether performance is generally better or worse in situations in which two actions within the same effector system are coordinated (intra-modal actions: e.g., two pedal or two manual actions) compared to situations requiring two different effector systems (cross-modal actions: e.g., a manual combined with a vocal action). Performance differences can be predicated, among others, in the light of encapsulation accounts. Encapsulation of modules on the output side of processing would suggest that actions in two different modules can be triggered simultaneously without significant interference between the actions. Thus, cross-modal actions should lead to better performance compared to intra-modal actions. We investigated this issue in two basic experiments, in which participants responded to a single stimulus (thereby maximizing control over input and central processing stages) with one or two either intra-modal or cross-modal responses (manual-manual vs. manual-oculomotor/manual-vocal in Experiment 1/2, respectively). The results represent clear evidence for a performance advantage of intra-modal over cross-modal action control across both effector system combinations and independent of the particular spatial compatibility relation between responses. The results suggest performance benefits by taking advantage of integrated, holistic representations of intra-modal action compounds.

Dissociating stimulus-response compatibility and modality compatibility in task switching

Modality compatibility (MC) describes the similarity between the modality of the stimulus and the modality of the anticipated response effect (e.g., auditory effects when speaking). Switching between two incompatible modality mappings (visual-vocal and auditory-manual) typically leads to larger costs than switching between two compatible modality mappings (visual-manual and auditory-vocal). However, it is unclear whether the influence of MC arises before or after task selection or response selection, or affects both processes. We investigated this issue by introducing a factor known to influence response selection, stimulus-response (S-R) compatibility, examining possible interactions with MC. In Experiment 1, stimulus location was task-irrelevant; participants responded manually or vocally to the meaning of visual and auditory colour words presented left or right (Simon task). In Experiment 2, stimulus location was task-relevant; participants responded manually or vocally, indicating the location (left or right) of visual or auditory stimuli, using a spatially compatible versus incompatible mapping rule (“element-level” S-R compatibility). Results revealed independent effects of S-R and modality compatibility in both experiments (n = 40 per experiment). Bayes factors suggested moderate but consistent evidence for the absence of an interaction. Independent effects suggest MC effects arise either before or after response selection, or possibly both. We propose that motor response initiation is associated with anticipatory activation of modality-specific sensory effects (e.g., auditory effects when speaking), which in turn facilitates the correct response in case of modality-compatible mappings (e.g., auditory-vocal) or reactivates, at the task-selection level, the incorrect task in case of modality-incompatible mappings (e.g., visual-vocal).

The role of working memory for task-order coordination in dual-task situations

Dual-task (DT) situations require task-order coordination processes that schedule the processing of two temporally overlapping tasks. Theories on task-order coordination suggest that these processes rely on order representations that are actively maintained and processed in working memory (WM). Preliminary evidence for this assumption stems from DT situations with variable task order, where repeating task order relative to the preceding trials results in improved performance compared to changing task order, indicating the processing of task-order information in WM between two succeeding trials. We directly tested this assumption by varying WM load during a DT with variable task order. In Experiment 1, WM load was manipulated by varying the number of stimulus–response mappings of the component tasks. In Experiment 2A, WM load was increased by embedding an additional WM updating task in the applied DT. In both experiments, the performance benefit for trials with repeated relative to trials with changed task order was reduced under high compared to low WM load. These results confirm our assumption that the processing of the task-order information relies on WM resources. In Experiment 2B, we tested whether the results of Experiment 2A can be attributed to introducing an additional task per se rather than to increased WM load by introducing an additional task with a low WM load. Importantly, in this experiment, the processing of order information was not affected. In sum, the results of the three experiments indicate that task-order coordination relies on order information which is maintained in an accessible state in WM during DT processing.

Cognitive-Postural Multitasking Training in Older Adults - Effects of Input-Output Modality Mappings on Cognitive Performance and Postural Control

Older adults exhibit impaired cognitive and balance performance, particularly under multi-task conditions, which can be improved through training. Compatibility of modality mappings in cognitive tasks (i.e., match between stimulus modality and anticipated sensory effects of motor responses), modulates physical and cognitive dual-task costs. However, the effects of modality specific training programs have not been evaluated yet. Here, we tested the effects of cognitive-postural multi-tasking training on the ability to coordinate task mappings under high postural demands in healthy older adults. Twenty-one adults aged 65-85 years were assigned to one of two groups. While group 1 performed cognitive-postural triple-task training with compatible modality mappings (i.e., visual-manual and auditory-vocal dual n-back tasks), group 2 performed the same tasks with incompatible modality mappings (i.e., visual-vocal and auditory-manual n-back tasks). Throughout the 6-weeks balance training intervention, working-memory load was gradually increased while base-of-support was reduced. Before training (T0), after a 6-week passive control period (T1), and immediately after the intervention (T2), participants performed spatial dual one-back tasks in semi-tandem stance position. Our results indicate improved working-memory performance and reduced dual-task costs for both groups after the passive control period, but no training-specific performance gains. Furthermore, balance performance did not improve in response to training. Notably, the cohort demonstrated meaningful interindividual variability in training responses. Our findings raise questions about practice effects and age-related heterogeneity of training responses following cognitive-motor training. Following multi-modal balance training, neither compatible nor incompatible modality mappings had an impact on the observed outcomes.

Dual-Task Interference in a Simulated Driving Environment: Serial or Parallel Processing?

When humans are required to perform two or more tasks concurrently, their performance declines as the tasks get closer together in time. Here, we investigated the mechanisms of this cognitive performance decline using a dual-task paradigm in a simulated driving environment, and using drift-diffusion modeling, examined if the two tasks are processed in a serial or a parallel manner. Participants performed a lane change task, along with an image discrimination task. We systematically varied the time difference between the onset of the two tasks (Stimulus Onset Asynchrony, SOA) and measured its effect on the amount of dual-task interference. Results showed that the reaction times (RTs) of the two tasks in the dual-task condition were higher than those in the single-task condition. SOA influenced the RTs of both tasks when they were presented second and the RTs of the image discrimination task when it was presented first. Results of drift-diffusion modeling indicated that dual-task performance affects both the rate of evidence accumulation and the delays outside the evidence accumulation period. These results suggest that a hybrid model containing features of both parallel and serial processing best accounts for the results. Next, manipulating the predictability of the order of the two tasks, we showed that in unpredictable conditions, the order of the response to the two tasks changes, causing attenuation in the effect of SOA. Together, our findings suggest higher-level executive functions are involved in managing the resources and controlling the processing of the tasks during dual-task performance in naturalistic settings.

Visual and central attention share a capacity limitation when the demands for serial item selection in visual search are high

Visual and central attention are limited in capacity. In conjunction search, visual attention is required to select the items and to bind their features (e.g., color, form, size), which results in a serial search process. In dual-tasks, central attention is required for response selection, but because central attention is limited in capacity, response selection can only be carried out for one task at a time. Here, we investigated whether visual and central attention rely on a common or on distinct capacity limitations. In two dual-task experiments, participants completed an auditory two-choice discrimination Task 1 and a conjunction search Task 2 that were presented with an experimentally modulated temporal interval between them (stimulus onset asynchrony [SOA]). In Experiment 1, Task 2 was a triple conjunction search task. Each item consisted of a conjunction of three features, so that target and distractors shared two features. In Experiment 2, Task 2 was a plus conjunction search task, in which target and distractors shared the same four features. The hypotheses for conjunction search time were derived from the locus-of-slack method. While plus conjunction search was performed after response selection in Task 1, a small part of triple conjunction search was still performed in parallel to response selection in Task 1. However, the between-experiment comparison was not significant, indicating that both search tasks may require central attention. Taken together, the present study provides evidence that visual and central attention share a common capacity limitation when conjunction search relies strongly on serial item selection.

Two sources of task prioritization: The interplay of effector-based and task order-based capacity allocation in the PRP paradigm

When processing of two tasks overlaps, performance is known to suffer. In the well-established psychological refractory period (PRP) paradigm, tasks are triggered by two stimuli with a short temporal delay (stimulus onset asynchrony; SOA), thereby allowing control of the degree of task overlap. A decrease of the SOA reliably yields longer RTs of the task associated with the second stimulus (Task 2) while performance in the other task (Task 1) remains largely unaffected. This Task 2-specific SOA effect is usually interpreted in terms of central capacity limitations. Particularly, it has been assumed that response selection in Task 2 is delayed due to the allocation of less capacity until this process has been completed in Task 1. Recently, another important factor determining task prioritization has been proposed—namely, the particular effector systems associated with tasks. Here, we study both sources of task prioritization simultaneously by systematically combining three different effector systems (pairwise combinations of oculomotor, vocal, and manual responses) in the PRP paradigm. Specifically, we asked whether task order-based task prioritization (SOA effect) is modulated as a function of Task 2 effector system. The results indicate a modulation of SOA effects when the same (oculomotor) Task 1 is combined with a vocal versus a manual Task 2. This is incompatible with the assumption that SOA effects are solely determined by Task 1 response selection duration. Instead, they support the view that dual-task processing bottlenecks are resolved by establishing a capacity allocation scheme fed by multiple input factors, including attentional weights associated with particular effector systems.

Sequential modulation of across-task congruency in the serial reaction time task

While detrimental effects of dual-tasking on the acquisition and usage of sequence knowledge in the Serial Reaction Time Task (SRTT) have been attributed to the integration of regularly and randomly sequenced events, direct evidence for across-task integration has been scarce. In two experiments, we paired two spatial tasks: A visual-manual SRTT (serial reaction time task) of length 4 and a two-choice task with random order of stimuli. We reasoned that across-task integration might result in SRTT- and two-choice task events being stored and retrieved together. Therefore, RT might be influenced by current congruency as well as by whether it is repeated or altered from the congruency level encountered for this SRTT event in the last loop of the sequence. We observed such a modulation in two experiments, suggesting that across-task integration indeed takes place and that the detrimental effect of dual-tasking on sequence learning is, at least in parts, based on across-task integration.

Cross-modal psychological refractory period in vision, audition, and haptics

People's parallel-processing ability is limited, as demonstrated by the psychological refractory period (PRP) effect: The reaction time to the second stimulus (RT2) increases as the stimulus onset asynchrony (SOA) between two stimuli decreases. Most theoretical models of PRP are independent of modalities. Previous research on PRP mainly focused on vision and audition as input modalities; tactile stimuli have not been fully explored. Research using other paradigms and involving tactile stimuli, however, found that dual-task performance depended on input modalities. This study explored PRP with all the combinations of input modalities. Thirty participants judged the magnitude (small or large) of two stimuli presented in different modalities with an SOA of 75-1,200 ms. PRP effect was observed, i.e., RT2 increased with a decreasing SOA, in all the modalities. Only in the auditory-tactile condition did the accuracy of Task 2 decrease with a decreasing SOA. In the auditory-tactile and tactile-visual conditions, RT to the first stimulus also increased with a decreasing SOA. Current models could only explain part of the results, and modality characteristics help to explain the overall data pattern better. Limitations and directions for future studies regarding reaction time, task difficulty, and response modalities are discussed.

More insight into the interplay of response selection and visual attention in dual-tasks: masked visual search and response selection are performed in parallel

Both response selection and visual attention are limited in capacity. According to the central bottleneck model, the response selection processes of two tasks in a dual-task situation are performed sequentially. In conjunction search, visual attention is required to select the items and to bind their features (e.g., color and form), which results in a serial search process. Search time increases as items are added to the search display (i.e., set size effect). When the search display is masked, visual attention deployment is restricted to a brief period of time and target detection decreases as a function of set size. Here, we investigated whether response selection and visual attention (i.e., feature binding) rely on a common or on distinct capacity limitations. In four dual-task experiments, participants completed an auditory Task 1 and a conjunction search Task 2 that were presented with an experimentally modulated temporal interval between them (Stimulus Onset Asynchrony, SOA). In Experiment 1, Task 1 was a two-choice discrimination task and the conjunction search display was not masked. In Experiment 2, the response selection difficulty in Task 1 was increased to a four-choice discrimination and the search task was the same as in Experiment 1. We applied the locus-of-slack method in both experiments to analyze conjunction search time, that is, we compared the set size effects across SOAs. Similar set size effects across SOAs (i.e., additive effects of SOA and set size) would indicate sequential processing of response selection and visual attention. However, a significantly smaller set size effect at short SOA compared to long SOA (i.e., underadditive interaction of SOA and set size) would indicate parallel processing of response selection and visual attention. In both experiments, we found underadditive interactions of SOA and set size. In Experiments 3 and 4, the conjunction search display in Task 2 was masked. Task 1 was the same as in Experiments 1 and 2, respectively. In both experiments, the d' analysis revealed that response selection did not affect target detection. Overall, Experiments 1-4 indicated that neither the response selection difficulty in the auditory Task 1 (i.e., two-choice vs. four-choice) nor the type of presentation of the search display in Task 2 (i.e., not masked vs. masked) impaired parallel processing of response selection and conjunction search. We concluded that in general, response selection and visual attention (i.e., feature binding) rely on distinct capacity limitations.

Cross-modal Action Complexity: Action- and Rule-related Memory Retrieval in Dual-response Control

Normally, we do not act within a single effector system only, but rather coordinate actions across several output modules (cross-modal action). Such cross-modal action demands can vary substantially with respect to their complexity in terms of the number of task-relevant response combinations and to-be-retrieved stimulus–response (S–R) mapping rules. In the present study, we study the impact of these two types of cross-modal action complexity on dual-response costs (i.e., performance differences between single- and dual-action demands). In Experiment 1, we combined a manual and an oculomotor task, each involving four response alternatives. Crucially, one (unconstrained) condition involved all 16 possible combinations of response alternatives, whereas a constrained condition involved only a subset of possible response combinations. The results revealed that preparing for a larger number of response combinations yielded a significant, but moderate increase in dual-response costs. In Experiment 2, we utilized one common lateralized auditory (e.g., left) stimulus to trigger incompatible response compounds (e.g., left saccade and right key press or vice versa). While one condition only involved one set of task-relevant S–R rules, another condition involved two sets of task-relevant rules (coded by stimulus type: noise/tone), while the number of task-relevant response combinations was the same in both conditions. Here, an increase in the number of to-be-retrieved S–R rules was associated with a substantial increase in dual-response costs that were also modulated on a trial-by-trial basis when switching between rules. Taken together, the results shed further light on the dependency of cross-modal action control on both action- and rule-related memory retrieval processes.

Are there differences in the dual-task walking variability of minimum toe clearance in chronic low back pain patients and healthy controls?

The control of minimum toe clearance (MTC, as quantified with its stride-to-stride variability during walking) is a promising marker to evaluate motor control. The control of MTC, compared to other gait parameters, was reported to have higher priority. The relationship between the control of MTC and other gait parameters should be examined to elucidate tripping mechanisms. This study aimed at investigating the variability of MTC, stride time and stride length in normal walking and in dual-task walking in back pain sufferers. Twelve patients with chronic low back pain and twelve healthy controls walked with inertial sensors attached on their feet with and without a cognitive dual task. Standard deviations of stride time, stride length and MTC were calculated. Regarding the comparison of dual-task walking in pain patients vs. controls, we found higher variability in stride time in the back-pain group. Higher dual-task walking variability was observed in stride length and stride time only in back pain sufferers. Regarding MTC, however, neither a difference between groups nor between walking conditions were found. We observed that individuals with pain, who generally show higher gait variability, are able to control MTC in a dual-task condition indicating that their central nervous system might prioritize control of MTC over other gait parameters. Cases in which also MTC variability increase because of a dual task might characterize alarming fall risk. Dual-task MTC variability should, therefore, be estimated in individuals with severe fall risk as in old individuals with pain, frail people or neurological patients.

When two actions are easier than one: how inhibitory control demands affect response processing

Numerous studies showed that the simultaneous execution of multiple actions is associated with performance costs. Here, we demonstrate that when highly automatic responses are involved, performance in single-response conditions can actually be worse than in dual-response conditions. Participants responded to peripheral visual stimuli with an eye movement (saccade), a manual key press, or both. To manipulate saccade automaticity, a central fixation cross either remained present throughout the trial (overlap condition, lower automaticity) or disappeared 200 ms before visual target onset (gap condition, greater automaticity). Crucially, single-response conditions yielded more performance errors than dual-response conditions (i.e., dual-response benefit), especially in gap trials. This was due to difficulties associated with inhibiting saccades when only manual responses were required, suggesting that response inhibition (remaining fixated) can be even more resource-demanding than overt response execution (saccade to peripheral target).

Task Switching, Modality Compatibility, and the Supra-Modal Function of Eye Movements

Previous research suggested that specific pairings of stimulus and response modalities (visual-manual and auditory-vocal tasks) lead to better dual-task performance than other pairings (visual-vocal and auditory-manual tasks). In the present task-switching study, we further examined this modality compatibility effect and investigated the role of response modality by additionally studying oculomotor responses as an alternative to manual responses. Interestingly, the switch cost pattern revealed a much stronger modality compatibility effect for groups in which vocal and manual responses were combined as compared to a group involving vocal and oculomotor responses, where the modality compatibility effect was largely abolished. We suggest that in the vocal-manual response groups the modality compatibility effect is based on cross-talk of central processing codes due to preferred stimulus-response modality processing pathways, whereas the oculomotor response modality may be shielded against cross-talk due to the supra-modal functional importance of visual orientation.

Oculomotor interference during manual response preparation: evidence from the response-cueing paradigm

Preparation provided by visual location cues is known to speed up behavior. However, the role of concurrent saccades in response to visual cues remains unclear. In this study, participants performed a spatial precueing task by pressing one of four response keys with one of four fingers (two of each hand) while eye movements were monitored. Prior to the stimulus, we presented a neutral cue (baseline), a hand cue (corresponding to left vs. right positions), or a finger cue (corresponding to inner vs. outer positions). Participants either remained fixated on a central fixation point or moved their eyes freely. The results demonstrated that saccades during the cueing interval altered the pattern of cueing effects. Finger cueing trials in which saccades were spatially incompatible (vs. compatible) with the subsequently required manual response exhibited slower manual RTs. We propose that interference between saccades and manual responses affects manual motor preparation.

Interference effects of stimulus-response modality pairings in dual tasks and their robustness

Recent evidence suggests that the degree of interference in dual-task situations depends crucially on the pairings of input- and output modalities of the two component tasks with increased dual-task costs for modality incompatible (i.e., visual-vocal and auditory-manual) compared to modality compatible (i.e., visual-manual and auditory-vocal) dual tasks. These effects of modality pairings in dual tasks have been related to the overlap of non-preferred processing pathways in modality incompatible tasks. Until now, modality compatibility has not yet been related to other sources of interference in a dual-task context, such as stimulus-response (S-R) compatibility or crosstalk. In the present study, we conducted two experiments using the paradigm of the psychological refractory period (PRP) to test the effects of S-R compatibility and crosstalk on the effects of modality compatibility in temporally overlapping task situations. Experiment 1 revealed an overadditive interaction between stimulus onset asynchrony and modality compatibility for tasks with S-R compatible mappings, indicating that modality compatibility effects are present in different task situations, even when S-R mappings are otherwise compatible. In Experiment 2, we aimed at pinpointing the boundaries of the effects of modality compatibility in dual-task situations. We showed that additional sources of dual-task interference in a modality compatible dual task could overwrite the pronounced PRP effect previously shown for modality incompatible tasks. Taken together, these data provide new evidence that the specific types of stimulus-response modality pairings are an additional factor that might interact with other sources of interference in dual-task situations.

Searching working memory for the source of dual-task costs

Dual-task costs depend on the specific pairings of stimulus and response modalities. Such findings are analogous to domain-specific effects in the working memory (WM) literature, in which items compete for limited capacity when they involve related types of information. The present study explicitly examines the relationship between modality-pairing effects on dual-task costs and domain-specificity effects on WM capacity. Participants maintained a sequence of either locations or tones in WM, and then performed a choice reaction time task in which they responded either vocally or manually. The stimuli for the choice reaction time task were held constant, but its response modality affected the interference observed in WM: vocal responses interfered with WM for tones and manual responses interfered with WM for locations. These findings indicate that response selection engages domain-specific WM processes and that interference within these processes may account for modality-pairing effects.