On the effects of multimodal information integration in multitasking

Advancements in Smart Wearable Mobility Aids for Visual Impairments: A Bibliometric Narrative Review

Research into new solutions for wearable assistive devices for the visually impaired is an important area of assistive technology (AT). This plays a crucial role in improving the functionality and independence of the visually impaired, helping them to participate fully in their daily lives and in various community activities. This study presents a bibliometric analysis of the literature published over the last decade on wearable assistive devices for the visually impaired, retrieved from the Web of Science Core Collection (WoSCC) using CiteSpace, to provide an overview of the current state of research, trends, and hotspots in the field. The narrative focuses on prominent innovations in recent years related to wearable assistive devices for the visually impaired based on sensory substitution technology, describing the latest achievements in haptic and auditory feedback devices, the application of smart materials, and the growing concern about the conflicting interests of individuals and societal needs. It also summarises the current opportunities and challenges facing the field and discusses the following insights and trends: (1) optimization of the transmission of haptic and auditory information while multitasking; (2) advance research on smart materials and foster cross-disciplinary collaboration among experts; and (3) balance the interests of individuals and society. Given the two essential directions, the low-cost, stand-alone pursuit of efficiency and the high-cost pursuit of high-quality services that are closely integrated with accessible infrastructure, the latest advances will gradually allow more freedom for ambient assisted living by using robotics and automated machines, while using sensor and human-machine interaction as bridges to promote the synchronization of machine intelligence and human cognition.

Individual Differences in the Impact of Distracting Environmental Sounds on the Performance of a Continuous Visual Task in Older Adults

Background/Objectives: Vulnerability to sound distraction is commonly reported in older adults with dementia and tends to be associated with adverse impacts on daily activity. However, study outcome heterogeneity is increasingly evident, with preserved resistance to distraction also evident. Contributory factors may include individual differences in distractibility in older adulthood per se, and failure to consider the influence of how difficult a person found the test. Methods: We therefore measured distractibility in a group of older adults by comparing the performance of a primary visual task (Swansea Test of Attentional Control), which includes an adaptive algorithm to take into account how difficult a person finds the test under both no-sound and sound conditions. Results: Analysis revealed no significant difference in group mean performance between no-sound versus sound conditions [t (33) = 0.181, p = 0.858; Cohen's effect size d = -0.028], but individual differences in performance both within and between sound and no-sound conditions were evident, indicating that for older adults, distracting sounds can be neutral, detrimental, or advantageous with respect to visual task performance. It was not possible to determine individual thresholds for whether sound versus no-sound conditions affected a person's actual behaviour. Conclusions: Nevertheless, our findings indicate how variable such effects may be in older adults, which in turn may help to explain outcome heterogeneity in studies including people living with dementia. Furthermore, such within-group heterogeneity highlights the importance of considering a person's individual performance in order to better understand their behaviour and initiate interventions as required.

Modeling High-Order Relationships: Brain-Inspired Hypergraph-Induced Multimodal-Multitask Framework for Semantic Comprehension

Semantic comprehension aims to reasonably reproduce people's real intentions or thoughts, e.g., sentiment, humor, sarcasm, motivation, and offensiveness, from multiple modalities. It can be instantiated as a multimodal-oriented multitask classification issue and applied to scenarios, such as online public opinion supervision and political stance analysis. Previous methods generally employ multimodal learning alone to deal with varied modalities or solely exploit multitask learning to solve various tasks, a few to unify both into an integrated framework. Moreover, multimodal-multitask cooperative learning could inevitably encounter the challenges of modeling high-order relationships, i.e., intramodal, intermodal, and intertask relationships. Related research of brain sciences proves that the human brain possesses multimodal perception and multitask cognition for semantic comprehension via decomposing, associating, and synthesizing processes. Thus, establishing a brain-inspired semantic comprehension framework to bridge the gap between multimodal and multitask learning becomes the primary motivation of this work. Motivated by the superiority of the hypergraph in modeling high-order relations, in this article, we propose a hypergraph-induced multimodal-multitask (HIMM) network for semantic comprehension. HIMM incorporates monomodal, multimodal, and multitask hypergraph networks to, respectively, mimic the decomposing, associating, and synthesizing processes to tackle the intramodal, intermodal, and intertask relationships accordingly. Furthermore, temporal and spatial hypergraph constructions are designed to model the relationships in the modality with sequential and spatial structures, respectively. Also, we elaborate a hypergraph alternative updating algorithm to ensure that vertices aggregate to update hyperedges and hyperedges converge to update their connected vertices. Experiments on the dataset with two modalities and five tasks verify the effectiveness of HIMM on semantic comprehension.

Stopping Speed in Response to Auditory and Visual Stop Signals Depends on Go Signal Modality

Past research has found that the speed of the action cancellation process is influenced by the sensory modality of the environmental change that triggers it. However, the effect on selective stopping processes (where participants must cancel only one component of a multicomponent movement) remains unknown, despite these complex movements often being required as we navigate our busy modern world. Thirty healthy adults (mean age = 31.1 years, SD = 10.5) completed five response-selective stop signal tasks featuring different combinations of "go signal" modality (the environmental change baring an imperative to initiate movement; auditory or visual) and "stop signal" modality (the environmental change indicating that action cancellation is required: auditory, visual, or audiovisual). EMG recordings of effector muscles allowed detailed comparison of the characteristics of voluntary action and cancellation between tasks. Behavioral and physiological measures of stopping speed demonstrated that the modality of the go signal influenced how quickly participants cancelled movement in response to the stop signal: Stopping was faster in two cross-modal experimental conditions (auditory go - visual stop; visual go - auditory stop), than in two conditions using the same modality for both signals. A separate condition testing for multisensory facilitation revealed that stopping was fastest when the stop signal consisted of a combined audiovisual stimulus, compared with all other go-stop stimulus combinations. These findings provide novel evidence regarding the role of attentional networks in action cancellation and suggest modality-specific cognitive resources influence the latency of the stopping process.

The Differential Effects of Multisensory Attentional Cues on Task Performance in VR Depending on the Level of Cognitive Load and Cognitive Capacity

As the utilization of VR is expanding across diverse fields, research on devising attentional cues that could optimize users' task performance in VR has become crucial. Since the cognitive load imposed by the context and the individual's cognitive capacity are representative factors that are known to determine task performance, we aimed to examine how the effects of multisensory attentional cues on task performance are modulated by the two factors. For this purpose, we designed a new experimental paradigm in which participants engaged in dual (N-back, visual search) tasks under different levels of cognitive load while an attentional cue (visual, tactile, or visuotactile) was presented to facilitate search performance. The results showed that multi-sensory attentional cues are generally more effective than uni-sensory cues in enhancing task performance, but the benefit of multi-sensory cues changes according to the level of cognitive load and the individual's cognitive capacity; the amount of benefit increases as the cognitive load is higher and the cognitive capacity is lower. The findings of this study provide practical implications for designing attentional cues to enhance VR task performance, considering both the complexity of the VR context and users' internal characteristics.

A touching advantage: cross-modal stop-signals improve reactive response inhibition

The ability to inhibit an already initiated response is crucial for navigating the environment. However, it is unclear which characteristics make stop-signals more likely to be processed efficiently. In three consecutive studies, we demonstrate that stop-signal modality and location are key factors that influence reactive response inhibition. Study 1 shows that tactile stop-signals lead to better performance compared to visual stop-signals in an otherwise visual choice-reaction task. Results of Study 2 reveal that the location of the stop-signal matters. Specifically, if a visual stop-signal is presented at a different location compared to the visual go-signal, then stopping performance is enhanced. Extending these results, study 3 suggests that tactile stop-signals and location-distinct visual stop-signals retain their performance enhancing effect when visual distractors are presented at the location of the go-signal. In sum, these results confirm that stop-signal modality and location influence reactive response inhibition, even in the face of concurrent distractors. Future research may extend and generalize these findings to other cross-modal setups.

The role of visual association cortices during response selection processes in interference-modulated response stopping

Response inhibition and the ability to navigate distracting information are both integral parts of cognitive control and are imperative to adaptive behavior in everyday life. Thus far, research has only inconclusively been able to draw inferences regarding the association between response stopping and the effects of interfering information. Using a novel combination of the Simon task and a stop signal task, the current study set out to investigate the behavioral as well as the neurophysiological underpinnings of the relationship between response stopping and interference processing. We tested n = 27 healthy individuals and combined temporal EEG signal decomposition with source localization methods to delineate the precise neurophysiological dynamics and functional neuroanatomical structures associated with conflict effects on response stopping. The results showed that stopping performance was compromised by conflicts. Importantly, these behavioral effects were reflected by specific aspects of information coded in the neurophysiological signal, indicating that conflict effects during response stopping are not mediated via purely perceptual processes. Rather, it is the processing of specific, stop-relevant stimulus features in the sensory regions during response selection, which underlies the emergence of conflict effects in response stopping. The findings connect research regarding response stopping with overarching theoretical frameworks of perception-action integration.

A cerebellum inspired spiking neural network as a multi-model for pattern classification and robotic trajectory prediction

Spiking neural networks were introduced to understand spatiotemporal information processing in neurons and have found their application in pattern encoding, data discrimination, and classification. Bioinspired network architectures are considered for event-driven tasks, and scientists have looked at different theories based on the architecture and functioning. Motor tasks, for example, have networks inspired by cerebellar architecture where the granular layer recodes sparse representations of the mossy fiber (MF) inputs and has more roles in motor learning. Using abstractions from cerebellar connections and learning rules of deep learning network (DLN), patterns were discriminated within datasets, and the same algorithm was used for trajectory optimization. In the current work, a cerebellum-inspired spiking neural network with dynamics of cerebellar neurons and learning mechanisms attributed to the granular layer, Purkinje cell (PC) layer, and cerebellar nuclei interconnected by excitatory and inhibitory synapses was implemented. The model’s pattern discrimination capability was tested for two tasks on standard machine learning (ML) datasets and on following a trajectory of a low-cost sensor-free robotic articulator. Tuned for supervised learning, the pattern classification capability of the cerebellum-inspired network algorithm has produced more generalized models than data-specific precision models on smaller training datasets. The model showed an accuracy of 72%, which was comparable to standard ML algorithms, such as MLP (78%), Dl4jMlpClassifier (64%), RBFNetwork (71.4%), and libSVM-linear (85.7%). The cerebellar model increased the network’s capability and decreased storage, augmenting faster computations. Additionally, the network model could also implicitly reconstruct the trajectory of a 6-degree of freedom (DOF) robotic arm with a low error rate by reconstructing the kinematic parameters. The variability between the actual and predicted trajectory points was noted to be ± 3 cm (while moving to a position in a cuboid space of 25 × 30 × 40 cm). Although a few known learning rules were implemented among known types of plasticity in the cerebellum, the network model showed a generalized processing capability for a range of signals, modulating the data through the interconnected neural populations. In addition to potential use on sensor-free or feed-forward based controllers for robotic arms and as a generalized pattern classification algorithm, this model adds implications to motor learning theory.

On the Role of Stimulus-Response Context in Inhibitory Control in Alcohol Use Disorder

The behavioral and neural dynamics of response inhibition deficits in alcohol use disorder (AUD) are still largely unclear, despite them possibly being key to the mechanistic understanding of the disorder. Our study investigated the effect of automatic vs. controlled processing during response inhibition in participants with mild-to-moderate AUD and matched healthy controls. For this, a Simon Nogo task was combined with EEG signal decomposition, multivariate pattern analysis (MVPA), and source localization methods. The final sample comprised n = 59 (32♂) AUD participants and n = 64 (28♂) control participants. Compared with the control group, AUD participants showed overall better response inhibition performance. Furthermore, the AUD group was less influenced by the modulatory effect of automatic vs. controlled processes during response inhibition (i.e., had a smaller Simon Nogo effect). The neurophysiological data revealed that the reduced Simon Nogo effect in the AUD group was associated with reduced activation differences between congruent and incongruent Nogo trials in the inferior and middle frontal gyrus. Notably, the drinking frequency (but not the number of AUD criteria we had used to distinguish groups) predicted the extent of the Simon Nogo effect. We suggest that the counterintuitive advantage of participants with mild-to-moderate AUD over those in the control group could be explained by the allostatic model of drinking effects.

A ventral stream-prefrontal cortex processing cascade enables working memory gating dynamics

The representation of incoming information, goals and the flexible processing of these are required for cognitive control. Efficient mechanisms are needed to decide when it is important that novel information enters working memory (WM) and when these WM 'gates' have to be closed. Compared to neural foundations of maintaining information in WM, considerably less is known about what neural mechanisms underlie the representational dynamics during WM gating. Using different EEG analysis methods, we trace the path of mental representations along the human cortex during WM gate opening and closing. We show temporally nested representational dynamics during WM gate opening and closing depending on multiple independent neural activity profiles. These activity profiles are attributable to a ventral stream-prefrontal cortex processing cascade. The representational dynamics start in the ventral stream during WM gate opening and WM gate closing before prefrontal cortical regions are modulated. A regional specific activity profile is shown within the prefrontal cortex depending on whether WM gates are opened or closed, matching overarching concepts of prefrontal cortex functions. The study closes an essential conceptual gap detailing the neural dynamics underlying how mental representations drive the WM gate to open or close to enable WM functions such as updating and maintenance.

Protocol to decode representations from EEG data with intermixed signals using temporal signal decomposition and multivariate pattern-analysis

The electroencephalogram (EEG) is one of the most widely used techniques in cognitive neuroscience. We present a protocol showing how to combine a temporal signal decomposition approach (RIDE, Residue iteration decomposition) with multivariate pattern analysis (MVPA) to obtain insights into the temporal stability of representations coded in distinct informational fractions of the EEG signal. In this protocol, we describe pre-processing of human EEG data, followed by the set-up and use of MATLAB-based toolboxes for RIDE and MVPA analysis.

For complete details on the use and execution of this protocol, please refer to Petruo et al. (2021).

•

A protocol for decoding temporally decomposed EEG signal

•

Steps for Residue iteration decomposition (RIDE) and handling the decomposed data

•

Steps for subsequent multivariate pattern analysis (MVPA) with different toolboxes

•

Recommendations for combined RIDE-MVPA research applications

Publisher’s note: Undertaking any experimental protocol requires adherence to local institutional guidelines for laboratory safety and ethics.

Alpha and Theta Bands Dynamics Serve Distinct Functions during Perception-Action Integration in Response Inhibition

The ability to inhibit responses is central for situational behavior. However, the mechanisms how sensory information is used to inform inhibitory control processes are incompletely understood. In the current study, we examined neurophysiological processes of perception-action integration in response inhibition using the theory of event coding as a conceptual framework. Based on theoretical considerations, we focused on theta and alpha band activity in close connection to the functional neuroanatomical level using EEG beamforming. Moreover, we performed a network-based analysis of theta and alpha band activity. We show a seesaw-like relationship between medial and superior frontal cortex theta band activity and frontoparietal cortex alpha band activity during perception-action integration in response inhibition, depending on the necessity to reconfigure perception-action associations. When perception-action integration was more demanding, because perception-action associations (bindings) have to be reconfigured, there was an increase of theta and a decrease of alpha band activity. Vice versa, when there was no need to reconfigure perception-action bindings, theta band activity was low and alpha band activity was high. However, theta band processes seem to be most important for perception-action integration in response inhibition, because only the sensor-level network organization of theta band activity showed variations depending on the necessity to reconfigure perception-action associations. When no reconfiguration was necessary, the network architecture was more small-world-like, likely enabling efficient processing. When reconfigurations were necessary, the network organization becomes more random. These differences were particularly strong for fractions of the neurophysiological signal supposed to reflect response selection processes.

Alterations of Prefrontal-Posterior Information Processing Patterns in Autism Spectrum Disorders

Autism spectrum disorder (ASD) is a heterogeneous disorder characterized by different levels of repetitive and stereotypic behavior as well as deficits in social interaction and communication. In this current study, we explored the changes in cerebral neural activities in ASD. The purpose of this study is to investigate whether there exists a dysfunction of interactive information processing between the prefrontal cortex and posterior brain regions in ASD. We investigated the atypical connectivity and information flow between the prefrontal cortex and posterior brain regions in ASD utilizing the entropy connectivity (a kind of directional connectivity) method. Eighty-nine patients with ASD and 94 typical developing (TD) teenagers participated in this study. Two-sample t-tests revealed weakened interactive entropy connectivity between the prefrontal cortex and posterior brain regions. This result indicates that there exists interactive prefrontal-posterior underconnectivity in ASD, and this disorder might lead to less prior knowledge being used and updated. Our proposals highlighted that aforementioned atypical change might accelerate the deoptimization of brain networks in ASD.

Multi-level decoding of task sets in neurophysiological data during cognitive flexibility

Cognitive flexibility is essential to achieve higher level goals. Cognitive theories assume that the activation/deactivation of goals and task rules is central to understand cognitive flexibility. However, how this activation/deactivation dynamic is implemented on a neurophysiological level is unclear. Using EEG-based multivariate pattern analysis (MVPA) methods, we show that activation of relevant information occurs parallel in time at multiple levels in the neurophysiological signal containing aspects of stimulus-related processing, response selection, and motor response execution, and relates to different brain regions. The intensity with which task sets are activated and processed dynamically decreases and increases. The temporal stability of these activations could, however, hardly explain behavioral performance. Instead, task set deactivation processes associated with left orbitofrontal regions and inferior parietal regions selectively acting on motor response task sets are relevant. The study shows how propositions from cognitive theories stressing the importance task set activation/deactivation during cognitive flexibility are implemented on a neurophysiological level.

•

Stimulus-related, motor, and response selection aspects of task set were decoded

•

Activation of task rule information occurs at multiple neurophysiological levels

•

Activation and deactivation of rule sets contributes to cognitive flexibility

Perception-action integration in young age-A cross-sectional EEG study

Humans differ in their capacity for integrating perceived events and related actions. The "Theory of event coding" (TEC) conceptualizes how stimuli and actions are cognitively bound into a common functional representation (or "code"), known as the "event file". To date, however, the neural processes underlying the development of event file coding mechanisms across age are largely unclear. We investigated age-related neural changes of event file coding from late childhood to early adulthood, using EEG signal decompositions methods. We included a group of healthy participants (n = 91) between 10 and 30 years, performing an event file paradigm. Results of this study revealed age-related effects on event file coding processes both at the behavioural and the neurophysiological level. Performance accuracy data showed that event file unbinding und rebinding processes become more efficient from late childhood to early adulthood. These behavioural effects are reflected by age-related effects in two neurophysiological subprocesses associated with the superior parietal cortex (BA7) as revealed in the analyses using EEG signal decomposition. The first process entails mapping and association processes between stimulus and response; whereas, the second comprises inhibitory control subprocesses subserving the selection of the relevant motor programme amongst competing response options.

Neural dynamics of stimulus-response representations during inhibitory control

The investigation of action control processes is one major field in cognitive neuroscience and several theoretical frameworks have been proposed. One established framework is the "Theory of Event Coding" (TEC). However, only rarely, this framework has been used in the context of response inhibition and how stimulus-response association or binding processes modulate response inhibition performance. Particularly the neural dynamics of stimulus-response representations during inhibitory control are elusive. To address this, we examined n = 40 healthy controls and combined temporal EEG signal decomposition with source localization and temporal generalization multivariate pattern analysis (MVPA). We show that overlaps in features of stimuli used to trigger either response execution or inhibition compromised task performance. According to TEC, this indicates that binding processes in event file representations impact response inhibition through partial repetition costs. In the EEG data, reconfiguration of event files modulated processes in time windows well-known to reflect distinct response inhibition mechanisms. Crucially, event file coding processes were only evident in a specific fraction of neurophysiological activity associated with the inferior parietal cortex (BA40). Within that specific fraction of neurophysiological activity, the decoding of the dynamics of event file representations using temporal generalization MVPA suggested that event file representations are stable across several hundred milliseconds, and that event file coding during inhibitory control is reflected by a sustained activation pattern of neural dynamics.NEW & NOTEWORTHY The "mental representation" of how stimulus input translate into the appropriate response is central for goal-directed behavior. However, little is known about the dynamics of such representations on the neurophysiological level when it comes to the inhibition of motor processes. This dynamic is shown in the current study.

Neurophysiological and functional neuroanatomical coding of statistical and deterministic rule information during sequence learning

Humans are capable of acquiring multiple types of information presented in the same information stream. It has been suggested that at least two parallel learning processes are important during learning of sequential patterns-statistical learning and rule-based learning. Yet, the neurophysiological underpinnings of these parallel learning processes are not fully understood. To differentiate between the simultaneous mechanisms at the single trial level, we apply a temporal EEG signal decomposition approach together with sLORETA source localization method to delineate whether distinct statistical and rule-based learning codes can be distinguished in EEG data and can be related to distinct functional neuroanatomical structures. We demonstrate that concomitant but distinct aspects of information coded in the N2 time window play a role in these mechanisms: mismatch detection and response control underlie statistical learning and rule-based learning, respectively, albeit with different levels of time-sensitivity. Moreover, the effects of the two learning mechanisms in the different temporally decomposed clusters of neural activity also differed from each other in neural sources. Importantly, the right inferior frontal cortex (BA44) was specifically implicated in visuomotor statistical learning, confirming its role in the acquisition of transitional probabilities. In contrast, visuomotor rule-based learning was associated with the prefrontal gyrus (BA6). The results show how simultaneous learning mechanisms operate at the neurophysiological level and are orchestrated by distinct prefrontal cortical areas. The current findings deepen our understanding on the mechanisms of how humans are capable of learning multiple types of information from the same stimulus stream in a parallel fashion.

When two is worse than one: The deleterious impact of multisensory stimulation on response inhibition

Multisensory facilitation is known to improve the perceptual performances and reaction times of participants in a wide range of tasks, from detection and discrimination to memorization. We asked whether a multimodal signal can similarly improve action inhibition using the stop-signal paradigm. Indeed, consistent with a crossmodal redundant signal effect that relies on multisensory neuronal integration, the threshold for initiating behavioral responses is known for being reached faster with multisensory stimuli. To evaluate whether this phenomenon also occurs for inhibition, we compared stop signals in unimodal (human faces or voices) versus audiovisual modalities in natural or degraded conditions. In contrast to the expected multisensory facilitation, we observed poorer inhibition efficiency in the audiovisual modality compared with the visual and auditory modalities. This result was corroborated by both response probabilities and stop-signal reaction times. The visual modality (faces) was the most effective. This is the first demonstration of an audiovisual impairment in the domain of perception and action. It suggests that when individuals are engaged in a high-level decisional conflict, bimodal stimulation is not processed as a simple multisensory object improving the performance but is perceived as concurrent visual and auditory information. This absence of unity increases task demand and thus impairs the ability to revise the response.

Anodal tDCS modulates specific processing codes during conflict monitoring associated with superior and middle frontal cortices

Conflict monitoring processes are central for cognitive control. Neurophysiological correlates of conflict monitoring (i.e. the N2 ERP) likely represent a mixture of different cognitive processes. Based on theoretical considerations, we hypothesized that effects of anodal tDCS (atDCS) in superior frontal areas affect specific subprocesses in neurophysiological activity during conflict monitoring. To investigate this, young healthy adults performed a Simon task while EEG was recorded. atDCS and sham tDCS were applied in a single-blind, cross-over study design. Using temporal signal decomposition in combination with source localization analyses, we demonstrated that atDCS effects on cognitive control are very specific: the detrimental effect of atDCS on response speed was largest in case of response conflicts. This however only showed in aspects of the decomposed N2 component, reflecting stimulus-response translation processes. In contrast to this, stimulus-related aspects of the N2 as well as purely response-related processes were not modulated by atDCS. EEG source localization analyses revealed that the effect was likely driven by activity modulations in the superior frontal areas, including the supplementary motor cortex (BA6), as well as middle frontal (BA9) and medial frontal areas (BA32). atDCS did not modulate effects of proprioceptive information on hand position, even though this aspect is known to be processed within the same brain areas. Physiological effects of atDCS likely modulate specific aspects of information processing during cognitive control.

Pushing to the Limits: What Processes during Cognitive Control are Enhanced by Reaction-Time Feedback?

To respond as quickly as possible in a given task is a widely used instruction in cognitive neuroscience; however, the neural processes modulated by this common experimental procedure remain largely elusive. We investigated the underlying neurophysiological processes combining electroencephalography (EEG) signal decomposition (residue iteration decomposition, RIDE) and source localization. We show that trial-based response speed instructions enhance behavioral performance in conflicting trials, but slightly impair performance in nonconflicting trials. The modulation seen in conflicting trials was found at several coding levels in EEG data using RIDE. In the S-cluster N2 time window, this modulation was associated with modulated activation in the posterior cingulate cortex and the superior frontal gyrus. Furthermore, in the C-cluster P3 time window, this modulation was associated with modulated activation in the middle frontal gyrus. Interestingly, in the R-cluster P3 time window, this modulation was strongest according to statistical effect sizes, associated with modulated activity in the primary motor cortex. Reaction-time feedback mainly modulates response motor execution processes, whereas attentional and response selection processes are less affected. The study underlines the importance of being aware of how experimental instructions influence the behavior and neurophysiological processes.

Task Switching and the Role of Motor Reprogramming in Parietal Structures

Human behaviour amazes with extraordinary flexibility and the underlying neural mechanisms have often been studied using task switching. Despite extensive research, the relative importance of "cognitive" and "motor" aspects during switching is unclear. In the current study we examine this question combining EEG analysis techniques and source localization to examine whether the selection of the response, or processes during the execution of the response, contribute most to switching effects. A clear dissociation was observed in the signal decomposition, since codes relating to motor aspects play a significant role in task switching and the scope of the switching costs. This was not the case for signals that denote reaction selection or decision processes that respond to selection or basic stimulus processing codes. On a functional neuroanatomical level, these modulations in motor processes showed a clear temporal sequence in that motor codes are processed primarily in superior parietal regions (Brodman area 7) and only then in premotor regions (Brodman area 6). The observed modulations may reflect motor reprogramming processes. The study shows how EEG signal analysis in combination with brain mapping methods can inform debates on theories of human cognitive flexibility.

Modality-specific neural mechanisms of cognitive control in a Stroop-like task

The successful resolution of ever-changing conflicting contexts requires efficient cognitive control. Previous studies have found similar neural patterns in conflict processing for different modalities using an event-related potential (ERP) approach and have concluded that cognitive control is supramodal. However, recent behavioral studies have found that conflict adaptation (a phenomenon with the reduction of congruency effect in the current trial after an incongruent trial as compared with a congruent trial) could not transfer across visual and auditory modalities and suggested that cognitive control is modality-specific, challenging the supramodal view. These discrepancies may have also arisen from methodological differences across studies. The current study examined the electroencephalographic profiles of a Stroop-like task to elucidate the modality-specific neural mechanisms of cognitive control. Participants were instructed to respond to a target always coming from the visual modality while disregarding the distractor coming from either the auditory or the visual modality. The results revealed significant congruency effects on both behavioral indices, i.e., reaction time and error rate, and ERP components, including the P3 and the conflict slow potential. Besides, the congruency effects on the amplitude of the P3 showed a negative correlation with reaction time, indicating an intrinsic link between these neural and behavioral indices. Furthermore, in the modality-repetition condition, conflict adaptation effects were significant on both reaction time and P3 amplitude, and the reaction time could be predicted by the P3 amplitude, while such effects were not observed in the modality-alternation condition. The time-frequency analysis also showed that conflict adaptation occurred in the modality-repetition condition, but not in the modality-alternation condition in low frequency bands, including the theta (4-8 Hz), alpha (8-12 Hz), and beta1 (12-20 Hz) bands. Taken together, our results revealed modality-specific patterns of the conflict adaptation effects on the P3 amplitude and oscillatory power (in theta, alpha, and beta1 bands), providing neural evidence for the modality specificity of cognitive control and expanding the boundaries of cognitive control.

Resting theta activity is associated with specific coding levels in event-related theta activity during conflict monitoring

Brain electrical activity in the theta frequency band is essential for cognitive control (e.g., during conflict monitoring), but is also evident in the resting state. The link between resting state theta activity and its relevance for theta-related neural mechanisms during cognitive control is still undetermined. Yet, theoretical considerations suggest that there may be a connection. To examine the link between resting state theta activity and conflict-related theta activity, we combined temporal EEG signal decomposition methods with time-frequency decomposition and beamforming methods in N = 86 healthy participants. Results indicate that resting state theta activity is closely associated with the strength of conflict-related neural activity at the level of ERPs and total theta power (consisting of phase-locked and nonphase-locked aspects of theta activity). The data reveal that resting state theta activity is related to a specific aspect of conflict-related theta activity, mainly in superior frontal regions and in the supplemental motor area (SMA, BA6) in particular. The signal decomposition showed that only stimulus-related, but not motor-response-related coding levels in the EEG signal and the event-related total theta activity were associated with resting theta activity. This specificity of effects may explain why the association between resting state theta activity and overt conflict monitoring performance may not be as strong as often assumed. The results suggest that resting state theta activity is particularly important to consider for input integration processes during cognitive control.

On the Reliability of Examining Dual-Tasking Abilities Using a Novel E-Health Device—A Proof of Concept Study in Multiple Sclerosis

The assessment of neuropsychological functions and especially dual-tasking abilities is considered to be increasingly relevant in the assessment of neurological disease, and Multiple Sclerosis (MS) in particular. However, the assessment of dual-tasking abilities is hindered by specific software requirements and extensive testing times. We designed a novel e-health (progressive web application-based) device for the assessment of dual-tasking abilities usable in “bedside” and outpatient clinic settings and examined its reliability in a sample of N = 184 MS patients in an outpatient setting. Moreover, we examined the relevance of dual-tasking assessment using this device with respect to clinically relevant parameters in MS. We show that a meaningful assessment of dual-tasking is possible within 6 min and that the behavioral readouts overall show good reliability depending on dual-tasking difficulty. We show that dual-tasking readouts were correlated with clinically relevant parameters (e.g., EDSS, disease duration, processing speed) and were not affected by fatigue levels. We consider the tested dual-tasking assessment device suitable for routine clinical neuropsychological assessments of dual-tasking abilities. Future studies may further evaluate this test regarding its suitability in the long-term follow up assessments and to assess dual-tasking abilities in other neurological and psychiatric disorders.

Why Cognitive–Cognitive Dual-Task Testing Assessment Should Be Implemented in Studies on Multiple Sclerosis and in Regular Clinical Practice

Cognitive impairment is prevalent and disabling in multiple sclerosis (MS) and is severely impacting quality of life (QoL). Aside its routine assessment in clinical care, it should more often be implemented as endpoint/outcome measure in clinical trials. However, a fundamental aspect—often neglected in clinical practice and clinical trials—is the assessment of multi-tasking and dual-tasking abilities. In this perspective article, we outline why, given the nature of MS, particularly the assessment of “cognitive–cognitive dual-tasking” is relevant in MS. We delineate how knowledge from basic cognitive science can inform the assessment of this important cognitive impairment in MS. Finally, we outline how the assessment of “cognitive–cognitive dual-tasking” can be implemented in computer-based screening tools (e-health devices) that can be used not only in clinical diagnostics but also in clinical trials.

Decoding Stimulus-Response Representations and Their Stability Using EEG-Based Multivariate Pattern Analysis

Goal-directed actions require proper associations between stimuli and response. This has been delineated by cognitive theory, for example, in the theory of event coding framework, which proposes that event files represent such bindings. Yet, how such event file representations are coded on a neurophysiological level is unknown. We close this gap combining temporal electroencephalography (EEG) signal decomposition methods and multivariate pattern analysis (MVPA). We show that undecomposed neurophysiological data is unsuitable to decode event file representations because different aspects of information coded in the neurophysiological signal reveal distinct and partly opposed dynamics in the representational content. This is confirmed by applying MVPA to temporal decomposed EEG data. After intermixed aspects of information in the EEG during response selection have been separated, a reliable examination of the event file’s representational content and its temporal stability was possible. We show that representations of stimulus–response bindings are activated and decay in a gradual manner and that event file representations resemble distributed neural activity. Especially representations of stimulus–response bindings, as well as stimulus-related representations, are coded and reveal temporal stability. Purely motor-related representations are not found in neurophysiological signals during event coding.

EEG Signal Decomposition Evidence for a Role of Perceptual Processes during Conflict-related Behavioral Adjustments in Middle Frontal Regions

Conflict monitoring processes are central to cope with fluctuating environmental demands. However, the efficacy of these processes depends on previous trial history/experience, which is reflected in the "congruency sequence effect" (CSE). Several theoretical accounts have been put forward to explain this effect. Some accounts stress the role of perceptual processes in the emergence of the CSE. As yet, it is elusive how these perceptual processes are implemented on a neural level. We examined this question using a newly developed moving dots flanker task. We combine decomposition methods of EEG data and source localization. We show that perceptual processes modulate the CSE and can be isolated in neurophysiological signals, especially in the N2 ERP time window. However, mechanisms relating perception to action are also coded and modulated in this time window. We show that middle frontal regions (BA 6) are associated with processes dealing with purely perceptual processes. Inferior frontal regions (BA 45) are associated with processes dealing with stimulus-response transition processes. Likely, the neurophysiological modulations reflect unbinding processes at the perceptual level, and stimulus-response translation level needed to respond correctly on the presented (changed) stimulus-response relationships. The data establish a direct relationship between psychological concepts focusing on perceptual processes during conflict monitoring and neurophysiological processes using signal decomposition.

Connecting EEG signal decomposition and response selection processes using the theory of event coding framework

The neurophysiological mechanisms underlying the integration of perception and action are an important topic in cognitive neuroscience. Yet, connections between neurophysiology and cognitive theoretical frameworks have rarely been established. The theory of event coding (TEC) details how perceptions and actions are associated (bound) in a common representational domain (the “event file”), but the neurophysiological mechanisms underlying these processes are hardly understood. We used complementary neurophysiological methods to examine the neurophysiology of event file processing (i.e., event‐related potentials [ERPs], temporal EEG signal decomposition, EEG source localization, time‐frequency decomposition, EEG network analysis). We show that the P3 ERP component and activity modulations in inferior parietal regions (BA40) reflect event file binding processes. The relevance of this parietal region is corroborated by source localization of temporally decomposed EEG data. We also show that temporal EEG signal decomposition reveals a pattern of results suggesting that event file processes can be dissociated from pure stimulus and response‐related processes in the EEG signal. Importantly, it is also documented that event file binding processes are reflected by modulations in the network architecture of theta frequency band activity. That is, when stimulus–response bindings in event files hamper response selection this was associated with a less efficient theta network organization. A more efficient organization was evident when stimulus–response binding in event files facilitated response selection. Small‐world network measures seem to reflect event file processing. The results show how cognitive‐theoretical assumptions of TEC can directly be mapped to the neurophysiology of response selection.

Low and high stimulation frequencies differentially affect automated response selection in the superior parietal cortex - implications for somatosensory area processes

Response inhibition as a central facet of executive functioning is no homogeneous construct. Interference inhibition constitutes a subcomponent of response inhibition and refers to inhibitory control over responses that are automatically triggered by irrelevant stimulus dimensions as measured by the Simon task. While there is evidence that the area-specific modulation of tactile information affects the act of action withholding, effects in the context of interference inhibition remain elusive. We conducted a tactile version of the Simon task with stimuli designed to be predominantly processed in the primary (40 Hz) or secondary (150 Hz) somatosensory cortex. On the basis of EEG recordings, we performed signal decomposition and source localization. Behavioral results reveal that response execution is more efficient when sensory information is mainly processed via SII, compared to SI sensory areas during non-conflicting trials. When accounting for intermingled coding levels by temporally decomposing EEG data, the results show that experimental variations depending on sensory area-specific processing differences specifically affect motor and not sensory processes. Modulations of motor-related processes are linked to activation differences in the superior parietal cortex (BA7). It is concluded that the SII cortical area supporting cognitive preprocessing of tactile input fosters automatic tactile information processing by facilitating stimulus-response mapping in posterior parietal regions.

Immediate early gene fingerprints of multi-component behaviour

The ability to execute different responses in an expedient temporal order is central for efficient goal-directed actions and often referred to as multi-component behaviour. However, the underlying neural mechanisms on a cellular level remain unclear. Here we establish a link between neural activity at the cellular level within functional neuroanatomical structures to this form of goal-directed behaviour by analyzing immediate early gene (IEG) expression in an animal model, the pigeon (Columba livia). We focus on the group of zif268 IEGs and ZENK in particular. We show that when birds have to cascade separate task goals, ZENK expression is increased in the avian equivalent of the mammalian prefrontal cortex, i.e. the nidopallium caudolaterale (NCL) as well as in the homologous striatum. Moreover, in the NCL as well as in the medial striatum (MSt), the degree of ZENK expression was highly correlated with the efficiency of multi-component behaviour. The results provide the first link between cellular IEG expression and behavioural outcome in multitasking situations. Moreover, the data suggest that the function of the fronto-striatal circuitry is comparable across species indicating that there is limited flexibility in the implementation of complex cognition such as multi-component behaviour within functional neuroanatomical structures.

Passive perceptual learning modulates motor inhibitory control in superior frontal regions

Response inhibition is of vital importance in the context of controlling inappropriate responses. The role of perceptual processes during inhibitory control has attracted increased interest. Yet, we are far from an understanding of the mechanisms. One candidate mechanism by which perceptual processes may affect response inhibition refers to “gain control” that is closely linked to the signal‐to‐noise ratio of incoming information. A means to modulate the signal‐to‐noise ratio and gain control mechanisms is perceptual learning. In the current study, we examine the impact of perceptual learning (i.e., passive repetitive sensory stimulation) on response inhibition combining EEG signal decomposition with source localization analyses. A tactile GO/NOGO paradigm was conducted to measure action restraint as one subcomponent of response inhibition. We show that passive perceptual learning modulates response inhibition processes. In particular, perceptual learning attenuates the detrimental effect of response automation during inhibitory control. Temporally decomposed EEG data show that stimulus‐related and not response selection processes during conflict monitoring are linked to these effects. The superior and middle frontal gyrus (BA6), as well as the motor cortex (BA4), are associated with the effects of perceptual learning on response inhibition. Reliable neurophysiological effects were not evident on the basis of standard ERPs, which has important methodological implications for perceptual learning research. The results detail how lower level sensory plasticity protocols affect higher‐order cognitive control functions in frontal cortical structures.

How perceptual ambiguity affects response inhibition processes

The ability to inhibit responses is a central requirement for goal-directed behavior but has been dominated by a top-down or cognitive control view. Only recently, the role of bottom-up perceptual factors were focused in research. However, studies usually use clearly distinguishable stimulus categories to trigger response execution or inhibition. In the current study, we present a novel Gabor patch Go/No-go task to induce perceptual ambiguity during response inhibition. To examine the neurophysiological processes in detail, we use EEG recordings and combined temporal EEG signal decomposition methods with source localization analyses. We show that perceptual similarity between Go and No-go trials compromises response inhibition performance. Interestingly, the EEG data show that this is due to a modulation of stimulus-response transition or decision processes, and not purely stimulus-related processes. This was possible by applying a temporal EEG decomposition method. We provide evidence that a prefrontal P2 (pP2) likely reflects decision processes on action execution using stimulus information. These processes were associated with superior and middle prefrontal regions (BA8). When these processes fail, occasions to execute a response become misinterpreted as occasions to inhibit a response. Successful and unsuccessful decisions to inhibit a response under high perceptual ambiguity seem to further depend on how well "what-decisions," supported by neural mechanisms in BA19, can be processed. However, these what-decisions seem to be closely linked to the specification of the required action. Stimulus processing is closely linked to response programming so that response control is already informed when uncertainty with regard to stimulus identity is detected.NEW & NOTEWORTHY This study introduces a novel Go/No-go paradigm and shows what neurophysiological subprocesses and functional neuroanatomical are involved during inhibitory control when ambiguous stimulus input is provided. The results show that bottom-up perceptual processes are important to consider during top-down controlled response inhibition. Stimulus processing is closely linked to response programming so that response control is already informed when uncertainty with regard to stimulus identity is detected.

Paradoxical response inhibition advantages in adolescent obsessive compulsive disorder result from the interplay of automatic and controlled processes

Response inhibition deficits have often been described in obsessive compulsive disorder (OCD). Yet, research on response inhibition in OCD focusses on “top-down” controlled mechanisms, and it has been neglected that response inhibition performance depends on the interplay of controlled and automatic processes during response selection. Based on pathophysiological considerations we test the counterintuitive hypothesis that OCD patients show superior inhibitory control when automatic mechanisms govern processes involved in response inhibition. We examined a group of adolescent OCD patients (n = 27) and healthy controls (n = 27) using a combined Simon-Go/NoGo task. This task is able to examine conjoint effects of automatic and controlled processes during response inhibition. EEG and source localization analyses were applied to examine the underlying neural mechanisms. OCD patients committed fewer false alarms than healthy controls (HC) in the congruent Simon-NoGo condition, which is dominated by automatic response selection mechanisms. On a neurophysiological (EEG) level, these effects were reflected by intensified correlates of ‘braking’ processes associated with modulation of right inferior prefrontal regions. There is no general response inhibition deficit in adolescent OCD. When considering conjoint effects of automatic and controlled processes during the inhibition of responses paradoxical response inhibition advantages can emerge in OCD. This is likely a result of otherwise pathological fronto-striatal hyperactivity and loss of a situation-specific modulation of response selection mechanisms in OCD.

•

Effects of automatic/controlled processes on response inhibition (RI) are studied.

•

OCD patients show better performance in automatic vs. controlled RI.

•

Underlying neurophysiological (EEG) processes are delineated.

•

Activation differences in the rIFG are associated with this effect.

•

Effects are discussed in neurobiological frameworks of OCD.

Neural Correlates of Enhanced Audiovisual Processing in the Bilingual Brain

Bilingualism is associated with enhancements in perceptual and cognitive processing necessary for juggling multiple languages. Recent psychophysical studies demonstrate bilinguals also show enhanced multisensory processing and more restricted temporal binding windows for integrating audiovisual information. Here, we probed the neural mechanisms of bilinguals' audiovisual benefits. We recorded neuroelectric responses in mono- and bi-lingual listeners to the double-flash paradigm in which auditory beeps concurrent with a single visual flash induces the perceptual illusion of multiple flashes. Relative to monolinguals, bilinguals showed less susceptibility to the illusion (fewer false perceptual reports) coupled with stronger and faster event-related potentials to audiovisual information. Source analyses of EEG data revealed monolinguals' increased propensity for erroneously perceiving audiovisual stimuli was attributed to increased activity in primary visual (V1) and auditory cortex (PAC), increases in multisensory association areas (BA 37), but reduced frontal activity (BA 10). Regional activations were associated with an opposite pattern of behaviors: whereas stronger V1 and PAC activity predicted slower behavioral responses, stronger frontal BA10 responses elicited faster judgments. Our results suggest bilinguals' higher precision in audiovisual perception reflects more veridical sensory coding of physical cues coupled with superior top-down gating of sensory information to suppress the generation of false percepts. Findings underscore that the plasticity afforded by speaking multiple languages shapes extra-linguistic brain regions and can enhance audiovisual brain processing in a domain-general manner.

The Intensity of Early Attentional Processing, but Not Conflict Monitoring, Determines the Size of Subliminal Response Conflicts

Response conflicts hamper goal-directed behavior and may be evoked by both consciously and subliminally (unconsciously) processed information. Yet, not much is known about the mechanisms and brain regions driving the size of subliminally induced conflicts. We hence combined a response conflict paradigm featuring subliminal primes and conscious flankers with in-depth neurophysiological (EEG) analyses, including source localization in a sample of N = 243 healthy subjects. Intra-individual differences in the size of subliminal conflicts were reflected both during early attentional stimulus processing (prime-associated N1 and target-associated P1 and N1 amplitudes) and conflict monitoring (N2 amplitudes). On the neuroanatomical level, this was reflected by activity modulations in the TPJ (BA39, BA40) and V2 (BA18), which are known to be involved in attentional stimulus processing and task set maintenance. In addition to a "standard" analysis of event-related potentials, we also conducted a purely data-driven machine learning approach using support vector machines (SVM) in order to identify neurophysiological features which do not only reflect the size of subliminal conflict, but actually allow to classify/predict it. This showed that only extremely early information processing (about 65 ms after the onset of the prime) was predictive of subliminal conflict size. Importantly, this predictive feature occurred before target information could even be processed and was reflected by activity in the left middle frontal gyrus (BA6) and insula (BA13). We conclude that differences in task set maintenance and potentially also in subliminal attentional processing of task-relevant features, but not conflict monitoring, determine the size of subliminally induced response conflicts.

Evidence for an altered architecture and a hierarchical modulation of inhibitory control processes in ADHD

Inhibitory control deficits are a hallmark in ADHD. Yet, inhibitory control includes a multitude of entities (e.g. ‘inhibition of interferences’ and ‘action inhibition’). Examining the interplay between these kinds of inhibitory control provides insights into the architecture of inhibitory control in ADHD. Combining a Simon task and a Go/Nogo task, we assessed the interplay of ‘inhibition of interferences’ and ‘action inhibition’. This was combined with EEG recordings, EEG data decomposition and source localization. Simon interference effects in Go trials were larger in ADHD. At the neurophysiological level, this insufficient inhibition of interferences in ADHD related to the superior parietal cortex. Simon interference effects were absent in action inhibition (Nogo) trials in ADHD, compared to controls. This was supported by bayesian statistics. The power of effects was higher than 95%. The differential effects between the groups were associated with modulations of neurophysiological response selection processes in the superior frontal gyrus. ADHD is not only associated with deficits in inhibitory control. Rather, the organization and architecture of the inhibitory control system is different in ADHD. Distinguishable inhibitory control processes operate on a hierarchical ‘first come, first serve’ basis and are not integrated in ADHD. This is a new facet of ADHD.

Machine learning provides novel neurophysiological features that predict performance to inhibit automated responses

Neurophysiological features like event-related potentials (ERPs) have long been used to identify different cognitive sub-processes that may contribute to task performance. It has however remained unclear whether “classical” ERPs are truly the best reflection or even causal to observable variations in behavior. Here, we used a data-driven strategy to extract features from neurophysiological data of n = 240 healthy young individuals who performed a Go/Nogo task and used machine learning methods in combination with source localization to identify the best predictors of inter-individual performance variations. Both Nogo-N2 and Nogo-P3 yielded predictions close to chance level, but a feature in between those two processes, associated with motor cortex activity (BA4), predicted group membership with up to ~68%. We further found two Nogo-associated features in the theta and alpha bands, that predicted behavioral performance with up to ~78%. Notably, the theta band feature contributed most to the prediction and occurred at the same time as the predictive ERP feature. Our approach provides a rigorous test for established neurophysiological correlates of response inhibition and suggests that other processes, which occur in between the Nogo-N2 and P3, might be of equal, if not even greater, importance.

Dual-Tasking in Multiple Sclerosis - Implications for a Cognitive Screening Instrument

The monitoring of cognitive functions is central to the assessment and consecutive management of multiple sclerosis (MS). Though, especially cognitive processes that are central to everyday behavior like dual-tasking are often neglected. We examined dual-task performance using a psychological-refractory period (PRP) task in N = 21 patients and healthy controls and conducted standard neuropsychological tests. In dual-tasking, MS patients committed more erroneous responses when dual-tasking was difficult. In easier conditions, performance of MS patients did not differ to controls. Interestingly, the response times were generally not affected by the difficulty of the dual task, showing that the deficits observed do not reflect simple motor deficits or deficits in information processing speed but point out deficits in executive control functions and response selection in particular. Effect sizes were considerably large with d∼0.80 in mild affected patients and the achieved power was above 99%. There are cognitive control and dual tasking deficits in MS that are not attributable to simple motor speed deficits. Scaling of the difficulty of dual-tasking makes the test applied suitable for a wide variety of MS-patients and may complement neuropsychological assessments in clinical care and research setting.

Response selection codes in neurophysiological data predict conjoint effects of controlled and automatic processes during response inhibition

The inhibition of prepotent responses is a requirement for goal-directed behavior and several factors determine corresponding successful response inhibition processes. One factor relates to the degree of automaticity of pre-potent response tendencies and another factor relates to the degree of cognitive control that is exerted during response inhibition. However, both factors can conjointly modulate inhibitory control. Cognitive theoretical concepts suggest that codings of stimulus-response translations may underlie such conjoint effects. Yet, it is unclear in how far such specific codes, as assumed in cognitive psychological concepts, are evident in neurophysiological processes and whether there are specific functional neuroanatomical structures associated with the processing of such codes. Applying a temporal decomposition method of EEG data in combination with source localization methods we show that there are different, intermingled codes (i.e., "stimulus codes" and "response selection codes") at the neurophysiological level during conjoint effects of "automatic" and "controlled" processes in response inhibition. Importantly, only "response selection codes" predict behavioral performance, and are subject to conjoint modulations by "automatic" and "controlled" processes. These modulations are associated with inferior and superior parietal areas (BA40/BA7), possibly reflecting an updating of internal representations when information is complex and probably difficult to categorize, but essential for behavioral control. Codes proposed by cognitive, psychological concepts seem to have a neurophysiological analogue that fits into current views on functions of inferior and superior parietal regions.