A matter of attention: Crossmodal congruence enhances and impairs performance in a novel trimodal matching paradigm

Touch Helps Hearing: Evidence From Continuous Audio-Tactile Stimulation

OBJECTIVES

Identifying target sounds in challenging environments is crucial for daily experiences. It is important to note that it can be enhanced by nonauditory stimuli, for example, through lip-reading in an ongoing conversation. However, how tactile stimuli affect auditory processing is still relatively unclear. Recent studies have shown that brief tactile stimuli can reliably facilitate auditory perception, while studies using longer-lasting audio-tactile stimulation yielded conflicting results. This study aimed to investigate the impact of ongoing pulsating tactile stimulation on basic auditory processing.

DESIGN

In experiment 1, the electroencephalogram (EEG) was recorded while 24 participants performed a loudness-discrimination task on a 4-Hz modulated tone-in-noise and received either in-phase, anti-phase, or no 4-Hz electrotactile stimulation above the median nerve. In experiment 2, another 24 participants were presented with the same tactile stimulation as before, but performed a tone-in-noise detection task while their selective auditory attention was manipulated.

RESULTS

We found that in-phase tactile stimulation enhanced EEG responses to the tone, whereas anti-phase tactile stimulation suppressed these responses. No corresponding tactile effects on loudness-discrimination performance were observed in experiment 1. Using a yes/no paradigm in experiment 2, we found that in-phase tactile stimulation, but not anti-phase tactile stimulation, improved detection thresholds. Selective attention also improved thresholds but did not modulate the observed benefit from in-phase tactile stimulation.

CONCLUSIONS

Our study highlights that ongoing in-phase tactile input can enhance basic auditory processing as reflected in scalp EEG and detection thresholds. This might have implications for the development of hearing enhancement technologies and interventions.

An analytical method to separate modality-specific and nonspecific sensory components of event-related potentials

Several models have been developed to analyse cortical activity in response to salient events constituted by multiple sensory modalities. In particular, additive models compare event-related potentials (ERPs) in response to stimuli from two or more concomitant sensory modalities with the ERPs evoked by unimodal stimuli, in order to study sensory interactions. In this approach, components that are not specific to a sensory modality are commonly disregarded, although they likely carry information about stimulus expectation and evaluation, attentional orientation and other cognitive processes. In this study, we present an analytical method to assess the contribution of modality-specific and nonspecific components to the ERP. We developed an experimental setup that recorded ERPs in response to four stimulus types (visual, auditory, and two somatosensory modalities to test for stimulus specificity) in three different conditions (unimodal, bimodal and trimodal stimulation) and recorded the saliency of these stimuli relative to the sensory background. Stimuli were delivered in pairs, in order to study the effects of habituation. To this end, spatiotemporal features (peak amplitudes and latencies at different scalp locations) were analysed using linear mixed models. Results showed that saliency relative to the sensory background increased with the number of concomitant stimuli. We also observed that the spatiotemporal features of modality-specific components derived from this method likely reflect the amount and type of sensory input. Furthermore, the nonspecific component reflected habituation occurring for the second stimulus in the pair. In conclusion, this method provides an alternative to study neural mechanisms of responses to multisensory stimulation.

Sensory capability and information integration independently explain the cognitive status of healthy older adults

While there is evidence that sensory processing and multisensory integration change with age, links between these alterations and their relation to cognitive status remain unclear. In this study, we assessed sensory thresholds and performance of healthy younger and older adults in a visuotactile delayed match-to-sample task. Using Bayesian structural equation modelling (BSEM), we explored the factors explaining cognitive status in the group of older adults. Additionally, we applied transcranial alternating current stimulation (tACS) to a parieto-central network found to underlie visuotactile interactions and working memory matching in our previous work. Response times and signal detection measures indicated enhanced multisensory integration and enhanced benefit from successful working memory matching in older adults. Further, tACS caused a frequency-specific speeding (20 Hz) and delaying (70 Hz) of responses. Data exploration suggested distinct underlying factors for sensory acuity and sensitivity d’ on the one side, and multisensory and working memory enhancement on the other side. Finally, BSEM showed that these two factors labelled ‘sensory capability’ and ‘information integration’ independently explained cognitive status. We conclude that sensory decline and enhanced information integration might relate to distinct processes of ageing and discuss a potential role of the parietal cortex in mediating augmented integration in older adults.

Crossmodal Congruency Enhances Performance of Healthy Older Adults in Visual-Tactile Pattern Matching

One of the pivotal challenges of aging is to maintain independence in the activities of daily life. In order to adapt to changes in the environment, it is crucial to continuously process and accurately combine simultaneous input from different sensory systems, i.e., crossmodal or multisensory integration. With aging, performance decreases in multiple domains, affecting bottom-up sensory processing as well as top-down control. However, whether this decline leads to impairments in crossmodal interactions remains an unresolved question. While some researchers propose that crossmodal interactions degrade with age, others suggest that they are conserved or even gain compensatory importance. To address this question, we compared the behavioral performance of older and young participants in a well-established crossmodal matching task, requiring the evaluation of congruency in simultaneously presented visual and tactile patterns. Older participants performed significantly worse than young controls in the crossmodal task when being stimulated at their individual unimodal visual and tactile perception thresholds. Performance increased with adjustment of stimulus intensities. This improvement was driven by better detection of congruent stimulus pairs, while the detection of incongruent pairs was not significantly enhanced. These results indicate that age-related impairments lead to poor performance in complex crossmodal scenarios and demanding cognitive tasks. Crossmodal congruency effects attenuate the difficulties of older adults in visuotactile pattern matching and might be an important factor to drive the benefits of older adults demonstrated in various crossmodal integration scenarios. Congruency effects might, therefore, be used to develop strategies for cognitive training and neurological rehabilitation.

Synchronization of Sensory Gamma Oscillations Promotes Multisensory Communication

Rhythmic neuronal activity in the gamma range is a signature of cortical processing and its synchronization across distant sites has been proposed as a fundamental mechanism of network interactions. While this has been shown within sensory streams, we tested whether cross talk between the senses relies on similar mechanisms. Direct sensory interactions in humans (male and female) were studied with a visual-tactile amplitude matching paradigm. In this task, congruent stimuli are associated with behavioral benefits, which are proposed to be mediated by increased binding between sensory cortices through coherent gamma oscillations. We tested this hypothesis by applying 4-in-1 multi-electrode transcranial alternating current stimulation (tACS) with 40 Hz over visual and somatosensory cortices. In phase stimulation (0°) was expected to strengthen binding and thereby enhance the congruence effect, while anti-phase (180°) stimulation was expected to have opposite effects. Gamma tACS was controlled by alpha (10 Hz) and sham stimulation, as well as by applying tACS unilaterally while visual-tactile stimuli were presented lateralized. Contrary to our expectations, gamma tACS over the relevant hemisphere delayed responses to congruent trials. Additionally, reanalysis of EEG data revealed decoupling of sensory gamma oscillations during congruent trials. We propose that gamma tACS prevented sensory decoupling and thereby limited the congruence effect. Together, our results favor the perspective that processing multisensory congruence involves corticocortical communication rather than feature binding. Furthermore, we found control stimulation over the irrelevant hemisphere to speed responses under alpha stimulation and to delay responses under gamma stimulation, consistent with the idea that contralateral alpha/gamma dynamics regulate cortical excitability.

Semantic Incongruency Interferes With Endogenous Attention in Cross-Modal Integration of Semantically Congruent Objects

Efficient multisensory integration is often influenced by other cognitive processes including, but not limited to, semantic congruency and focused endogenous attention. Semantic congruency can re-allocate processing resources to the location of a congruent stimulus, while attention can prioritize the integration of multi-sensory stimuli under focus. Here, we explore the robustness of this phenomenon in the context of three stimuli, two of which are in the focus of endogenous attention. Participants completed an endogenous attention task with a stimulus compound consisting of 3 different objects: (1) a visual object (V) in the foreground, (2) an auditory object (A), and (3) a visual background scene object (B). Three groups of participants focused their attention on either the visual object and auditory sound (Group VA, n = 30), the visual object and the background (VB, n = 27), or the auditory sound and the background (AB, n = 30), and judged the semantic congruency of the objects under focus. Congruency varied systematically across all 3 stimuli: All stimuli could be semantically incongruent (e.g., V, ambulance; A, church bell; and B, swimming-pool) or all could be congruent (e.g., V, lion; A, roar; and B, savannah), or two objects could be congruent with the remaining one incongruent to the other two (e.g., V, duck; A, quack; and B, phone booth). Participants exhibited a distinct pattern of errors: when participants attended two congruent objects (e.g., group VA: V, lion; A, roar), in the presence of an unattended, incongruent third object (e.g., B, bath room) they tended to make more errors than in any other stimulus combination. Drift diffusion modeling of the behavioral data revealed a significantly smaller drift rate in two-congruent-attended condition, indicating slower evidence accumulation, which was likely due to interference from the unattended, incongruent object. Interference with evidence accumulation occurred independently of which pair of objects was in the focus of attention, which suggests that the vulnerability of congruency judgments to incongruent unattended distractors is not affected by sensory modalities. A control analysis ruled out the simple explanation of a negative response bias. These findings implicate that our perceptual system is highly sensitive to semantic incongruencies even when they are not endogenously attended.

Frontal and parietal alpha oscillations reflect attentional modulation of cross-modal matching

Multisensory perception is shaped by both attentional selection of relevant sensory inputs and exploitation of stimulus-driven factors that promote cross-modal binding. Underlying mechanisms of both top-down and bottom-up modulations have been linked to changes in alpha/gamma dynamics in primary sensory cortices and temporoparietal cortex. Accordingly, it has been proposed that alpha oscillations provide pulsed inhibition for gamma activity and thereby dynamically route cortical information flow. In this study, we employed a recently introduced multisensory paradigm incorporating both bottom-up and top-down aspects of cross-modal attention in an EEG study. The same trimodal stimuli were presented in two distinct attentional conditions, focused on visual-tactile or audio-visual components, for which cross-modal congruence of amplitude changes had to be evaluated. Neither top-down nor bottom-up cross-modal attention modulated alpha or gamma power in primary sensory cortices. Instead, we found alpha band effects in bilateral frontal and right parietal cortex. We propose that frontal alpha oscillations reflect the origin of top-down control regulating perceptual gains and that modulations of parietal alpha oscillations relates to intersensory re-orienting. Taken together, we suggest that the idea of selective cortical routing via alpha oscillations can be extended from sensory cortices to the frontoparietal attention network.

Auditory Stimulus Detection Partially Depends on Visuospatial Attentional Resources

Humans’ ability to detect relevant sensory information while being engaged in a demanding task is crucial in daily life. Yet, limited attentional resources restrict information processing. To date, it is still debated whether there are distinct pools of attentional resources for each sensory modality and to what extent the process of multisensory integration is dependent on attentional resources. We addressed these two questions using a dual task paradigm. Specifically, participants performed a multiple object tracking task and a detection task either separately or simultaneously. In the detection task, participants were required to detect visual, auditory, or audiovisual stimuli at varying stimulus intensities that were adjusted using a staircase procedure. We found that tasks significantly interfered. However, the interference was about 50% lower when tasks were performed in separate sensory modalities than in the same sensory modality, suggesting that attentional resources are partly shared. Moreover, we found that perceptual sensitivities were significantly improved for audiovisual stimuli relative to unisensory stimuli regardless of whether attentional resources were diverted to the multiple object tracking task or not. Overall, the present study supports the view that attentional resource allocation in multisensory processing is task-dependent and suggests that multisensory benefits are not dependent on attentional resources.