On the 'visual' in 'audio-visual integration': a hypothesis concerning visual pathways

Auditory, Visual, and Cross-Modal Mismatch Negativities in the Rat Auditory and Visual Cortices

When the brain tries to acquire an elaborate model of the world, multisensory integration should contribute to building predictions based on the various pieces of information, and deviance detection should repeatedly update these predictions by detecting “errors” from the actual sensory inputs. Accumulating evidence such as a hierarchical organization of the deviance-detection system indicates that the deviance-detection system can be interpreted in the predictive coding framework. Herein, we targeted mismatch negativity (MMN) as a type of prediction-error signal and investigated the relationship between multisensory integration and MMN. In particular, we studied whether and how cross-modal information processing affected MMN in rodents. We designed a new surface microelectrode array and simultaneously recorded visual and auditory evoked potentials from the visual and auditory cortices of rats under anesthesia. Then, we mapped MMNs for five types of deviant stimuli: single-modal deviants in (i) the visual oddball and (ii) auditory oddball paradigms, eliciting single-modal MMN; (iii) congruent audio-visual deviants, (iv) incongruent visual deviants, and (v) incongruent auditory deviants in the audio-visual oddball paradigm, eliciting cross-modal MMN. First, we demonstrated that visual MMN exhibited deviance detection properties and that the first-generation focus of visual MMN was localized in the visual cortex, as previously reported in human studies. Second, a comparison of MMN amplitudes revealed a non-linear relationship between single-modal and cross-modal MMNs. Moreover, congruent audio-visual MMN exhibited characteristics of both visual and auditory MMNs—its latency was similar to that of auditory MMN, whereas local blockage of N-methyl-D-aspartic acid receptors in the visual cortex diminished it as well as visual MMN. These results indicate that cross-modal information processing affects MMN without involving strong top-down effects, such as those of prior knowledge and attention. The present study is the first electrophysiological evidence of cross-modal MMN in animal models, and future studies on the neural mechanisms combining multisensory integration and deviance detection are expected to provide electrophysiological evidence to confirm the links between MMN and predictive coding theory.

Altered intra- and inter-network functional connectivity in patients with persistent postural-perceptual dizziness

•

Intra-network functional connectivity (FC) in the right precuneus within the posterior default mode network was decreased in PPPD patients.

•

Intra-network FC between the right precuneus and the bilateral precuneus and left premotor cortex was decreased.

•

Intra-network FC between the right precuneus and bilateral corpus callosum was enhanced.

•

Inter-network FC was increased between the visual network and auditory, sensorimotor networks.

•

FC changes were negatively correlated with dizziness handicap inventory functional scores.

Background

Persistent postural-perceptual dizziness (PPPD) is a functional vestibular disorder characterized by persistent dizziness, unsteadiness, and non-spinning vertigo. It is the most common cause of chronic vestibular syndrome, but its pathogenesis is currently unclear. Recent studies have indicated that sensory integration may be altered in PPPD patients.

Objective

Using independent component analysis (ICA) combined with seed-based functional connectivity analysis, we aimed to analyze changes in brain network functional connectivity (FC) in PPPD patients during the resting state and to explore the underlying pathogenesis of PPPD, particularly the abnormal integration of multiple sensations.

Methods

Study subjects included 12 PPPD patients and 12 healthy controls and were recruited from January to August 2018. Detailed medical data were collected from all participants. Vestibular function, neurological and medical examinations were conducted to exclude other diseases associated with chronic dizziness. Functional MRI was performed on all subjects. ICA and seed-based functional connectivity analysis were performed to examine changes in intra- and inter-network FC in PPPD patients.

Results

In total, 13 independent components were identified using ICA. Compared with healthy controls, PPPD patients showed decreased intra-network FC in the right precuneus within the posterior default mode network. Moreover, seed-based functional connectivity analysis showed decreased intra-network FC between the right precuneus and the bilateral precuneus and left premotor cortex, and enhanced FC between the right precuneus and bilateral corpus callosum. With respect to the inter-network, FC in PPPD patients was increased between the occipital pole visual network and auditory, sensorimotor networks, as well as the lateral visual and auditory networks. Additional analyses showed that FC changes were negatively correlated with dizziness handicap inventory functional scores.

Conclusion

In PPPD patients, dysfunction in the precuneus may cause abnormalities in external environment monitoring and in posture and movement regulation. Compensatory strategies may then be adopted to maintain balance. At the local level, information exchange between the two cerebral hemispheres is enhanced via the corpus callosum. At the whole brain level, through enhancement of functional activities of the visual network, the integration of multiple sensations and the regulation of posture and movement are primarily driven by visual information.

A comparative analysis of response times shows that multisensory benefits and interactions are not equivalent

Multisensory signals allow faster responses than the unisensory components. While this redundant signals effect (RSE) has been studied widely with diverse signals, no modelling approach explored the RSE systematically across studies. For a comparative analysis, here, we propose three steps: The first quantifies the RSE compared to a simple, parameter-free race model. The second quantifies processing interactions beyond the race mechanism: history effects and so-called violations of Miller's bound. The third models the RSE on the level of response time distributions using a context-variant race model with two free parameters that account for the interactions. Mimicking the diversity of studies, we tested different audio-visual signals that target the interactions using a 2 × 2 design. We show that the simple race model provides overall a strong prediction of the RSE. Regarding interactions, we found that history effects do not depend on low-level feature repetition. Furthermore, violations of Miller's bound seem linked to transient signal onsets. Critically, the latter dissociates from the RSE, demonstrating that multisensory interactions and multisensory benefits are not equivalent. Overall, we argue that our approach, as a blueprint, provides both a general framework and the precision needed to understand the RSE when studied across diverse signals and participant groups.

Audio-visual interaction in visual motion detection: Synchrony versus Asynchrony

OBJECTIVE

Detection and identification of moving targets is of paramount importance in everyday life, even if it is not widely tested in optometric practice, mostly for technical reasons. There are clear indications in the literature that in perception of moving targets, vision and hearing interact, for example in noisy surrounds and in understanding speech. The main aim of visual perception, the ability that optometry aims to optimize, is the identification of objects, from everyday objects to letters, but also the spatial orientation of subjects in natural surrounds. To subserve this aim, corresponding visual and acoustic features from the rich spectrum of signals supplied by natural environments have to be combined.

METHODS

Here, we investigated the influence of an auditory motion stimulus on visual motion detection, both with a concrete (left/right movement) and an abstract auditory motion (increase/decrease of pitch).

RESULTS

We found that incongruent audiovisual stimuli led to significantly inferior detection compared to the visual only condition. Additionally, detection was significantly better in abstract congruent than incongruent trials. For the concrete stimuli the detection threshold was significantly better in asynchronous audiovisual conditions than in the unimodal visual condition.

CONCLUSION

We find a clear but complex pattern of partly synergistic and partly inhibitory audio-visual interactions. It seems that asynchrony plays only a positive role in audiovisual motion while incongruence mostly disturbs in simultaneous abstract configurations but not in concrete configurations. As in speech perception in hearing-impaired patients, patients suffering from visual deficits should be able to benefit from acoustic information.

Spatiotemporal Relationships among Audiovisual Stimuli Modulate Auditory Facilitation of Visual Target Discrimination

Sensory information is multimodal; through audiovisual interaction, task-irrelevant auditory stimuli tend to speed response times and increase visual perception accuracy. However, mechanisms underlying these performance enhancements have remained unclear. We hypothesize that task-irrelevant auditory stimuli might provide reliable temporal and spatial cues for visual target discrimination and behavioral response enhancement. Using signal detection theory, the present study investigated the effects of spatiotemporal relationships on auditory facilitation of visual target discrimination. Three experiments were conducted where an auditory stimulus maintained reliable temporal and/or spatial relationships with visual target stimuli. Results showed that perception sensitivity (d') to visual target stimuli was enhanced only when a task-irrelevant auditory stimulus maintained reliable spatiotemporal relationships with a visual target stimulus. When only reliable spatial or temporal information was contained, perception sensitivity was not enhanced. These results suggest that reliable spatiotemporal relationships between visual and auditory signals are required for audiovisual integration during a visual discrimination task, most likely due to a spread of attention. These results also indicate that auditory facilitation of visual target discrimination follows from late-stage cognitive processes rather than early stage sensory processes.

Sounds can boost the awareness of visual events through attention without cross-modal integration

Cross-modal interactions can lead to enhancement of visual perception, even for visual events below awareness. However, the underlying mechanism is still unclear. Can purely bottom-up cross-modal integration break through the threshold of awareness? We used a binocular rivalry paradigm to measure perceptual switches after brief flashes or sounds which, sometimes, co-occurred. When flashes at the suppressed eye coincided with sounds, perceptual switches occurred the earliest. Yet, contrary to the hypothesis of cross-modal integration, this facilitation never surpassed the assumption of probability summation of independent sensory signals. A follow-up experiment replicated the same pattern of results using silent gaps embedded in continuous noise, instead of sounds. This manipulation should weaken putative sound-flash integration, although keep them salient as bottom-up attention cues. Additional results showed that spatial congruency between flashes and sounds did not determine the effectiveness of cross-modal facilitation, which was again not better than probability summation. Thus, the present findings fail to fully support the hypothesis of bottom-up cross-modal integration, above and beyond the independent contribution of two transient signals, as an account for cross-modal enhancement of visual events below level of awareness.

Age-related changes in auditory and visual interactions in temporal rate perception

We investigated how aging affects the integration of temporal rate for auditory flutter (amplitude modulation) presented with visual flicker. Since older adults were poorer at detecting auditory amplitude modulation, modulation depth was individually adjusted so that temporal rate was equally discriminable for 10 Hz flutter and flicker, thereby balancing the reliability of rate information available to each sensory modality. With age-related sensory differences normalized in this way, rate asynchrony skewed both auditory and visual rate judgments to the same extent in younger and older adults. Therefore, reliability-based weighting of temporal rate is preserved in older adults. Concurrent presentation of synchronous 10 Hz flicker and flutter improved temporal rate discrimination consistent with statistically optimal integration in younger but not older adults. In a control experiment, younger adults were presented with the same physical auditory stimulus as older adults. This time, rate asynchrony skewed perceived rate with greater auditory weighting rather than balanced integration. Taken together, our results indicate that integration of discrepant auditory and visual rates is not altered due to the healthy aging process once sensory deficits are accounted for, but that aging does abolish the minor improvement in discrimination performance seen in younger observers when concordant rates are integrated.

Effect of mechanical tactile noise on amplitude of visual evoked potentials: multisensory stochastic resonance

The present investigation documents the electrophysiological occurrence of multisensory stochastic resonance in the human visual pathway elicited by tactile noise. We define multisensory stochastic resonance of brain evoked potentials as the phenomenon in which an intermediate level of input noise of one sensory modality enhances the brain evoked response of another sensory modality. Here we examined this phenomenon in visual evoked potentials (VEPs) modulated by the addition of tactile noise. Specifically, we examined whether a particular level of mechanical Gaussian noise applied to the index finger can improve the amplitude of the VEP. We compared the amplitude of the positive P100 VEP component between zero noise (ZN), optimal noise (ON), and high mechanical noise (HN). The data disclosed an inverted U-like graph for all the subjects, thus demonstrating the occurrence of a multisensory stochastic resonance in the P100 VEP.