Visually induced gains in pitch discrimination: Linking audio-visual processing with auditory abilities

Laminar organization of visual responses in core and parabelt auditory cortex

Audiovisual (AV) interaction has been shown in many studies of auditory cortex. However, the underlying processes and circuits are unclear because few studies have used methods that delineate the timing and laminar distribution of net excitatory and inhibitory processes within areas, much less across cortical levels. This study examined laminar profiles of neuronal activity in auditory core (AC) and parabelt (PB) cortices recorded from macaques during active discrimination of conspecific faces and vocalizations. We found modulation of multi-unit activity (MUA) in response to isolated visual stimulation, characterized by a brief deep MUA spike, putatively in white matter, followed by mid-layer MUA suppression in core auditory cortex; the later suppressive event had clear current source density concomitants, while the earlier MUA spike did not. We observed a similar facilitation-suppression sequence in the PB, with later onset latency. In combined AV stimulation, there was moderate reduction of responses to sound during the visual-evoked MUA suppression interval in both AC and PB. These data suggest a common sequence of afferent spikes, followed by synaptic inhibition; however, differences in timing and laminar location may reflect distinct visual projections to AC and PB.

Musicians show more integrated neural processing of contextually relevant acoustic features

Little is known about expertise-related plasticity of neural mechanisms for auditory feature integration. Here, we contrast two diverging hypotheses that musical expertise is associated with more independent or more integrated predictive processing of acoustic features relevant to melody perception. Mismatch negativity (MMNm) was recorded with magnetoencephalography (MEG) from 25 musicians and 25 non-musicians, exposed to interleaved blocks of a complex, melody-like multi-feature paradigm and a simple, oddball control paradigm. In addition to single deviants differing in frequency (F), intensity (I), or perceived location (L), double and triple deviants were included reflecting all possible feature combinations (FI, IL, LF, FIL). Following previous work, early neural processing overlap was approximated in terms of MMNm additivity by comparing empirical MMNms obtained with double and triple deviants to modeled MMNms corresponding to summed constituent single-deviant MMNms. Significantly greater subadditivity was found in musicians compared to non-musicians, specifically for frequency-related deviants in complex, melody-like stimuli. Despite using identical sounds, expertise effects were absent from the simple oddball paradigm. This novel finding supports the integrated processing hypothesis whereby musicians recruit overlapping neural resources facilitating more integrative representations of contextually relevant stimuli such as frequency (perceived as pitch) during melody perception. More generally, these specialized refinements in predictive processing may enable experts to optimally capitalize upon complex, domain-relevant, acoustic cues.

Visual Influences on Auditory Behavioral, Neural, and Perceptual Processes: A Review

In a naturalistic environment, auditory cues are often accompanied by information from other senses, which can be redundant with or complementary to the auditory information. Although the multisensory interactions derived from this combination of information and that shape auditory function are seen across all sensory modalities, our greatest body of knowledge to date centers on how vision influences audition. In this review, we attempt to capture the state of our understanding at this point in time regarding this topic. Following a general introduction, the review is divided into 5 sections. In the first section, we review the psychophysical evidence in humans regarding vision's influence in audition, making the distinction between vision's ability to enhance versus alter auditory performance and perception. Three examples are then described that serve to highlight vision's ability to modulate auditory processes: spatial ventriloquism, cross-modal dynamic capture, and the McGurk effect. The final part of this section discusses models that have been built based on available psychophysical data and that seek to provide greater mechanistic insights into how vision can impact audition. The second section reviews the extant neuroimaging and far-field imaging work on this topic, with a strong emphasis on the roles of feedforward and feedback processes, on imaging insights into the causal nature of audiovisual interactions, and on the limitations of current imaging-based approaches. These limitations point to a greater need for machine-learning-based decoding approaches toward understanding how auditory representations are shaped by vision. The third section reviews the wealth of neuroanatomical and neurophysiological data from animal models that highlights audiovisual interactions at the neuronal and circuit level in both subcortical and cortical structures. It also speaks to the functional significance of audiovisual interactions for two critically important facets of auditory perception-scene analysis and communication. The fourth section presents current evidence for alterations in audiovisual processes in three clinical conditions: autism, schizophrenia, and sensorineural hearing loss. These changes in audiovisual interactions are postulated to have cascading effects on higher-order domains of dysfunction in these conditions. The final section highlights ongoing work seeking to leverage our knowledge of audiovisual interactions to develop better remediation approaches to these sensory-based disorders, founded in concepts of perceptual plasticity in which vision has been shown to have the capacity to facilitate auditory learning.

Perceptual timing precision with vibrotactile, auditory, and multisensory stimuli

The growing use of vibrotactile signaling devices makes it important to understand the perceptual limits on vibrotactile information processing. To promote that understanding, we carried out a pair of experiments on vibrotactile, auditory, and bimodal (synchronous vibrotactile and auditory) temporal acuity. On each trial, subjects experienced a set of isochronous, standard intervals (400 ms each), followed by one interval of variable duration (400 ± 1-80 ms). Intervals were demarcated by short vibrotactile, auditory, or bimodal pulses. Subjects categorized the timing of the last interval by describing the final pulse as either "early" or "late" relative to its predecessors. In Experiment 1, each trial contained three isochronous standard intervals, followed by an interval of variable length. In Experiment 2, the number of isochronous standard intervals per trial varied, from one to four. Psychometric modeling revealed that vibrotactile stimulation produced poorer temporal discrimination than either auditory or bimodal stimulation. Moreover, auditory signals dominated bimodal sensitivity, and inter-individual differences in temporal discriminability were reduced with bimodal stimulation. Additionally, varying the number of isochronous intervals in a trial failed to improve temporal sensitivity in either modality, suggesting that memory played a key role in judgments of interval duration.

Audiovisual structural connectivity in musicians and non-musicians: a cortical thickness and diffusion tensor imaging study

Our sensory systems provide complementary information about the multimodal objects and events that are the target of perception in everyday life. Professional musicians' specialization in the auditory domain is reflected in the morphology of their brains, which has distinctive characteristics, particularly in areas related to auditory and audio-motor activity. Here, we combined diffusion tensor imaging (DTI) with a behavioral measure of visually induced gain in pitch discrimination, and we used measures of cortical thickness (CT) correlations to assess how auditory specialization and musical expertise are reflected in the structural architecture of white and grey matter relevant to audiovisual processing. Across all participants (n = 45), we found a correlation (p < 0.001) between reliance on visual cues in pitch discrimination and the fractional anisotropy (FA) in the left inferior fronto-occipital fasciculus (IFOF), a structure connecting visual and auditory brain areas. Group analyses also revealed greater cortical thickness correlation between visual and auditory areas in non-musicians (n = 28) compared to musicians (n = 17), possibly reflecting musicians' auditory specialization (FDR < 10%). Our results corroborate and expand current knowledge of functional specialization with a specific focus on audition, and highlight the fact that perception is essentially multimodal while uni-sensory processing is a specialized task.

Visual and Auditory Information During Decision Making in Sport

In 2 experiments, the authors investigated the effects of bimodal integration in a sport-specific task. Beach volleyball players were required to make a tactical decision, responding either verbally or via a motor response, after being presented with visual, auditory, or both kinds of stimuli in a beach volleyball scenario. In Experiment 1, players made the correct decision in a game situation more often when visual and auditory information were congruent than in trials in which they experienced only one of the modalities or incongruent information. Decision-making accuracy was greater when motor, rather than verbal, responses were given. Experiment 2 replicated this congruence effect using different stimulus material and showed a decreasing effect of visual stimulation on decision making as a function of shorter visual stimulus durations. In conclusion, this study shows that bimodal integration of congruent visual and auditory information results in more accurate decision making in sport than unimodal information.