Neural correlates of audiovisual integration in music reading

The impact of musical training in symbolic and non-symbolic audiovisual judgements of magnitude

Quantity estimation can be represented in either an analog or symbolic manner and recent evidence now suggests that analog and symbolic representation of quantities interact. Nonetheless, those two representational forms of quantities may be enhanced by convergent multisensory information. Here, we elucidate those interactions using high-density electroencephalography (EEG) and an audiovisual oddball paradigm. Participants were presented simultaneous audiovisual tokens in which the co-varying pitch of tones was combined with the embedded cardinality of dot patterns. Incongruencies were elicited independently from symbolic and non-symbolic modality within the audio-visual percept, violating the newly acquired rule that “the higher the pitch of the tone, the larger the cardinality of the figure.” The effect of neural plasticity in symbolic and non-symbolic numerical representations of quantities was investigated through a cross-sectional design, comparing musicians to musically naïve controls. Individual’s cortical activity was reconstructed and statistically modeled for a predefined time-window of the evoked response (130–170 ms). To summarize, we show that symbolic and non-symbolic processing of magnitudes is re-organized in cortical space, with professional musicians showing altered activity in motor and temporal areas. Thus, we argue that the symbolic representation of quantities is altered through musical training.

Post-Attentive Integration and Topographic Map Distribution During Audiovisual Processing in Dyslexia: A P300 Event-Related Component Analysis

Background

Research on audiovisual post-attentive integration has been carried out using a variety of experimental paradigms and experimental groups but not yet studied in dyslexia. We investigated post-attentive integration and topographic voltage distribution in children with dyslexia by analysing the P300 event-related potential (ERP) component.

Methods

We used a 128-child ERP net for the ERP experiment. Two types of stimuli were presented as either congruent or incongruent stimuli. Congruent stimuli included a matching auditory sound with an animal image, whereas incongruent stimuli included unmatched animal sounds. A total of 24 age-matched children were recruited in the control (n = 12) and dyslexia (n = 12) groups. Children pressed button ‘1’ or ‘2’ when presented with congruent or incongruent stimuli, respectively. The P300 amplitudes and latencies with topographic voltage distribution were analysed for both groups.

Results

The dyslexia group evoked significantly higher P300 amplitudes at the T4 area than the control group. No significant differences were found in cases of P300 latency. Moreover, the dyslexia group demonstrated a higher intensity of P300 voltage distribution in the right parietal and left occipital areas than the control group.

Conclusion

Post-attentive integration for children with dyslexia is higher and that this integration process implicated the parietal and occipital areas.

Auditory but Not Audiovisual Cues Lead to Higher Neural Sensitivity to the Statistical Regularities of an Unfamiliar Musical Style

It is still a matter of debate whether visual aids improve learning of music. In a multisession study, we investigated the neural signatures of novel music sequence learning with or without aids (auditory-only: AO, audiovisual: AV). During three training sessions on three separate days, participants (nonmusicians) reproduced (note by note on a keyboard) melodic sequences generated by an artificial musical grammar. The AV group (n = 20) had each note color-coded on screen, whereas the AO group (n = 20) had no color indication. We evaluated learning of the statistical regularities of the novel music grammar before and after training by presenting melodies ending on correct or incorrect notes and by asking participants to judge the correctness and surprisal of the final note, while EEG was recorded. We found that participants successfully learned the new grammar. Although the AV group, as compared to the AO group, reproduced longer sequences during training, there was no significant difference in learning between groups. At the neural level, after training, the AO group showed a larger N100 response to low-probability compared with high-probability notes, suggesting an increased neural sensitivity to statistical properties of the grammar; this effect was not observed in the AV group. Our findings indicate that visual aids might improve sequence reproduction while not necessarily promoting better learning, indicating a potential dissociation between sequence reproduction and learning. We suggest that the difficulty induced by auditory-only input during music training might enhance cognitive engagement, thereby improving neural sensitivity to the underlying statistical properties of the learned material.

Visually induced gains in pitch discrimination: Linking audio-visual processing with auditory abilities

Perception is fundamentally a multisensory experience. The principle of inverse effectiveness (PoIE) states how the multisensory gain is maximal when responses to the unisensory constituents of the stimuli are weak. It is one of the basic principles underlying multisensory processing of spatiotemporally corresponding crossmodal stimuli that are well established at behavioral as well as neural levels. It is not yet clear, however, how modality-specific stimulus features influence discrimination of subtle changes in a crossmodally corresponding feature belonging to another modality. Here, we tested the hypothesis that reliance on visual cues to pitch discrimination follow the PoIE at the interindividual level (i.e., varies with varying levels of auditory-only pitch discrimination abilities). Using an oddball pitch discrimination task, we measured the effect of varying visually perceived vertical position in participants exhibiting a wide range of pitch discrimination abilities (i.e., musicians and nonmusicians). Visual cues significantly enhanced pitch discrimination as measured by the sensitivity index d', and more so in the crossmodally congruent than incongruent condition. The magnitude of gain caused by compatible visual cues was associated with individual pitch discrimination thresholds, as predicted by the PoIE. This was not the case for the magnitude of the congruence effect, which was unrelated to individual pitch discrimination thresholds, indicating that the pitch-height association is robust to variations in auditory skills. Our findings shed light on individual differences in multisensory processing by suggesting that relevant multisensory information that crucially aids some perceivers' performance may be of less importance to others, depending on their unisensory abilities.

Cross-modal music integration in expert memory: Evidence from eye movements

The study investigated the cross-modal integration hypothesis for expert musicians using eye tracking. Twenty randomized excerpts of classical music were presented in two modes (auditory and visual), at the same time (simultaneously) or successively (sequentially). Musicians (N = 53, 26 experts and 27 non-experts) were asked to detect a note modified between the auditory and visual versions, either in the same major/minor key or violating the key. Experts carried out the task faster and with greater accuracy than non-experts. Sequential presentation was more difficult than simultaneous (longer fixations and higher error rates) and the modified notes were more easily detected when violating the key (fewer errors), but with longer fixations (speed/accuracy trade-off strategy). Experts detected the modified note faster, especially in the simultaneous condition in which cross-modal integration may be applied. These results support the hypothesis that the main difference between experts and non-experts derives from the difference in knowledge structures in memory built over time with practice. They also suggest that these high-level knowledge structures in memory contain harmony and tonal rules, arguing in favour of cross-modal integration capacities for experts, which are related to and can be explained by the long-term working memory (LTWM) model of expert memory (e.g. (18; 22).

Causal inference and temporal predictions in audiovisual perception of speech and music

To form a coherent percept of the environment, the brain must integrate sensory signals emanating from a common source but segregate those from different sources. Temporal regularities are prominent cues for multisensory integration, particularly for speech and music perception. In line with models of predictive coding, we suggest that the brain adapts an internal model to the statistical regularities in its environment. This internal model enables cross-sensory and sensorimotor temporal predictions as a mechanism to arbitrate between integration and segregation of signals from different senses.

Reorganization of the thalamocortical network in musicians

The cortico-thalamocortical network is relevant to music performance in that the network can regulate sensitivity to afferent input or sound, mediate the integration of multimodal information required for the performance, and play a role in skilled performance control. We, therefore, predicted that this network would be reorganized via musical training-induced neuroplasticity. To test this hypothesis, we analyzed resting-state functional connectivity of the thalamocortical network in musicians (n=35) and nonmusicians (n=35). The seed-to-voxel functional connectivity analysis of the left thalamus seed showed enhanced connectivity voxels in the precuneus/posterior cingulate cortex (PCC) in musicians compared with nonmusicians. Region of interest (ROI)-to-ROI functional connectivity analysis showed that the auditory areas were also more strongly connected with the left thalamus in musicians. Discriminant analysis using the ROI-to-ROI functional connectivity data of the precuneus/PCC and auditory areas as predictors yielded an 87% correct discrimination of musicians from nonmusicians. Therefore, we can conclude that, as a consequence of long-term musical training, musicians have a characteristically organized thalamocortical network. The precuneus and PCC are principal nodes of the default mode network and play a pivotal role in the manipulation of mental imagery. We propose that the reorganized thalamocortical network in musicians contributes not only to higher sensitivity to sound but also to the integration of mental imagery with sound, which are both presumed to be important for better music performance.