Active learning of novel sound-producing objects: motor reactivation and enhancement of visuo-motor connectivity

A meta-analysis of letter-sound integration: Assimilation and accommodation in the superior temporal gyrus

Despite being a relatively new cultural phenomenon, the ability to perform letter-sound integration is readily acquired even though it has not had time to evolve in the brain. Leading theories of how the brain accommodates literacy acquisition include the neural recycling hypothesis and the assimilation-accommodation hypothesis. The neural recycling hypothesis proposes that a new cultural skill is developed by "invading" preexisting neural structures to support a similar cognitive function, while the assimilation-accommodation hypothesis holds that a new cognitive skill relies on direct invocation of preexisting systems (assimilation) and adds brain areas based on task requirements (accommodation). Both theories agree that letter-sound integration may be achieved by reusing pre-existing functionally similar neural bases, but differ in their proposals of how this occurs. We examined the evidence for each hypothesis by systematically comparing the similarities and differences between letter-sound integration and two other types of preexisting and functionally similar audiovisual (AV) processes, namely object-sound and speech-sound integration, by performing an activation likelihood estimation (ALE) meta-analysis. All three types of AV integration recruited the left posterior superior temporal gyrus (STG), while speech-sound integration additionally activated the bilateral middle STG and letter-sound integration directly invoked the AV areas involved in speech-sound integration. These findings suggest that letter-sound integration may reuse the STG for speech-sound and object-sound integration through an assimilation-accommodation mechanism.

The importance of covert memory consolidation in schizophrenia: Dysfunctional network profiles of the hippocampus and the dorsolateral prefrontal cortex

Altered brain network profiles in schizophrenia (SCZ) during memory consolidation are typically observed during task-active periods such as encoding or retrieval. However active processes are also sub served by covert periods of memory consolidation. These periods are active in that they allow memories to be recapitulated even in the absence of overt sensorimotor processing. It is plausible that regions central to memory formation like the dlPFC and the hippocampus, exert network signatures during covert periods. Are these signatures altered in patients? The question is clinically relevant because real world learning and memory is facilitated by covert processing, and may be impaired in schizophrenia. Here, we compared network signatures of the dlPFC and the hippocampus during covert periods of a learning and memory task. Because behavioral proficiency increased non-linearly, functional connectivity of the dlPFC and hippocampus [psychophysiological interaction (PPI)] was estimated for each of the Early (linear increases in performance) and Late (asymptotic performance) covert periods. During Early periods, we observed hypo-modulation by the hippocampus but hyper-modulation by dlPFC. Conversely, during Late periods, we observed hypo-modulation by both the dlPFC and the hippocampus. We stitch these results into a conceptual model of network deficits during covert periods of memory consolidation.

Enriched learning: behavior, brain, and computation

The presence of complementary information across multiple sensory or motor modalities during learning, referred to as multimodal enrichment, can markedly benefit learning outcomes. Why is this? Here, we integrate cognitive, neuroscientific, and computational approaches to understanding the effectiveness of enrichment and discuss recent neuroscience findings indicating that crossmodal responses in sensory and motor brain regions causally contribute to the behavioral benefits of enrichment. The findings provide novel evidence for multimodal theories of enriched learning, challenge assumptions of longstanding cognitive theories, and provide counterevidence to unimodal neurobiologically inspired theories. Enriched educational methods are likely effective not only because they may engage greater levels of attention or deeper levels of processing, but also because multimodal interactions in the brain can enhance learning and memory.

Real-Life Neuroscience: An Ecological Approach to Brain and Behavior Research

Owing to advances in neuroimaging technology, the past couple of decades have witnessed a surge of research on brain mechanisms that underlie human cognition. Despite the immense development in cognitive neuroscience, the vast majority of neuroimaging experiments examine isolated agents carrying out artificial tasks in sensory and socially deprived environments. Thus, the understanding of the mechanisms of various domains in cognitive neuroscience, including social cognition and episodic memory, is sorely lacking. Here we focus on social and memory research as representatives of cognitive functions and propose that mainstream, lab-based experimental designs in these fields suffer from two fundamental limitations, pertaining to person-dependent and situation-dependent factors. The person-dependent factor addresses the issue of limiting the active role of the participants in lab-based paradigms that may interfere with their sense of agency and embodiment. The situation-dependent factor addresses the issue of the artificial decontextualized environment in most available paradigms. Building on recent findings showing that real-life as opposed to controlled experimental paradigms involve different mechanisms, we argue that adopting a real-life approach may radically change our understanding of brain and behavior. Therefore, we advocate in favor of a paradigm shift toward a nonreductionist approach, exploiting portable technology in semicontrolled environments, to explore behavior in real life.

Learning to Expect: Predicting Sounds During Movement Is Related to Sensorimotor Association During Listening

Sensory experiences, such as sound, often result from our motor actions. Over time, repeated sound-producing performance can generate sensorimotor associations. However, it is not clear how sensory and motor information are associated. Here, we explore if sensory prediction is associated with the formation of sensorimotor associations during a learning task. We recorded event-related potentials (ERPs) while participants produced index and little finger-swipes on a bespoke device, generating novel sounds. ERPs were also obtained as participants heard those sounds played back. Peak suppression was compared to assess sensory prediction. Additionally, transcranial magnetic stimulation (TMS) was used during listening to generate finger-motor evoked potentials (MEPs). MEPs were recorded before and after training upon hearing these sounds, and then compared to reveal sensorimotor associations. Finally, we explored the relationship between these components. Results demonstrated that an increased positive-going peak (e.g., P2) and a suppressed negative-going peak (e.g., N2) were recorded during action, revealing some sensory prediction outcomes (P2: p = 0.050, ηp2 = 0.208; N2: p = 0.001, ηp2 = 0.474). Increased MEPs were also observed upon hearing congruent sounds compared with incongruent sounds (i.e., associated to a finger), demonstrating precise sensorimotor associations that were not present before learning (Index finger: p < 0.001, ηp2 = 0.614; Little finger: p < 0.001, ηp2 = 0.529). Consistent with our broad hypotheses, a negative association between the MEPs in one finger during listening and ERPs during performance of the other was observed (Index finger MEPs and Fz N1 action ERPs; r = −0.655, p = 0.003). Overall, data suggest that predictive mechanisms are associated with the fine-tuning of sensorimotor associations.

Functional fixedness in tool use: Learning modality, limitations and individual differences

Functional fixedness is a cognitive bias that describes how previous knowledge of a tool's function can negatively impact the use of this tool in novel contexts. As such, functional fixedness disturbs the use of tools during mechanical problem solving. Little is known about whether this bias emerges from different experiences with tools, whether it occurs regardless of problem difficulty, or whether there are protective factors against it. To resolve the first issue, we created five experimental groups: Reading (R), Video (V), Manual (M), No Functional Fixedness (NFF), and No Training (NT). The R group learned to use tools by reading a description of their use, the V group by watching an instructional video, and the M group through direct instruction and active manipulation of the tools. To resolve the remaining two issues, we created mechanical puzzles of distinct difficulty and used tests of intuitive physics, fine motor skills, and creativity. Results showed that misleading functional knowledge is at the core of functional fixedness, and that this bias generates cognitive impasses in simple puzzles, but it does not play a role in higher difficulty problems. Additionally, intuitive physics and motor skills were protective factors against its emergence, but creativity did not influence it. Although functional fixedness leads to inaccurate problem solving, our results suggest that its effects are more limited than previously assumed.

Gesture for generalization: gesture facilitates flexible learning of words for actions on objects

Verb learning is difficult for children (Gentner, ), partially because children have a bias to associate a novel verb not only with the action it represents, but also with the object on which it is learned (Kersten & Smith, ). Here we investigate how well 4- and 5-year-old children (N = 48) generalize novel verbs for actions on objects after doing or seeing the action (e.g., twisting a knob on an object) or after doing or seeing a gesture for the action (e.g., twisting in the air near an object). We find not only that children generalize more effectively through gesture experience, but also that this ability to generalize persists after a 24-hour delay.

"Low road" to rehabilitation: a perspective on subliminal sensory neuroprosthetics

Fear can propagate parallelly through both cortical and subcortical pathways. It can instigate memory consolidation habitually and might allow internal simulation of movements independent of the cortical structures. This perspective suggests delivery of subliminal, aversive and kinematic audiovisual stimuli via neuroprosthetics in patients with neocortical dysfunctions. We suggest possible scenarios by which these stimuli might bypass damaged neocortical structures and possibly assisting in motor relearning. Anticipated neurophysiological mechanisms and methodological scenarios have been discussed in this perspective. This approach introduces novel perspectives into neuropsychology as to how subcortical pathways might be used to induce motor relearning.

The Importance of Handwriting Experience on the Development of the Literate Brain

Handwriting experience can have significant effects on the ability of young children to recognize letters. Why handwriting has this facilitative effect and how this is accomplished were explored in a series of studies using overt behavioral measures and functional neuroimaging of the brain in 4- to 5-year-old children. My colleagues and I showed that early handwriting practice affects visual symbol recognition because it results in the production of variable visual forms that aid in symbol understanding. Further, the mechanisms that support this understanding lay in the communication between visual and motor systems in the brain: Handwriting serves to link visual processing with motor experience, facilitating subsequent letter recognition skills. These results are interpreted in the larger context of the facilitatory effect that learning through action has on perceptual capabilities.

The effects of auditory and visual cues on timing synchronicity for robotic rehabilitation

In this paper, we explore how the integration of auditory and visual cues can help teach the timing of motor skills for the purpose of motor function rehabilitation. We conducted a study using Amazon's Mechanical Turk in which 106 participants played a virtual therapy game requiring wrist movements. To validate that our results would translate to trends that could also be observed during robotic rehabilitation sessions, we recreated this experiment with 11 participants using a robotic wrist rehabilitation system as means to control the therapy game. During interaction with the therapy game, users were asked to learn and reconstruct a tapping sequence as defined by musical notes flashing on the screen. Participants were divided into 2 test groups: (1) control: participants only received visual cues to prompt them on the timing sequence, and (2) experimental: participants received both visual and auditory cues to prompt them on the timing sequence. To evaluate performance, the timing and length of the sequence were measured. Performance was determined by calculating the number of trials needed before the participant was able to master the specific aspect of the timing task. In the virtual experiment, the group that received visual and auditory cues was able to master all aspects of the timing task faster than the visual cue only group with p-values < 0.05. This trend was also verified for participants using the robotic arm exoskeleton in the physical experiment.

Neural evidence for phonologically based language production deficits in older adults: An fMRI investigation of age-related differences in picture-word interference

INTRODUCTION

Older adults often show declines in phonological aspects of language production, particularly for low-frequency words, but maintain strong semantic systems. However, there are different theories about the mechanism that may underlie such age-related differences in language (e.g., age-related declines in transmission of activation or inhibition).

METHODS

This study used fMRI to investigate whether age-related differences in language production are associated with transmission deficits or inhibition deficits. We used the picture-word interference paradigm to examine age-related differences in picture naming as a function of both target frequency and the relationship between the target picture and distractor word.

RESULTS

We found that the presence of a categorically related distractor led to greater semantic elaboration by older adults compared to younger adults, as evidenced by older adults' increased recruitment of regions including the left middle frontal gyrus and bilateral precuneus. When presented with a phonologically related distractor, patterns of neural activation are consistent with previously observed age deficits in phonological processing, including age-related reductions in the recruitment of regions such as the left middle temporal gyrus and right supramarginal gyrus. Lastly, older, but not younger, adults show increased brain activation of the pre- and postcentral gyri as a function of decreasing target frequency when target pictures are paired with a phonological distractor, suggesting that cuing the phonology of the target disproportionately aids production of low-frequency items.

CONCLUSIONS

Overall, this pattern of results is generally consistent with the transmission deficit hypothesis, illustrating that links within the phonological system, but not the semantic system, are weakened with age.

Attention Cueing and Activity Equally Reduce False Alarm Rate in Visual-Auditory Associative Learning through Improving Memory

In our daily life, we continually exploit already learned multisensory associations and form new ones when facing novel situations. Improving our associative learning results in higher cognitive capabilities. We experimentally and computationally studied the learning performance of healthy subjects in a visual-auditory sensory associative learning task across active learning, attention cueing learning, and passive learning modes. According to our results, the learning mode had no significant effect on learning association of congruent pairs. In addition, subjects' performance in learning congruent samples was not correlated with their vigilance score. Nevertheless, vigilance score was significantly correlated with the learning performance of the non-congruent pairs. Moreover, in the last block of the passive learning mode, subjects significantly made more mistakes in taking non-congruent pairs as associated and consciously reported lower confidence. These results indicate that attention and activity equally enhanced visual-auditory associative learning for non-congruent pairs, while false alarm rate in the passive learning mode did not decrease after the second block. We investigated the cause of higher false alarm rate in the passive learning mode by using a computational model, composed of a reinforcement learning module and a memory-decay module. The results suggest that the higher rate of memory decay is the source of making more mistakes and reporting lower confidence in non-congruent pairs in the passive learning mode.

Active Drumming Experience Increases Infants' Sensitivity to Audiovisual Synchrony during Observed Drumming Actions

In the current study, we examined the role of active experience on sensitivity to multisensory synchrony in six-month-old infants in a musical context. In the first of two experiments, we trained infants to produce a novel multimodal effect (i.e., a drum beat) and assessed the effects of this training, relative to no training, on their later perception of the synchrony between audio and visual presentation of the drumming action. In a second experiment, we then contrasted this active experience with the observation of drumming in order to test whether observation of the audiovisual effect was as effective for sensitivity to multimodal synchrony as active experience. Our results indicated that active experience provided a unique benefit above and beyond observational experience, providing insights on the embodied roots of (early) music perception and cognition.

Atypical multisensory integration in Niemann-Pick type C disease - towards potential biomarkers

Background

Niemann-Pick type C (NPC) is an autosomal recessive disease in which cholesterol and glycosphingolipids accumulate in lysosomes due to aberrant cell-transport mechanisms. It is characterized by progressive and ultimately terminal neurological disease, but both pre-clinical studies and direct human trials are underway to test the safety and efficacy of cholesterol clearing compounds, with good success already observed in animal models. Key to assessing the effectiveness of interventions in patients, however, is the development of objective neurobiological outcome measures. Multisensory integration mechanisms present as an excellent candidate since they necessarily rely on the fidelity of long-range neural connections between the respective sensory cortices (e.g. the auditory and visual systems).

Methods

A simple way to test integrity of the multisensory system is to ask whether individuals respond faster to the occurrence of a bisensory event than they do to the occurrence of either of the unisensory constituents alone. Here, we presented simple auditory, visual, and audio-visual stimuli in random sequence. Participants responded as fast as possible with a button push. One 11-year-old and two 14-year-old boys with NPC participated in the experiment and their results were compared to those of 35 age-matched neurotypical boys.

Results

Reaction times (RTs) to the stimuli when presented simultaneously were significantly faster than when they were presented alone in the neurotypical children, a facilitation that could not be accounted for by probability summation, as evidenced by violation of the so-called ‘race’ model. In stark contrast, the NPC boys showed no such speeding, despite the fact that their unisensory RTs fell within the distribution of RTs observed in the neurotypicals.

Conclusions

These results uncover a previously undescribed deficit in multisensory integrative abilities in NPC, with implications for ongoing treatment of the clinical symptoms of these children. They also suggest that multisensory processes may represent a good candidate biomarker against which to test the efficacy of therapeutic interventions.

Electronic supplementary material

The online version of this article (doi:10.1186/s13023-014-0149-x) contains supplementary material, which is available to authorized users.

Tones and numbers: a combined EEG-MEG study on the effects of musical expertise in magnitude comparisons of audiovisual stimuli

This study investigated the cortical responses underlying magnitude comparisons of multisensory stimuli and examined the effect that musical expertise has in this process. The comparative judgments were based on a newly learned rule binding the auditory and visual stimuli within the context of magnitude comparisons: "the higher the pitch of the tone, the larger the number presented." The cortical responses were measured by simultaneous MEG\EEG recordings and a combined source analysis with individualized realistic head models was performed. Musical expertise effects were investigated by comparing musicians to non-musicians. Congruent audiovisual stimuli, corresponding to the newly learned rule, elicited activity in frontotemporal and occipital areas. In contrast, incongruent stimuli activated temporal and parietal regions. Musicians when compared with nonmusicians showed increased differences between congruent and incongruent stimuli in a prefrontal region, thereby indicating that music expertise may affect multisensory comparative judgments within a generalized representation of analog magnitude.

Multisensory integration and neuroplasticity in the human cerebral cortex

There is a strong interaction between multisensory processing and the neuroplasticity of the human brain. On one hand, recent research demonstrates that experience and training in various domains modifies how information from the different senses is integrated; and, on the other hand multisensory training paradigms seem to be particularly effective in driving functional and structural plasticity. Multisensory training affects early sensory processing within separate sensory domains, as well as the functional and structural connectivity between uni- and multisensory brain regions. In this review, we discuss the evidence for interactions of multisensory processes and brain plasticity and give an outlook on promising clinical applications and open questions.