Touch, sound and vision in human superior temporal sulcus

Neural correlates of taste reactivity in autism spectrum disorder

Selective or 'picky' eating habits are common among those with autism spectrum disorder (ASD). These behaviors are often related to aberrant sensory experience in individuals with ASD, including heightened reactivity to food taste and texture. However, very little is known about the neural mechanisms that underlie taste reactivity in ASD. In the present study, food-related neural responses were evaluated in 21 young adult and adolescent males diagnosed with ASD without intellectual disability, and 21 typically-developing (TD) controls. Taste reactivity was assessed using the Adolescent/Adult Sensory Profile, a clinical self-report measure. Functional magnetic resonance imaging was used to evaluate hemodynamic responses to sweet (vs. neutral) tastants and food pictures. Subjects also underwent resting-state functional connectivity scans.The ASD and TD individuals did not differ in their hemodynamic response to gustatory stimuli. However, the ASD subjects, but not the controls, exhibited a positive association between self-reported taste reactivity and the response to sweet tastants within the insular cortex and multiple brain regions associated with gustatory perception and reward. There was a strong interaction between diagnostic group and taste reactivity on tastant response in brain regions associated with ASD pathophysiology, including the bilateral anterior superior temporal sulcus (STS). This interaction of diagnosis and taste reactivity was also observed in the resting state functional connectivity between the anterior STS and dorsal mid-insula (i.e., gustatory cortex).These results suggest that self-reported heightened taste reactivity in ASD is associated with heightened brain responses to food-related stimuli and atypical functional connectivity of primary gustatory cortex, which may predispose these individuals to maladaptive and unhealthy patterns of selective eating behavior.

Trial registration

(clinicaltrials.gov identifier) NCT01031407. Registered: December 14, 2009.

Function-specific and Enhanced Brain Structural Connectivity Mapping via Joint Modeling of Diffusion and Functional MRI

A joint structural-functional brain network model is presented, which enables the discovery of function-specific brain circuits, and recovers structural connections that are under-estimated by diffusion MRI (dMRI). Incorporating information from functional MRI (fMRI) into diffusion MRI to estimate brain circuits is a challenging task. Usually, seed regions for tractography are selected from fMRI activation maps to extract the white matter pathways of interest. The proposed method jointly analyzes whole brain dMRI and fMRI data, allowing the estimation of complete function-specific structural networks instead of interactively investigating the connectivity of individual cortical/sub-cortical areas. Additionally, tractography techniques are prone to limitations, which can result in erroneous pathways. The proposed framework explicitly models the interactions between structural and functional connectivity measures thereby improving anatomical circuit estimation. Results on Human Connectome Project (HCP) data demonstrate the benefits of the approach by successfully identifying function-specific anatomical circuits, such as the language and resting-state networks. In contrast to correlation-based or independent component analysis (ICA) functional connectivity mapping, detailed anatomical connectivity patterns are revealed for each functional module. Results on a phantom (Fibercup) also indicate improvements in structural connectivity mapping by rejecting false-positive connections with insufficient support from fMRI, and enhancing under-estimated connectivity with strong functional correlation.

Different neural substrates for precision stepping and fast online step adjustments in youth

Humans can navigate through challenging environments (e.g., cluttered or uneven terrains) by modifying their preferred gait pattern (e.g., step length, step width, or speed). Growing behavioral and neuroimaging evidence suggests that the ability to modify preferred step patterns requires the recruitment of cognitive resources. In children, it is argued that prolonged development of complex gait is related to the ongoing development of involved brain regions, but this has not been directly investigated yet. Here, we aimed to elucidate the relationship between structural brain properties and complex gait in youth aged 9–18 years. We used volumetric analyses of cortical grey matter (GM) and whole-brain voxelwise statistical analyses of white matter (WM), and utilized a treadmill-based precision stepping task to investigate complex gait. Moreover, precision stepping was performed on step targets which were either unperturbed or perturbed (i.e., unexpectedly shifting to a new location). Our main findings revealed that larger unperturbed precision step error was associated with decreased WM microstructural organization of tracts that are particularly associated with attentional and visual processing functions. These results strengthen the hypothesis that precision stepping on unperturbed step targets is driven by cortical processes. In contrast, no significant correlations were found between perturbed precision stepping and cortical structures, indicating that other (neural) mechanisms may be more important for this type of stepping.

Taking two to tango: fMRI analysis of improvised joint action with physical contact

Many forms of joint action involve physical coupling between the participants, such as when moving a sofa together or dancing a tango. We report the results of a novel two-person functional MRI study in which trained couple dancers engaged in bimanual contact with an experimenter standing next to the bore of the magnet, and in which the two alternated between being the leader and the follower of joint improvised movements. Leading showed a general pattern of self-orientation, being associated with brain areas involved in motor planning, navigation, sequencing, action monitoring, and error correction. In contrast, following showed a far more sensory, externally-oriented pattern, revealing areas involved in somatosensation, proprioception, motion tracking, social cognition, and outcome monitoring. We also had participants perform a "mutual" condition in which the movement patterns were pre-learned and the roles were symmetric, thereby minimizing any tendency toward either leading or following. The mutual condition showed greater activity in brain areas involved in mentalizing and social reward than did leading or following. Finally, the analysis of improvisation revealed the dual importance of motor-planning and working-memory areas. We discuss these results in terms of theories of both joint action and improvisation.

Processing of Egomotion-Consistent Optic Flow in the Rhesus Macaque Cortex

The cortical network that processes visual cues to self-motion was characterized with functional magnetic resonance imaging in 3 awake behaving macaques. The experimental protocol was similar to previous human studies in which the responses to a single large optic flow patch were contrasted with responses to an array of 9 similar flow patches. This distinguishes cortical regions where neurons respond to flow in their receptive fields regardless of surrounding motion from those that are sensitive to whether the overall image arises from self-motion. In all 3 animals, significant selectivity for egomotion-consistent flow was found in several areas previously associated with optic flow processing, and notably dorsal middle superior temporal area, ventral intra-parietal area, and VPS. It was also seen in areas 7a (Opt), STPm, FEFsem, FEFsac and in a region of the cingulate sulcus that may be homologous with human area CSv. Selectivity for egomotion-compatible flow was never total but was particularly strong in VPS and putative macaque CSv. Direct comparison of results with the equivalent human studies reveals several commonalities but also some differences.

Brain mechanisms for processing discriminative and affective touch in 7-month-old infants

Affective touch has been associated with affiliative behavior during early stages of infant development; however, its underlying brain mechanisms are still poorly understood. This study used fNIRS (functional near-infrared spectroscopy) to examine both affective and discriminative touch in 7- month-old infants (n=35). Infants were provided affective stimuli on the forearm for 10 sec followed by a 20 sec rest period. The protocol was repeated for discriminative touch, and both affective and discriminative stimuli were given in a counterbalanced order. Brain activation (oxy-hemoglobin and deoxy-hemoglobin levels) in the somatosensory and temporal regions was registered during administration of the stimuli. There was an increase in oxy-hemoglobin and decrease in deoxy-hemoglobin only in the somatosensory region in response to both affective and discriminative touch. No other activations were found. Seven-month-old infants’ brain activation in the somatosensory cortex was similar for both discriminative and affective touch, but the stimuli did not elicit any activation in the temporal region/ pSTS. Our study is the first to suggest that 7-month-old infants do not yet recruit socio-emotional brain areas in response to affective touch.

Cortical multisensory connectivity is present near birth in humans

How the newborn brain adapts to its new multisensory environment has been a subject of debate. Although an early theory proposed that the brain acquires multisensory features as a result of postnatal experience, recent studies have demonstrated that the neonatal brain is already capable of processing multisensory information. For multisensory processing to be functional, it is a prerequisite that multisensory convergence among neural connections occur. However, multisensory connectivity has not been examined in human neonates nor are its location(s) or afferent sources understood. We used resting state functional MRI (fMRI) in two independent cohorts of infants to examine the functional connectivity of two cortical areas known to be multisensory in adults: the intraparietal sulcus (IPS) and the superior temporal sulcus (STS). In the neonate, the IPS was found to demonstrate significant functional connectivity with visual association and somatosensory association areas, while the STS showed significant functional connectivity with the visual association areas, primary auditory cortex, and somatosensory association areas. Our findings establish that each of these areas displays functional communication with cortical regions representing various sensory modalities. This demonstrates the presence of cortical areas with converging sensory inputs, representing that the functional architecture needed for multisensory processing is already present within the first weeks of life.

The Neural Basis of Speech Perception through Lipreading and Manual Cues: Evidence from Deaf Native Users of Cued Speech

We present here the first neuroimaging data for perception of Cued Speech (CS) by deaf adults who are native users of CS. CS is a visual mode of communicating a spoken language through a set of manual cues which accompany lipreading and disambiguate it. With CS, sublexical units of the oral language are conveyed clearly and completely through the visual modality without requiring hearing. The comparison of neural processing of CS in deaf individuals with processing of audiovisual (AV) speech in normally hearing individuals represents a unique opportunity to explore the similarities and differences in neural processing of an oral language delivered in a visuo-manual vs. an AV modality. The study included deaf adult participants who were early CS users and native hearing users of French who process speech audiovisually. Words were presented in an event-related fMRI design. Three conditions were presented to each group of participants. The deaf participants saw CS words (manual + lipread), words presented as manual cues alone, and words presented to be lipread without manual cues. The hearing group saw AV spoken words, audio-alone and lipread-alone. Three findings are highlighted. First, the middle and superior temporal gyrus (excluding Heschl's gyrus) and left inferior frontal gyrus pars triangularis constituted a common, amodal neural basis for AV and CS perception. Second, integration was inferred in posterior parts of superior temporal sulcus for audio and lipread information in AV speech, but in the occipito-temporal junction, including MT/V5, for the manual cues and lipreading in CS. Third, the perception of manual cues showed a much greater overlap with the regions activated by CS (manual + lipreading) than lipreading alone did. This supports the notion that manual cues play a larger role than lipreading for CS processing. The present study contributes to a better understanding of the role of manual cues as support of visual speech perception in the framework of the multimodal nature of human communication.

Distributed Visual–Vestibular Processing in the Cerebral Cortex of Man and Macaque

Recent advances in understanding the neurobiological underpinnings of visual–vestibular interactions underlying self-motion perception are reviewed with an emphasis on comparisons between the macaque and human brains. In both species, several distinct cortical regions have been identified that are active during both visual and vestibular stimulation and in some of these there is clear evidence for sensory integration. Several possible cross-species homologies between cortical regions are identified. A key feature of cortical organization is that the same information is apparently represented in multiple, anatomically diverse cortical regions, suggesting that information about self-motion is used for different purposes in different brain regions.

Oscillatory activity in auditory cortex reflects the perceptual level of audio-tactile integration

Cross-modal interactions between sensory channels have been shown to depend on both the spatial disparity and the perceptual similarity between the presented stimuli. Here we investigate the behavioral and neural integration of auditory and tactile stimulus pairs at different levels of spatial disparity. Additionally, we modulated the amplitudes of both stimuli in either a coherent or non-coherent manner. We found that both auditory and tactile localization performance was biased towards the stimulus in the respective other modality. This bias linearly increases with stimulus disparity and is more pronounced for coherently modulated stimulus pairs. Analyses of electroencephalographic (EEG) activity at temporal–cortical sources revealed enhanced event-related potentials (ERPs) as well as decreased alpha and beta power during bimodal as compared to unimodal stimulation. However, while the observed ERP differences are similar for all stimulus combinations, the extent of oscillatory desynchronization varies with stimulus disparity. Moreover, when both stimuli were subjectively perceived as originating from the same direction, the reduction in alpha and beta power was significantly stronger. These observations suggest that in the EEG the level of perceptual integration is mainly reflected by changes in ongoing oscillatory activity.

Posterior Superior Temporal Sulcus Responses Predict Perceived Pleasantness of Skin Stroking

Love and affection is expressed through a range of physically intimate gestures, including caresses. Recent studies suggest that posterior temporal lobe areas typically associated with visual processing of social cues also respond to interpersonal touch. Here, we asked whether these areas are selective to caress-like skin stroking. We collected functional magnetic resonance imaging data from 23 healthy participants and compared brain responses to skin stroking and vibration. We did not find any significant differences between stroking and vibration in the posterior temporal lobe; however, right posterior superior temporal sulcus (pSTS) responses predicted healthy participant's perceived pleasantness of skin stroking, but not vibration. These findings link right pSTS responses to individual variability in perceived pleasantness of caress-like tactile stimuli. We speculate that the right pSTS may play a role in the translation of tactile stimuli into positively valenced, socially relevant interpersonal touch and that this system may be affected in disorders associated with impaired attachment.

Multisensory integration substantiates distributed and overlapping neural networks

The hypothesis that highly overlapping networks underlie brain functions (neural reuse) is decisively supported by three decades of multisensory research. Multisensory areas process information from more than one sensory modality and therefore represent the best examples of neural reuse. Recent evidence of multisensory processing in primary visual cortices further indicates that neural reuse is a basic feature of the brain.

ALE meta-analysis reveals dissociable networks for affective and discriminative aspects of touch

Emotionally-laden tactile stimulation-such as a caress on the skin or the feel of velvet-may represent a functionally distinct domain of touch, underpinned by specific cortical pathways. In order to determine whether, and to what extent, cortical functional neuroanatomy supports a distinction between affective and discriminative touch, an activation likelihood estimate (ALE) meta-analysis was performed. This meta-analysis statistically mapped reported functional magnetic resonance imaging (fMRI) activations from 17 published affective touch studies in which tactile stimulation was associated with positive subjective evaluation (n = 291, 34 experimental contrasts). A separate ALE meta-analysis mapped regions most likely to be activated by tactile stimulation during detection and discrimination tasks (n = 1,075, 91 experimental contrasts). These meta-analyses revealed dissociable regions for affective and discriminative touch, with posterior insula (PI) more likely to be activated for affective touch, and primary somatosensory cortices (SI) more likely to be activated for discriminative touch. Secondary somatosensory cortex had a high likelihood of engagement by both affective and discriminative touch. Further, meta-analytic connectivity (MCAM) analyses investigated network-level co-activation likelihoods independent of task or stimulus, across a range of domains and paradigms. Affective-related PI and discriminative-related SI regions co-activated with different networks, implicated in dissociable functions, but sharing somatosensory co-activations. Taken together, these meta-analytic findings suggest that affective and discriminative touch are dissociable both on the regional and network levels. However, their degree of shared activation likelihood in somatosensory cortices indicates that this dissociation reflects functional biases within tactile processing networks, rather than functionally and anatomically distinct pathways.

Behavioral, perceptual, and neural alterations in sensory and multisensory function in autism spectrum disorder

Although sensory processing challenges have been noted since the first clinical descriptions of autism, it has taken until the release of the fifth edition of the Diagnostic and Statistical Manual of Mental Disorders (DSM-5) in 2013 for sensory problems to be included as part of the core symptoms of autism spectrum disorder (ASD) in the diagnostic profile. Because sensory information forms the building blocks for higher-order social and cognitive functions, we argue that sensory processing is not only an additional piece of the puzzle, but rather a critical cornerstone for characterizing and understanding ASD. In this review we discuss what is currently known about sensory processing in ASD, how sensory function fits within contemporary models of ASD, and what is understood about the differences in the underlying neural processing of sensory and social communication observed between individuals with and without ASD. In addition to highlighting the sensory features associated with ASD, we also emphasize the importance of multisensory processing in building perceptual and cognitive representations, and how deficits in multisensory integration may also be a core characteristic of ASD.

Multisensory Stimulation to Improve Low- and Higher-Level Sensory Deficits after Stroke: A Systematic Review

The aim of this systematic review was to integrate and assess evidence for the effectiveness of multisensory stimulation (i.e., stimulating at least two of the following sensory systems: visual, auditory, and somatosensory) as a possible rehabilitation method after stroke. Evidence was considered with a focus on low-level, perceptual (visual, auditory and somatosensory deficits), as well as higher-level, cognitive, sensory deficits. We referred to the electronic databases Scopus and PubMed to search for articles that were published before May 2015. Studies were included which evaluated the effects of multisensory stimulation on patients with low- or higher-level sensory deficits caused by stroke. Twenty-one studies were included in this review and the quality of these studies was assessed (based on eight elements: randomization, inclusion of control patient group, blinding of participants, blinding of researchers, follow-up, group size, reporting effect sizes, and reporting time post-stroke). Twenty of the twenty-one included studies demonstrate beneficial effects on low- and/or higher-level sensory deficits after stroke. Notwithstanding these beneficial effects, the quality of the studies is insufficient for valid conclusion that multisensory stimulation can be successfully applied as an effective intervention. A valuable and necessary next step would be to set up well-designed randomized controlled trials to examine the effectiveness of multisensory stimulation as an intervention for low- and/or higher-level sensory deficits after stroke. Finally, we consider the potential mechanisms of multisensory stimulation for rehabilitation to guide this future research.

Tactile representation of the head and shoulders assessed by fMRI in the nonhuman primate

In nonhuman primates, tactile representation at the cortical level has mostly been studied using single-cell recordings targeted to specific cortical areas. In this study, we explored the representation of tactile information delivered to the face or the shoulders at the whole brain level, using functional magnetic resonance imaging (fMRI) in the nonhuman primate. We used air puffs delivered to the center of the face, the periphery of the face, or the shoulders. These stimulations elicited activations in numerous cortical areas, encompassing the primary and secondary somatosensory areas, prefrontal and premotor areas, and parietal, temporal, and cingulate areas as well as low-level visual cortex. Importantly, a specific parieto-temporo-prefrontal network responded to the three stimulations but presented a marked preference for air puffs directed to the center of the face. This network corresponds to areas that are also involved in near-space representation, as well as in the multisensory integration of information at the interface between this near space and the skin of the face, and is probably involved in the construction of a peripersonal space representation around the head.

Keeping time in the brain: Autism spectrum disorder and audiovisual temporal processing

A growing area of interest and relevance in the study of autism spectrum disorder (ASD) focuses on the relationship between multisensory temporal function and the behavioral, perceptual, and cognitive impairments observed in ASD. Atypical sensory processing is becoming increasingly recognized as a core component of autism, with evidence of atypical processing across a number of sensory modalities. These deviations from typical processing underscore the value of interpreting ASD within a multisensory framework. Furthermore, converging evidence illustrates that these differences in audiovisual processing may be specifically related to temporal processing. This review seeks to bridge the connection between temporal processing and audiovisual perception, and to elaborate on emerging data showing differences in audiovisual temporal function in autism. We also discuss the consequence of such changes, the specific impact on the processing of different classes of audiovisual stimuli (e.g. speech vs. nonspeech, etc.), and the presumptive brain processes and networks underlying audiovisual temporal integration. Finally, possible downstream behavioral implications, and possible remediation strategies are outlined. Autism Res 2016, 9: 720-738. © 2015 International Society for Autism Research, Wiley Periodicals, Inc.

The social mysteries of the superior temporal sulcus

The superior temporal sulcus (STS) is implicated in a variety of social processes, ranging from language perception to simulating the mental processes of others (theory of mind). In a new study, Deen and colleagues use functional magnetic resonance imaging (fMRI) to show a regular anterior-posterior organization in the STS for different social tasks.

Re-examining overlap between tactile and visual motion responses within hMT+ and STS

Here, we examine overlap between tactile and visual motion BOLD responses within the human MT+ complex. Although several studies have reported tactile responses overlapping with hMT+, many used group average analyses, leaving it unclear whether these responses were restricted to subregions of hMT+. Moreover, previous studies either employed a tactile task or passive stimulation, leaving it unclear whether or not tactile responses in hMT+ are simply the consequence of visual imagery. Here, we carried out a replication of one of the classic papers finding tactile responses in hMT+. We mapped MT and MST in individual subjects using visual field localizers. We then examined responses to tactile motion on the arm, either presented passively or in the presence of a visual task performed at fixation designed to minimize visualization of the concurrent tactile stimulation. To our surprise, without a visual task, we found only weak tactile motion responses in MT (6% of voxels showing tactile responses) and MST (2% of voxels). With an unrelated visual task designed to withdraw attention from the tactile modality, responses in MST were reduced to almost nothing (<1% regions). Consistent with previous results, we did observe tactile responses in STS regions superior and anterior to hMT+. Despite the lack of individual overlap, group-averaged responses produced strong spurious overlap between tactile and visual motion responses within hMT+ that resembled those observed in previous studies. The weak nature of tactile responses in hMT+ (and their abolition by withdrawal of attention) suggests that hMT+ may not serve as a supramodal motion processing module.

Prestimulus oscillatory alpha power and connectivity patterns predispose perceptual integration of an audio and a tactile stimulus

To efficiently perceive and respond to the external environment, our brain has to perceptually integrate or segregate stimuli of different modalities. The temporal relationship between the different sensory modalities is therefore essential for the formation of different multisensory percepts. In this magnetoencephalography study, we created a paradigm where an audio and a tactile stimulus were presented by an ambiguous temporal relationship so that perception of physically identical audiotactile stimuli could vary between integrated (emanating from the same source) and segregated. This bistable paradigm allowed us to compare identical bimodal stimuli that elicited different percepts, providing a possibility to directly infer multisensory interaction effects. Local differences in alpha power over bilateral inferior parietal lobules (IPLs) and superior parietal lobules (SPLs) preceded integrated versus segregated percepts of the two stimuli (audio and tactile). Furthermore, differences in long-range cortical functional connectivity seeded in rIPL (region of maximum difference) revealed differential patterns that predisposed integrated or segregated percepts encompassing secondary areas of all different modalities and prefrontal cortex. We showed that the prestimulus brain states predispose the perception of the audiotactile stimulus both in a global and a local manner. Our findings are in line with a recent consistent body of findings on the importance of prestimulus brain states for perception of an upcoming stimulus. This new perspective on how stimuli originating from different modalities are integrated suggests a non-modality specific network predisposing multisensory perception.

Stereological study of pyramidal neurons in the human superior temporal gyrus from childhood to adulthood

The association cortex of the superior temporal gyrus (STG) is implicated in complex social and linguistic functions. Thus, reliable methods for quantifying cellular variation in this region could greatly benefit researchers interested in addressing the cellular correlates of typical and atypical function associated with these critical cognitive abilities. To facilitate this task, we first present a general set of cytoarchitectonic criteria targeted specifically toward stereological analyses of thick, Nissl-stained sections for the homotypical cortex of the STG, referred to here as BA22/TA. Second, we use the optical fractionator to estimate pyramidal neuron number and the nucleator for pyramidal somal and nuclear volume. We also investigated the influence of age and sex on these parameters, as well as set a typically developing baseline for future comparisons. In 11 typically developing cases aged 4-48 years, the most distinguishing features of BA22/TA were the presence of distinct granular layers, a prominent, jagged layer IIIc, and a distinctly staining VIa. The average number of neurons was 91 ± 15 million, the volume of pyramidal soma 1,512 µm(3) , and the nuclear volume 348 µm(3) . We found no correlation with age and neuron number. In contrast, pyramidal somal and nuclear volume were both negatively correlated and linearly associated with age in regression analyses. We found no significant sex differences. Overall, the data support the idea that postnatal neuron numbers are relatively stable through development but also suggest that neuronal volume may be subject to important developmental variation. Both measures are critical variables in the study of developmental neuropathology.

An integrative neural model of social perception, action observation, and theory of mind

In the field of social neuroscience, major branches of research have been instrumental in describing independent components of typical and aberrant social information processing, but the field as a whole lacks a comprehensive model that integrates different branches. We review existing research related to the neural basis of three key neural systems underlying social information processing: social perception, action observation, and theory of mind. We propose an integrative model that unites these three processes and highlights the posterior superior temporal sulcus (pSTS), which plays a central role in all three systems. Furthermore, we integrate these neural systems with the dual system account of implicit and explicit social information processing. Large-scale meta-analyses based on Neurosynth confirmed that the pSTS is at the intersection of the three neural systems. Resting-state functional connectivity analysis with 1000 subjects confirmed that the pSTS is connected to all other regions in these systems. The findings presented in this review are specifically relevant for psychiatric research especially disorders characterized by social deficits such as autism spectrum disorder.

Spatial imagery relies on a sensory independent, though sensory sensitive, functional organization within the parietal cortex: a fMRI study of angle discrimination in sighted and congenitally blind individuals

Although vision offers distinctive information to space representation, individuals who lack vision since birth often show perceptual and representational skills comparable to those found in sighted individuals. However, congenitally blind individuals may result in impaired spatial analysis, when engaging in 'visual' spatial features (e.g., perspective or angle representation) or complex spatial mental abilities. In the present study, we measured behavioral and brain responses using functional magnetic resonance imaging in sighted and congenitally blind individuals during spatial imagery based on a modified version of the mental clock task (e.g., angle discrimination) and a simple recognition control condition, as conveyed across distinct sensory modalities: visual (sighted individuals only), tactile and auditory. Blind individuals were significantly less accurate during the auditory task, but comparable-to-sighted during the tactile task. As expected, both groups showed common neural activations in intraparietal and superior parietal regions across visual and non-visual spatial perception and imagery conditions, indicating the more abstract, sensory independent functional organization of these cortical areas, a property that we named supramodality. At the same time, however, comparisons in brain responses and functional connectivity patterns across experimental conditions demonstrated also a functional lateralization, in a way that correlated with the distinct behavioral performance in blind and sighted individuals. Specifically, blind individuals relied more on right parietal regions, mainly in the tactile and less in the auditory spatial processing. In sighted, spatial representation across modalities relied more on left parietal regions. In conclusions, intraparietal and superior parietal regions subserve supramodal spatial representations in sighted and congenitally blind individuals. Differences in their recruitment across non-visual spatial processing in sighted and blind individuals may be related to distinctive behavioral performance and/or mental strategies adopted when they deal with the same spatial representation as conveyed through different sensory modalities.

Changes in the theta band coherence during motor task after hand immobilization

Many different factors can temporarily or permanently impair movement and impairs cortical organization, e.g. hand immobilization. Such changes have been widely studied using electroencephalography. Within this context, we have investigated the immobilization effects through the theta band coherence analysis, in order to find out whether the immobilization period causes any changes in the inter and intra-hemispheric coherence within the cerebral cortex, as well as to observe whether the theta band provides any information about the neural mechanisms involved during the motor act. We analyzed the cortical changes that occurred after 48 hours of hand immobilization. The theta band coherence was study through electroencephalography in 30 healthy subjects, divided into two groups (control and experimental). Within both groups, the subjects executed a task involving flexion and extension of the index finger, before and after 48 hours. The experimental group, however, was actually submitted to hand immobilization. We were able to observe an increase in the coupling within the experimental group in the frontal, parietal and temporal regions, and a decrease in the motor area. In order to execute manual tasks after some time of movement restriction, greater coherence is present in areas related to attention, movement preparation and sensorimotor integration processes. These results may contribute to a detailed assessment of involved neurophysiological mechanism in motor act execution.

Semantic retrieval during overt picture description: Left anterior temporal or the parietal lobe?

Retrieval of semantic representations is a central process during overt speech production. There is an increasing consensus that an amodal semantic 'hub' must exist that draws together modality-specific representations of concepts. Based on the distribution of atrophy and the behavioral deficit of patients with the semantic variant of fronto-temporal lobar degeneration, it has been proposed that this hub is localized within both anterior temporal lobes (ATL), and is functionally connected with verbal 'output' systems via the left ATL. An alternative view, dating from Geschwind's proposal in 1965, is that the angular gyrus (AG) is central to object-based semantic representations. In this fMRI study we examined the connectivity of the left ATL and parietal lobe (PL) with whole brain networks known to be activated during overt picture description. We decomposed each of these two brain volumes into 15 regions of interest (ROIs), using independent component analysis. A dual regression analysis was used to establish the connectivity of each ROI with whole brain-networks. An ROI within the left anterior superior temporal sulcus (antSTS) was functionally connected to other parts of the left ATL, including anterior ventromedial left temporal cortex (partially attenuated by signal loss due to susceptibility artifact), a large left dorsolateral prefrontal region (including 'classic' Broca's area), extensive bilateral sensory-motor cortices, and the length of both superior temporal gyri. The time-course of this functionally connected network was associated with picture description but not with non-semantic baseline tasks. This system has the distribution expected for the production of overt speech with appropriate semantic content, and the auditory monitoring of the overt speech output. In contrast, the only left PL ROI that showed connectivity with brain systems most strongly activated by the picture-description task, was in the superior parietal lobe (supPL). This region showed connectivity with predominantly posterior cortical regions required for the visual processing of the pictorial stimuli, with additional connectivity to the dorsal left AG and a small component of the left inferior frontal gyrus. None of the other PL ROIs that included part of the left AG were activated by Speech alone. The best interpretation of these results is that the left antSTS connects the proposed semantic hub (specifically localized to ventral anterior temporal cortex based on clinical neuropsychological studies) to posterior frontal regions and sensory-motor cortices responsible for the overt production of speech.

Functional Heterogeneity and Convergence in the Right Temporoparietal Junction

The right temporoparietal junction (rTPJ) is engaged by tasks that manipulate biological motion processing, Theory of Mind attributions, and attention reorienting. The proximity of activations elicited by these tasks raises the question of whether these tasks share common cognitive component processes that are subserved by common neural substrates. Here, we used high-resolution whole-brain functional magnetic resonance imaging in a within-subjects design to determine whether these tasks activate common regions of the rTPJ. Each participant was presented with the 3 tasks in the same imaging session. In a whole-brain analysis, we found that only the right and left TPJs were activated by all 3 tasks. Multivoxel pattern analysis revealed that the regions of overlap could still discriminate the 3 tasks. Notably, we found significant cross-task classification in the right TPJ, which suggests a shared neural process between the 3 tasks. Taken together, these results support prior studies that have indicated functional heterogeneity within the rTPJ but also suggest a convergence of function within a region of overlap. These results also call for further investigation into the nature of the function subserved in this overlap region.

Neural Mechanisms of Body Awareness in Infants

The ability to differentiate one's body from others is a fundamental aspect of social perception and has been shown to involve the integration of sense modalities attributable to the self. Though behavioral studies in infancy have investigated infants' discrimination of body-related multisensory stimuli, whether they attribute this information as belonging to the self is still unknown. In human adults, neuroimaging studies have demonstrated the recruitment of a specific set of brain regions in response to body-related multisensory integration. To test whether the infant brain integrates this information similarly to adults, in a first functional near-infrared spectroscopy study we investigated the role of visual–proprioceptive feedback when temporal cues are manipulated by showing 5-month-old infants an online video of their own face while the infant was performing movements. To explore the role of body-related contingency further, in a second study we investigated whether cortical activation in response to self-initiated movements and external tactile stimulation was similar to that found in the first study. Our results indicate that infants' specialized cortical activation in response to body-related contingencies is similar to brain activation seen in response to body awareness in adults.

Temporal factors affecting somatosensory-auditory interactions in speech processing

Speech perception is known to rely on both auditory and visual information. However, sound-specific somatosensory input has been shown also to influence speech perceptual processing (Ito et al., 2009). In the present study, we addressed further the relationship between somatosensory information and speech perceptual processing by addressing the hypothesis that the temporal relationship between orofacial movement and sound processing contributes to somatosensory–auditory interaction in speech perception. We examined the changes in event-related potentials (ERPs) in response to multisensory synchronous (simultaneous) and asynchronous (90 ms lag and lead) somatosensory and auditory stimulation compared to individual unisensory auditory and somatosensory stimulation alone. We used a robotic device to apply facial skin somatosensory deformations that were similar in timing and duration to those experienced in speech production. Following synchronous multisensory stimulation the amplitude of the ERP was reliably different from the two unisensory potentials. More importantly, the magnitude of the ERP difference varied as a function of the relative timing of the somatosensory–auditory stimulation. Event-related activity change due to stimulus timing was seen between 160 and 220 ms following somatosensory onset, mostly around the parietal area. The results demonstrate a dynamic modulation of somatosensory–auditory convergence and suggest the contribution of somatosensory information for speech processing process is dependent on the specific temporal order of sensory inputs in speech production.

A Predominantly Visual Subdivision of The Right Temporo-Parietal Junction (vTPJ)

A multiplicity of sensory and cognitive functions has been attributed to the large cortical region at the temporo-parietal junction (TPJ). Using functional MRI, we report that a small region lateralized within the right TPJ responds robustly to certain simple visual stimuli ("vTPJ"). The vTPJ was found in all right hemispheres (n = 7), posterior to the auditory cortex. To manipulate stimuli and attention, subjects were presented with a mixture of visual and auditory stimuli in a concurrent block design in 2 experiments: (1) A simple visual stimulus (a grating pattern modulating in mean luminance) elicited robust responses in the vTPJ, whether or not the subject attended to vision and(2) a drifting low-contrast dartboard pattern of constant mean luminance evoked robust responses in the vTPJ when it was task-relevant (visual task), and smaller responses when it was not (auditory task). The results suggest a focal, visually responsive region within the right TPJ that is powerfully driven by certain visual stimuli (luminance fluctuations), and that can be driven by other visual stimuli when the subject is attending. The precise localization of this visually responsive region is helpful in segmenting the TPJ and to better understand its role in visual awareness and related disorders such as extinction and neglect.

The functional organization of the left STS: a large scale meta-analysis of PET and fMRI studies of healthy adults

The superior temporal sulcus (STS) in the left hemisphere is functionally diverse, with sub-areas implicated in both linguistic and non-linguistic functions. However, the number and boundaries of distinct functional regions remain to be determined. Here, we present new evidence, from meta-analysis of a large number of positron emission tomography (PET) and functional magnetic resonance imaging (fMRI) studies, of different functional specificity in the left STS supporting a division of its middle to terminal extent into at least three functional areas. The middle portion of the left STS stem (fmSTS) is highly specialized for speech perception and the processing of language material. The posterior portion of the left STS stem (fpSTS) is highly versatile and involved in multiple functions supporting semantic memory and associative thinking. The fpSTS responds to both language and non-language stimuli but the sensitivity to non-language material is greater. The horizontal portion of the left STS stem and terminal ascending branches (ftSTS) display intermediate functional specificity, with the anterior-dorsal ascending branch (fatSTS) supporting executive functions and motor planning and showing greater sensitivity to language material, and the horizontal stem and posterior-ventral ascending branch (fptSTS) supporting primarily semantic processing and displaying greater sensitivity to non-language material. We suggest that the high functional specificity of the left fmSTS for speech is an important means by which the human brain achieves exquisite affinity and efficiency for native speech perception. In contrast, the extreme multi-functionality of the left fpSTS reflects the role of this area as a cortical hub for semantic processing and the extraction of meaning from multiple sources of information. Finally, in the left ftSTS, further functional differentiation between the dorsal and ventral aspect is warranted.

Neural correlates of finger gnosis

Neuropsychological studies have described patients with a selective impairment of finger identification in association with posterior parietal lesions. However, evidence of the role of these areas in finger gnosis from studies of the healthy human brain is still scarce. Here we used functional magnetic resonance imaging to identify the brain network engaged in a novel finger gnosis task, the intermanual in-between task (IIBT), in healthy participants. Several brain regions exhibited a stronger blood oxygenation level-dependent (BOLD) response in IIBT than in a control task that did not explicitly rely on finger gnosis but used identical stimuli and motor responses as the IIBT. The IIBT involved stronger signal in the left inferior parietal lobule (IPL), bilateral precuneus (PCN), bilateral premotor cortex, and left inferior frontal gyrus. In all regions, stimulation of nonhomologous fingers of the two hands elicited higher BOLD signal than stimulation of homologous fingers. Only in the left anteromedial IPL (a-mIPL) and left PCN did signal strength decrease parametrically from nonhomology, through partial homology, to total homology with stimulation delivered synchronously to the two hands. With asynchronous stimulation, the signal was stronger in the left a-mIPL than in any other region, possibly indicating retention of task-relevant information. We suggest that the left PCN may contribute a supporting visuospatial representation via its functional connection to the right PCN. The a-mIPL may instead provide the core substrate of an explicit bilateral body structure representation for the fingers that when disrupted can produce the typical symptoms of finger agnosia.

Shared mushroom body circuits underlie visual and olfactory memories in Drosophila

In nature, animals form memories associating reward or punishment with stimuli from different sensory modalities, such as smells and colors. It is unclear, however, how distinct sensory memories are processed in the brain. We established appetitive and aversive visual learning assays for Drosophila that are comparable to the widely used olfactory learning assays. These assays share critical features, such as reinforcing stimuli (sugar reward and electric shock punishment), and allow direct comparison of the cellular requirements for visual and olfactory memories. We found that the same subsets of dopamine neurons drive formation of both sensory memories. Furthermore, distinct yet partially overlapping subsets of mushroom body intrinsic neurons are required for visual and olfactory memories. Thus, our results suggest that distinct sensory memories are processed in a common brain center. Such centralization of related brain functions is an economical design that avoids the repetition of similar circuit motifs.

DOI:http://dx.doi.org/10.7554/eLife.02395.001

Animals tend to associate good and bad things with certain visual scenes, smells and other kinds of sensory information. If we get food poisoning after eating a new food, for example, we tend to associate the taste and smell of the new food with feelings of illness. This is an example of a negative ‘associative memory’, and it can persist for months, even when we know that our sickness was not caused by the new food itself but by some foreign body that should not have been in the food. The same is true for positive associative memories.

It is known that many associative memories contain information from more than one of the senses. Our memory of a favorite food, for instance, includes its scent, color and texture, as well as its taste. However, little is known about the ways in which information from the different senses is processed in the brain. Does each sense have its own dedicated memory circuit, or do multiple senses converge to the same memory circuit?

A number of studies have used olfactory (smell) and visual stimuli to study the basic neuroscience that underpins associative memories in fruit flies. The olfactory experiments traditionally use sugar and electric shocks to induce positive and negative associations with various scents. However, the visual experiments use other methods to induce associations with colors. This means that it is difficult to combine and compare the results of olfactory and visual experiments.

Now, Vogt, Schnaitmann et al. have developed a transparent grid that can be used to administer electric shocks in visual experiments. This allows direct comparisons to be made between the neuronal processing of visual associative memories and the neural processing of olfactory associative memories.

Vogt, Schnaitmann et al. showed that both visual and olfactory stimuli are modulated in the same subset of dopamine neurons for positive associative memories. Similarly, another subset of dopamine neurons was found to drive negative memories of both the visual and olfactory stimuli. The work of Vogt, Schnaitmann et al. shows that associative memories are processed by a centralized circuit that receives both visual and olfactory inputs, thus reducing the number of memory circuits needed for such memories.

DOI:http://dx.doi.org/10.7554/eLife.02395.002

An ALE meta-analysis on the audiovisual integration of speech signals

The brain improves speech processing through the integration of audiovisual (AV) signals. Situations involving AV speech integration may be crudely dichotomized into those where auditory and visual inputs contain (1) equivalent, complementary signals (validating AV speech) or (2) inconsistent, different signals (conflicting AV speech). This simple framework may allow the systematic examination of broad commonalities and differences between AV neural processes engaged by various experimental paradigms frequently used to study AV speech integration. We conducted an activation likelihood estimation metaanalysis of 22 functional imaging studies comprising 33 experiments, 311 subjects, and 347 foci examining "conflicting" versus "validating" AV speech. Experimental paradigms included content congruency, timing synchrony, and perceptual measures, such as the McGurk effect or synchrony judgments, across AV speech stimulus types (sublexical to sentence). Colocalization of conflicting AV speech experiments revealed consistency across at least two contrast types (e.g., synchrony and congruency) in a network of dorsal stream regions in the frontal, parietal, and temporal lobes. There was consistency across all contrast types (synchrony, congruency, and percept) in the bilateral posterior superior/middle temporal cortex. Although fewer studies were available, validating AV speech experiments were localized to other regions, such as ventral stream visual areas in the occipital and inferior temporal cortex. These results suggest that while equivalent, complementary AV speech signals may evoke activity in regions related to the corroboration of sensory input, conflicting AV speech signals recruit widespread dorsal stream areas likely involved in the resolution of conflicting sensory signals.

Distinct cortical locations for integration of audiovisual speech and the McGurk effect

Audiovisual (AV) speech integration is often studied using the McGurk effect, where the combination of specific incongruent auditory and visual speech cues produces the perception of a third illusory speech percept. Recently, several studies have implicated the posterior superior temporal sulcus (pSTS) in the McGurk effect; however, the exact roles of the pSTS and other brain areas in "correcting" differing AV sensory inputs remain unclear. Using functional magnetic resonance imaging (fMRI) in ten participants, we aimed to isolate brain areas specifically involved in processing congruent AV speech and the McGurk effect. Speech stimuli were composed of sounds and/or videos of consonant-vowel tokens resulting in four stimulus classes: congruent AV speech (AVCong), incongruent AV speech resulting in the McGurk effect (AVMcGurk), acoustic-only speech (AO), and visual-only speech (VO). In group- and single-subject analyses, left pSTS exhibited significantly greater fMRI signal for congruent AV speech (i.e., AVCong trials) than for both AO and VO trials. Right superior temporal gyrus, medial prefrontal cortex, and cerebellum were also identified. For McGurk speech (i.e., AVMcGurk trials), two clusters in the left posterior superior temporal gyrus (pSTG), just posterior to Heschl's gyrus or on its border, exhibited greater fMRI signal than both AO and VO trials. We propose that while some brain areas, such as left pSTS, may be more critical for the integration of AV speech, other areas, such as left pSTG, may generate the "corrected" or merged percept arising from conflicting auditory and visual cues (i.e., as in the McGurk effect). These findings are consistent with the concept that posterior superior temporal areas represent part of a "dorsal auditory stream," which is involved in multisensory integration, sensorimotor control, and optimal state estimation (Rauschecker and Scott, 2009).

Short-term plasticity of visuo-haptic object recognition

Functional magnetic resonance imaging (fMRI) studies have provided ample evidence for the involvement of the lateral occipital cortex (LO), fusiform gyrus (FG), and intraparietal sulcus (IPS) in visuo-haptic object integration. Here we applied 30 min of sham (non-effective) or real offline 1 Hz repetitive transcranial magnetic stimulation (rTMS) to perturb neural processing in left LO immediately before subjects performed a visuo-haptic delayed-match-to-sample task during fMRI. In this task, subjects had to match sample (S1) and target (S2) objects presented sequentially within or across vision and/or haptics in both directions (visual-haptic or haptic-visual) and decide whether or not S1 and S2 were the same objects. Real rTMS transiently decreased activity at the site of stimulation and remote regions such as the right LO and bilateral FG during haptic S1 processing. Without affecting behavior, the same stimulation gave rise to relative increases in activation during S2 processing in the right LO, left FG, bilateral IPS, and other regions previously associated with object recognition. Critically, the modality of S2 determined which regions were recruited after rTMS. Relative to sham rTMS, real rTMS induced increased activations during crossmodal congruent matching in the left FG for haptic S2 and the temporal pole for visual S2. In addition, we found stronger activations for incongruent than congruent matching in the right anterior parahippocampus and middle frontal gyrus for crossmodal matching of haptic S2 and in the left FG and bilateral IPS for unimodal matching of visual S2, only after real but not sham rTMS. The results imply that a focal perturbation of the left LO triggers modality-specific interactions between the stimulated left LO and other key regions of object processing possibly to maintain unimpaired object recognition. This suggests that visual and haptic processing engage partially distinct brain networks during visuo-haptic object matching.

When vision is not an option: children's integration of auditory and haptic information is suboptimal

When visual information is available, human adults, but not children, have been shown to reduce sensory uncertainty by taking a weighted average of sensory cues. In the absence of reliable visual information (e.g. extremely dark environment, visual disorders), the use of other information is vital. Here we ask how humans combine haptic and auditory information from childhood. In the first experiment, adults and children aged 5 to 11 years judged the relative sizes of two objects in auditory, haptic, and non-conflicting bimodal conditions. In Experiment 2, different groups of adults and children were tested in non-conflicting and conflicting bimodal conditions. In Experiment 1, adults reduced sensory uncertainty by integrating the cues optimally, while children did not. In Experiment 2, adults and children used similar weighting strategies to solve audio–haptic conflict. These results suggest that, in the absence of visual information, optimal integration of cues for discrimination of object size develops late in childhood.

Flexibility and Stability in Sensory Processing Revealed Using Visual-to-Auditory Sensory Substitution

The classical view of sensory processing involves independent processing in sensory cortices and multisensory integration in associative areas. This hierarchical structure has been challenged by evidence of multisensory responses in sensory areas, and dynamic weighting of sensory inputs in associative areas, thus far reported independently. Here, we used a visual-to-auditory sensory substitution algorithm (SSA) to manipulate the information conveyed by sensory inputs while keeping the stimuli intact. During scan sessions before and after SSA learning, subjects were presented with visual images and auditory soundscapes. The findings reveal 2 dynamic processes. First, crossmodal attenuation of sensory cortices changed direction after SSA learning from visual attenuations of the auditory cortex to auditory attenuations of the visual cortex. Secondly, associative areas changed their sensory response profile from strongest response for visual to that for auditory. The interaction between these phenomena may play an important role in multisensory processing. Consistent features were also found in the sensory dominance in sensory areas and audiovisual convergence in associative area Middle Temporal Gyrus. These 2 factors allow for both stability and a fast, dynamic tuning of the system when required.

Auditory cortex activation to natural speech and simulated cochlear implant speech measured with functional near-infrared spectroscopy

The primary goal of most cochlear implant procedures is to improve a patient's ability to discriminate speech. To accomplish this, cochlear implants are programmed so as to maximize speech understanding. However, programming a cochlear implant can be an iterative, labor-intensive process that takes place over months. In this study, we sought to determine whether functional near-infrared spectroscopy (fNIRS), a non-invasive neuroimaging method which is safe to use repeatedly and for extended periods of time, can provide an objective measure of whether a subject is hearing normal speech or distorted speech. We used a 140 channel fNIRS system to measure activation within the auditory cortex in 19 normal hearing subjects while they listed to speech with different levels of intelligibility. Custom software was developed to analyze the data and compute topographic maps from the measured changes in oxyhemoglobin and deoxyhemoglobin concentration. Normal speech reliably evoked the strongest responses within the auditory cortex. Distorted speech produced less region-specific cortical activation. Environmental sounds were used as a control, and they produced the least cortical activation. These data collected using fNIRS are consistent with the fMRI literature and thus demonstrate the feasibility of using this technique to objectively detect differences in cortical responses to speech of different intelligibility.

Multisensory integration mechanisms during aging

The rapid demographical shift occurring in our society implies that understanding of healthy aging and age-related diseases is one of our major future challenges. Sensory impairments have an enormous impact on our lives and are closely linked to cognitive functioning. Due to the inherent complexity of sensory perceptions, we are commonly presented with a complex multisensory stimulation and the brain integrates the information from the individual sensory channels into a unique and holistic percept. The cerebral processes involved are essential for our perception of sensory stimuli and becomes especially important during the perception of emotional content. Despite ongoing deterioration of the individual sensory systems during aging, there is evidence for an increase in, or maintenance of, multisensory integration processing in aging individuals. Within this comprehensive literature review on multisensory integration we aim to highlight basic mechanisms and potential compensatory strategies the human brain utilizes to help maintain multisensory integration capabilities during healthy aging to facilitate a broader understanding of age-related pathological conditions. Further our goal was to identify where further research is needed.

Natural stimuli from three coherent modalities enhance behavioral responses and electrophysiological cortical activity in humans

Cues that involve a number of sensory modalities are processed in the brain in an interactive multimodal manner rather than independently for each modality. We studied multimodal integration in a natural, yet fully controlled scene, implemented as an interactive game in an auditory-haptic-visual virtual environment. In this imitation of a natural scene, the targets of perception were ecologically valid uni-, bi- and tri-modal manifestations of a simple event-a ball hitting a wall. Subjects were engaged in the game while their behavioral and early cortical electrophysiological responses were measured. Behavioral results confirmed that tri-modal cues were detected faster and more accurately than bi-modal cues, which, likewise, showed advantages over unimodal responses. Event-Related Potentials (ERPs) were recorded, and the first 200 ms following stimulus onset was analyzed to reveal the latencies of cortical multimodal interactions as estimated by sLORETA. These electrophysiological findings indicated bi-modal as well as tri-modal interactions beginning very early (~30 ms), uniquely for each multimodal combination. The results suggest that early cortical multimodal integration accelerates cortical activity and, in turn, enhances performance measures. This acceleration registers on the scalp as sub-additive cortical activation.

Left lateralized enhancement of orofacial somatosensory processing due to speech sounds

PURPOSE

Somatosensory information associated with speech articulatory movements affects the perception of speech sounds and vice versa, suggesting an intimate linkage between speech production and perception systems. However, it is unclear which cortical processes are involved in the interaction between speech sounds and orofacial somatosensory inputs. The authors examined whether speech sounds modify orofacial somatosensory cortical potentials that were elicited using facial skin perturbations.

METHOD

Somatosensory event-related potentials in EEG were recorded in 3 background sound conditions (pink noise, speech sounds, and nonspeech sounds) and also in a silent condition. Facial skin deformations that are similar in timing and duration to those experienced in speech production were used for somatosensory stimulation.

RESULTS

The authors found that speech sounds reliably enhanced the first negative peak of the somatosensory event-related potential when compared with the other 3 sound conditions. The enhancement was evident at electrode locations above the left motor and premotor area of the orofacial system. The result indicates that speech sounds interact with somatosensory cortical processes that are produced by speech-production-like patterns of facial skin stretch.

CONCLUSION

Neural circuits in the left hemisphere, presumably in left motor and premotor cortex, may play a prominent role in the interaction between auditory inputs and speech-relevant somatosensory processing.

Neural integration of speech and gesture in schizophrenia: evidence for differential processing of metaphoric gestures

Gestures are an important component of interpersonal communication. Especially, complex multimodal communication is assumed to be disrupted in patients with schizophrenia. In healthy subjects, differential neural integration processes for gestures in the context of concrete [iconic (IC) gestures] and abstract sentence contents [metaphoric (MP) gestures] had been demonstrated. With this study we wanted to investigate neural integration processes for both gesture types in patients with schizophrenia. During functional magnetic resonance imaging-data acquisition, 16 patients with schizophrenia (P) and a healthy control group (C) were shown videos of an actor performing IC and MP gestures and associated sentences. An isolated gesture (G) and isolated sentence condition (S) were included to separate unimodal from bimodal effects at the neural level. During IC conditions (IC > G ∩ IC > S) we found increased activity in the left posterior middle temporal gyrus (pMTG) in both groups. Whereas in the control group the left pMTG and the inferior frontal gyrus (IFG) were activated for the MP conditions (MP > G ∩ MP > S), no significant activation was found for the identical contrast in patients. The interaction of group (P/C) and gesture condition (MP/IC) revealed activation in the bilateral hippocampus, the left middle/superior temporal and IFG. Activation of the pMTG for the IC condition in both groups indicates intact neural integration of IC gestures in schizophrenia. However, failure to activate the left pMTG and IFG for MP co-verbal gestures suggests a disturbed integration of gestures embedded in an abstract sentence context. This study provides new insight into the neural integration of co-verbal gestures in patients with schizophrenia.

Tactile stimulation and hemispheric asymmetries modulate auditory perception and neural responses in primary auditory cortex

Although multisensory integration has been an important area of recent research, most studies focused on audiovisual integration. Importantly, however, the combination of audition and touch can guide our behavior as effectively which we studied here using psychophysics and functional magnetic resonance imaging (fMRI). We tested whether task-irrelevant tactile stimuli would enhance auditory detection, and whether hemispheric asymmetries would modulate these audiotactile benefits using lateralized sounds. Spatially aligned task-irrelevant tactile stimuli could occur either synchronously or asynchronously with the sounds. Auditory detection was enhanced by non-informative synchronous and asynchronous tactile stimuli, if presented on the left side. Elevated fMRI-signals to left-sided synchronous bimodal stimulation were found in primary auditory cortex (A1). Adjacent regions (planum temporale, PT) expressed enhanced BOLD-responses for synchronous and asynchronous left-sided bimodal conditions. Additional connectivity analyses seeded in right-hemispheric A1 and PT for both bimodal conditions showed enhanced connectivity with right-hemispheric thalamic, somatosensory and multisensory areas that scaled with subjects' performance. Our results indicate that functional asymmetries interact with audiotactile interplay which can be observed for left-lateralized stimulation in the right hemisphere. There, audiotactile interplay recruits a functional network of unisensory cortices, and the strength of these functional network connections is directly related to subjects' perceptual sensitivity.

Parietal connectivity mediates multisensory facilitation

Our senses interact in daily life through multisensory integration, facilitating perceptual processes and behavioral responses. The neural mechanisms proposed to underlie this multisensory facilitation include anatomical connections directly linking early sensory areas, indirect connections to higher-order multisensory regions, as well as thalamic connections. Here we examine the relationship between white matter connectivity, as assessed with diffusion tensor imaging, and individual differences in multisensory facilitation and provide the first demonstration of a relationship between anatomical connectivity and multisensory processing in typically developed individuals. Using a whole-brain analysis and contrasting anatomical models of multisensory processing we found that increased connectivity between parietal regions and early sensory areas was associated with the facilitation of reaction times to multisensory (auditory-visual) stimuli. Furthermore, building on prior animal work suggesting the involvement of the superior colliculus in this process, using probabilistic tractography we determined that the strongest cortical projection area connected with the superior colliculus includes the region of connectivity implicated in our independent whole-brain analysis.

Combined diffusion-weighted and functional magnetic resonance imaging reveals a temporal-occipital network involved in auditory-visual object processing

Functional magnetic resonance imaging (MRI) showed that the superior temporal and occipital cortex are involved in multisensory integration. Probabilistic fiber tracking based on diffusion-weighted MRI suggests that multisensory processing is supported by white matter connections between auditory cortex and the temporal and occipital lobe. Here, we present a combined functional MRI and probabilistic fiber tracking study that reveals multisensory processing mechanisms that remained undetected by either technique alone. Ten healthy participants passively observed visually presented lip or body movements, heard speech or body action sounds, or were exposed to a combination of both. Bimodal stimulation engaged a temporal-occipital brain network including the multisensory superior temporal sulcus (msSTS), the lateral superior temporal gyrus (lSTG), and the extrastriate body area (EBA). A region-of-interest (ROI) analysis showed multisensory interactions (e.g., subadditive responses to bimodal compared to unimodal stimuli) in the msSTS, the lSTG, and the EBA region. Moreover, sounds elicited responses in the medial occipital cortex. Probabilistic tracking revealed white matter tracts between the auditory cortex and the medial occipital cortex, the inferior occipital cortex (IOC), and the superior temporal sulcus (STS). However, STS terminations of auditory cortex tracts showed limited overlap with the msSTS region. Instead, msSTS was connected to primary sensory regions via intermediate nodes in the temporal and occipital cortex. Similarly, the lSTG and EBA regions showed limited direct white matter connections but instead were connected via intermediate nodes. Our results suggest that multisensory processing in the STS is mediated by separate brain areas that form a distinct network in the lateral temporal and inferior occipital cortex.