Transcranial Magnetic Stimulation over Left Inferior Frontal and Posterior Temporal Cortex Disrupts Gesture-Speech Integration

Social perception in deaf individuals: A meta-analysis of neuroimaging studies

Deaf individuals may report difficulties in social interactions. However, whether these difficulties depend on deafness affecting social brain circuits is controversial. Here, we report the first meta-analysis comparing brain activations of hearing and (prelingually) deaf individuals during social perception. Our findings showed that deafness does not impact on the functional mechanisms supporting social perception. Indeed, both deaf and hearing control participants recruited regions of the action observation network during performance of different social tasks employing visual stimuli, and including biological motion perception, face identification, action observation, viewing, identification and memory for signs and lip reading. Moreover, we found increased recruitment of the superior-middle temporal cortex in deaf individuals compared with hearing participants, suggesting a preserved and augmented function during social communication based on signs and lip movements. Overall, our meta-analysis suggests that social difficulties experienced by deaf individuals are unlikely to be associated with brain alterations but may rather depend on non-supportive environments.

Neurochronometry of choice-induced preference changes: when do preferences actually change?

According to cognitive dissonance theory, a discrepancy between preferences and actions may lead to the revaluation of preferences, increasing preference for the chosen options and decreasing for the rejected options. This phenomenon is known as the spreading of alternatives (SoA), which results in a choice-induced preference change (CIPC). Previous neuroimaging studies have identified several brain regions that play a role in cognitive dissonance. However, the neurochronometry of the cognitive mechanisms underlying CIPC is a topic of debate. In other words, does it occur during the difficult choice, immediately after the choice, or when people encounter the options again? Furthermore, it remains unclear what is the exact time point, relative to the onset of facing options, either within the choice or after it, when the attitudes start to be revised. We argue that applying online protocols of transcranial magnetic stimulation (TMS), during or immediately after the choice process, could be the most efficient way to better understand the temporal dynamics of the SoA effect. TMS allows for achieving high temporal and spatial resolution, modulating the activity of areas of interest, and examining the causal relationships. Besides, unlike the offline TMS, the online instrument allows tracking of the neurochronometry of attitude change, by varying stimulation onsets and durations with respect to the option stimuli. Based on scrupulous analysis of previous findings, employing online TMS studies of conflict monitoring, cognitive control, and CIPC neuroimaging results, we conclude that the use of online TMS is critical to examine the neurochronometry of CIPC.

TMS reveals a two-stage priming circuit of gesture-speech integration

Introduction

Naturalistically, multisensory information of gesture and speech is intrinsically integrated to enable coherent comprehension. Such cross-modal semantic integration is temporally misaligned, with the onset of gesture preceding the relevant speech segment. It has been proposed that gestures prime subsequent speech. However, there are unresolved questions regarding the roles and time courses that the two sources of information play in integration.

Methods

In two between-subject experiments of healthy college students, we segmented the gesture-speech integration period into 40-ms time windows (TWs) based on two separately division criteria, while interrupting the activity of the integration node of the left posterior middle temporal gyrus (pMTG) and the left inferior frontal gyrus (IFG) with double-pulse transcranial magnetic stimulation (TMS). In Experiment 1, we created fixed time-advances of gesture over speech and divided the TWs from the onset of speech. In Experiment 2, we differentiated the processing stages of gesture and speech and segmented the TWs in reference to the speech lexical identification point (IP), while speech onset occurred at the gesture semantic discrimination point (DP).

Results

The results showed a TW-selective interruption of the pMTG and IFG only in Experiment 2, with the pMTG involved in TW1 (-120 ~ -80 ms of speech IP), TW2 (-80 ~ -40 ms), TW6 (80 ~ 120 ms) and TW7 (120 ~ 160 ms) and the IFG involved in TW3 (-40 ~ 0 ms) and TW6. Meanwhile no significant disruption of gesture-speech integration was reported in Experiment 1.

Discussion

We determined that after the representation of gesture has been established, gesture-speech integration occurs such that speech is first primed in a phonological processing stage before gestures are unified with speech to form a coherent meaning. Our findings provide new insights into multisensory speech and co-speech gesture integration by tracking the causal contributions of the two sources of information.

Shared and distinct routes in speech and gesture imitation: Evidence from stroke

Dual-route models of high-level (praxis) actions distinguish between an "indirect" semantic route mediating meaningful gesture imitation, and a "direct" sensory-motor route mediates meaningless gesture imitation. Similarly, dual-route language models distinguish between an indirect route mediating production and repetition of words, and a direct route mediating non-word repetition. Although aphasia and limb apraxia frequently co-occur following left-hemisphere cerebrovascular accident (LCVA), it is unclear which aspects of these functional-neuroanatomic dual-route architectures are shared across praxis and language domains. This study focused on gesture imitation to test the hypothesis that semantic information (and portions of the indirect route) are shared across domains, whereas two distinct dorsal routes mediate sensory-motor mapping. Forty chronic LCVA and 17 neurotypical controls completed semantic memory and language tasks and imitated 3 types of gesture stimuli: (1) labeled/"named" meaningful, (2) unnamed meaningful, and (3) meaningless gestures. The comparison of accuracy between meaningless versus unnamed meaningful gestures examined the benefits of semantic information, while the comparison of unnamed meaningful versus named meaningful imitation examined additional benefits of linguistic cueing. Mixed-effects models examined group by task interaction effects on gesture ability. We found that for patients with LCVA, unnamed meaningful gestures were imitated more accurately than meaningless gestures, suggesting that semantic information was beneficial, but there was no benefit of labeling. Reduced benefit of semantic information on gesture accuracy was associated with lesions to inferior frontal and posterior temporal regions as well as semantic memory performance on a pictorial (non-gesture) task. In contrast, there was no relationship between meaningless gesture imitation and nonword repetition, indicating that measures of direct route performance are not associated across language and action. These results provide preliminary evidence that portions of the indirect semantic route are shared across the language and action domains, while two direct sensory-motor mapping routes mediate word repetition and gesture imitation.

Gesture-Speech Integration in Typical and Atypical Adolescent Readers

This study investigated gesture-speech integration (GSI) among adolescents who are deaf or hard of hearing (DHH) and those with typical hearing. Thirty-eight adolescents (19 with hearing loss) performed a Stroop-like task in which they watched 120 short video clips of gestures and actions twice at random. Participants were asked to press one button if the visual content of the speaker's movements was related to a written word and to press another button if it was unrelated to a written word while accuracy rates and response times were recorded. We found stronger GSI effects among DHH participants than hearing participants. The semantic congruency effect was significantly larger in DHH participants than in hearing participants, and results of our experiments indicated a significantly larger gender congruency effect in DHH participants as compared to hearing participants. Results of this study shed light on GSI among DHH individuals and suggest future avenues for research examining the impact of gesture on language processing and communication in this population.

A meta-analysis on neural changes of cognitive training for mental disorders in executive function tasks: increase or decrease brain activation?

BACKGROUND

Cognitive impairment is often found in patients with psychiatric disorders, and cognitive training (CT) has been shown to help these patients. To better understand the mechanisms of CT, many neuroimaging studies have investigated the neural changes associated with it. However, the results of those studies have been inconsistent, making it difficult to draw conclusions from the literature. Therefore, the objective of this meta-analysis was to identify consistent patterns in the literature of neural changes associated with CT for psychiatric disorders.

METHODS

We searched for cognitive training imaging studies in PubMed, Cochrane library, Scopus, and ProQuest electronic databases. We conducted an activation likelihood estimation (ALE) for coordinate-based meta-analysis of neuroimaging studies, conduct behavioral analysis of brain regions identified by ALE analysis, conduct behavioral analysis of brain regions identified by ALE analysis, and then created a functional meta-analytic connectivity model (fMACM) of the resulting regions.

RESULTS

Results showed that CT studies consistently reported increased activation in the left inferior frontal gyrus (IFG) and decreased activation in the left precuneus and cuneus from pre- to post- CT.

CONCLUSION

CT improves cognitive function by supporting language and memory function, and reducing neuronal resources associated with basic visual processing.

Covert Attention to Gestures Is Sufficient for Information Uptake

Numerous studies have explored the benefit of iconic gestures in speech comprehension. However, only few studies have investigated how visual attention was allocated to these gestures in the context of clear versus degraded speech and the way information is extracted for enhancing comprehension. This study aimed to explore the effect of iconic gestures on comprehension and whether fixating the gesture is required for information extraction. Four types of gestures (i.e., semantically and syntactically incongruent iconic gestures, meaningless configurations, and congruent iconic gestures) were presented in a sentence context in three different listening conditions (i.e., clear, partly degraded or fully degraded speech). Using eye tracking technology, participants’ gaze was recorded, while they watched video clips after which they were invited to answer simple comprehension questions. Results first showed that different types of gestures differently attract attention and that the more speech was degraded, the less participants would pay attention to gestures. Furthermore, semantically incongruent gestures appeared to particularly impair comprehension although not being fixated while congruent gestures appeared to improve comprehension despite also not being fixated. These results suggest that covert attention is sufficient to convey information that will be processed by the listener.

George Frankl: an undervalued voice in the history of autism

This paper aims to propose that the psychiatrist George Frankl had more than a marginal role in the early history of autism. Frankl's conception of autism as characterized by a lack of affective language has influenced both Asperger and Kanner. First, this proposal is historically supported; second it is corroborated by Frankl's unpublished manuscript on Autism. We found that Frankl's perspective about autism was, and still can be, considered innovative for multiple reasons. Specifically, Frankl proposed that autism could cover a spectrum of conditions; that it is a state of mind that is not necessarily abnormal; and that it is a neurobiological condition, which primarily needs to be understood by others. Finally, Frankl's concepts of affective contact and affective language are reconsidered with reference to contemporary neuropsychology from which autism emerges not as a higher-order cognitive deficit, but as a result of an impairment of primordial ability to process low level sensory, motor and perceptual information gained through experiencing other persons.

The role of the left and right inferior frontal gyrus in processing metaphoric and unrelated co-speech gestures

Gestures are an integral part of in-person conversations and complement the meaning of the speech they accompany. The neural processing of co-speech gestures is supported by a mostly left-lateralized network of fronto-temporal regions. However, in contrast to iconic gestures, metaphoric as well as unrelated gestures have been found to more strongly engage the left and right inferior frontal gyrus (IFG), respectively. With this study, we conducted the first systematic comparison of all three types of gestures and resulting potential laterality effects. During collection of functional imaging data, 74 subjects were presented with 5 s videos of abstract speech with related metaphoric gestures, concrete speech with related iconic gestures and concrete speech with unrelated gestures. They were asked to judge whether the content of the speech and gesture matched or not. Differential contrasts revealed that both abstract related and concrete unrelated compared to concrete related stimuli elicited stronger activation of the bilateral IFG. Analyses of lateralization indices for IFG activation further showed a left hemispheric dominance for metaphoric gestures and a right hemispheric dominance for unrelated gestures. Our results give support to the hypothesis that the bilateral IFG is activated specifically when processing load for speech-gesture combinations is high. In addition, laterality effects indicate a stronger involvement of the right IFG in mismatch detection and conflict processing, whereas the left IFG performs the actual integration of information from speech and gesture.

Event related spectral perturbations of gesture congruity: Visuospatial resources are recruited for multimodal discourse comprehension

Here we examine the role of visuospatial working memory (WM) during the comprehension of multimodal discourse with co-speech iconic gestures. EEG was recorded as healthy adults encoded either a sequence of one (low load) or four (high load) dot locations on a grid and rehearsed them until a free recall response was collected later in the trial. During the rehearsal period of the WM task, participants observed videos of a speaker describing objects in which half of the trials included semantically related co-speech gestures (congruent), and the other half included semantically unrelated gestures (incongruent). Discourse processing was indexed by oscillatory EEG activity in the alpha and beta bands during the videos. Across all participants, effects of speech and gesture incongruity were more evident in low load trials than in high load trials. Effects were also modulated by individual differences in visuospatial WM capacity. These data suggest visuospatial WM resources are recruited in the comprehension of multimodal discourse.

The Role of Iconic Gestures in Speech Comprehension: An Overview of Various Methodologies

Iconic gesture-speech integration is a relatively recent field of investigation with numerous researchers studying its various aspects. The results obtained are just as diverse. The definition of iconic gestures is often overlooked in the interpretations of results. Furthermore, while most behavioral studies have demonstrated an advantage of bimodal presentation, brain activity studies show a diversity of results regarding the brain regions involved in the processing of this integration. Clinical studies also yield mixed results, some suggesting parallel processing channels, others a unique and integrated channel. This review aims to draw attention to the methodological variations in research on iconic gesture-speech integration and how they impact conclusions regarding the underlying phenomena. It will also attempt to draw together the findings from other relevant research and suggest potential areas for further investigation in order to better understand processes at play during speech integration process.

Rapid invisible frequency tagging reveals nonlinear integration of auditory and visual information

During communication in real-life settings, the brain integrates information from auditory and visual modalities to form a unified percept of our environment. In the current magnetoencephalography (MEG) study, we used rapid invisible frequency tagging (RIFT) to generate steady-state evoked fields and investigated the integration of audiovisual information in a semantic context. We presented participants with videos of an actress uttering action verbs (auditory; tagged at 61 Hz) accompanied by a gesture (visual; tagged at 68 Hz, using a projector with a 1,440 Hz refresh rate). Integration difficulty was manipulated by lower-order auditory factors (clear/degraded speech) and higher-order visual factors (congruent/incongruent gesture). We identified MEG spectral peaks at the individual (61/68 Hz) tagging frequencies. We furthermore observed a peak at the intermodulation frequency of the auditory and visually tagged signals (fvisual - fauditory  = 7 Hz), specifically when lower-order integration was easiest because signal quality was optimal. This intermodulation peak is a signature of nonlinear audiovisual integration, and was strongest in left inferior frontal gyrus and left temporal regions; areas known to be involved in speech-gesture integration. The enhanced power at the intermodulation frequency thus reflects the ease of lower-order audiovisual integration and demonstrates that speech-gesture information interacts in higher-order language areas. Furthermore, we provide a proof-of-principle of the use of RIFT to study the integration of audiovisual stimuli, in relation to, for instance, semantic context.

Non-invasive brain stimulation in limb praxis and apraxia: A scoping review in healthy subjects and patients with stroke

Non-invasive brain stimulation (NIBS) techniques are widely used in research settings to investigate brain mechanisms and increasingly being used for treatment purposes. The aim of this study was to systematically identify and review the current literature on NIBS studies of limb praxis and apraxia in healthy subjects and stroke patients with a scoping review using PRISMA-ScR guidelines. MEDLINE-PubMed, EMBASE and PsycINFO were searched. Inclusion criteria were English peer-reviewed studies focusing on the investigation of limb praxis/apraxia using repetitive transcranial magnetic stimulation (rTMS), or transcranial direct current stimulation (tDCS). Fourteen out of 139 records met the inclusion criteria, including thirteen studies with healthy subjects and one with stroke patients. The results of our systematic review suggest that in healthy subjects NIBS over left inferior parietal lobe (IPL) mainly interfered with gesture processing, by either affecting reaction times in judgment tasks or real gesturing. First promising results suggest that inhibitory continuous theta burst stimulation (cTBS) over right IPL may enhance gesturing in healthy subjects, explained by transcallosal facilitation of left IPL. In stroke patients, excitatory anodal tDCS over left IPL may improve limb apraxia. However, larger well powered and sham-controlled clinical trials are needed to expand on these proof-of-concept results, before NIBS could be a treatment option to improve limb apraxia in stroke patients.

Multimodal comprehension in left hemisphere stroke patients

Hand gestures, imagistically related to the content of speech, are ubiquitous in face-to-face communication. Here we investigated people with aphasia's (PWA) processing of speech accompanied by gestures using lesion-symptom mapping. Twenty-nine PWA and 15 matched controls were shown a picture of an object/action and then a video-clip of a speaker producing speech and/or gestures in one of the following combinations: speech-only, gesture-only, congruent speech-gesture, and incongruent speech-gesture. Participants' task was to indicate, in different blocks, whether the picture and the word matched (speech task), or whether the picture and the gesture matched (gesture task). Multivariate lesion analysis with Support Vector Regression Lesion-Symptom Mapping (SVR-LSM) showed that benefit for congruent speech-gesture was associated with 1) lesioned voxels in anterior fronto-temporal regions including inferior frontal gyrus (IFG), and sparing of posterior temporal cortex and lateral temporal-occipital regions (pTC/LTO) for the speech task, and 2) conversely, lesions to pTC/LTO and sparing of anterior regions for the gesture task. The two tasks did not share overlapping voxels. Costs from incongruent speech-gesture pairings were associated with lesioned voxels in these same anterior (for the speech task) and posterior (for the gesture task) regions, but crucially, also shared voxels in superior temporal gyrus (STG) and middle temporal gyrus (MTG), including the anterior temporal lobe. These results suggest that IFG and pTC/LTO contribute to extracting semantic information from speech and gesture, respectively; however, they are not causally involved in integrating information from the two modalities. In contrast, regions in anterior STG/MTG are associated with performance in both tasks and may thus be critical to speech-gesture integration. These conclusions are further supported by associations between performance in the experimental tasks and performance in tests assessing lexical-semantic processing and gesture recognition.

Gesture, communication, and adult acquired hearing loss

Nonverbal communication, specifically hand and arm movements (commonly known as gesture), has long been recognized and explored as a significant element in human interaction as well as potential compensatory behavior for individuals with communication difficulties. The use of gesture as a compensatory communication method in expressive and receptive human communication disorders has been the subject of much investigation. Yet within the context of adult acquired hearing loss, gesture has received limited research attention and much remains unknown about patterns of nonverbal behaviors in conversations in which hearing loss is a factor. This paper presents key elements of the background of gesture studies and the theories of gesture function and production followed by a review of research focused on adults with hearing loss and the role of gesture and gaze in rehabilitation. The current examination of the visual resource of co-speech gesture in the context of everyday interactions involving adults with acquired hearing loss suggests the need for the development of an evidence base to effect enhancements and changes in the way in which rehabilitation services are conducted.

Single Session Transcranial Magnetic Stimulation Ameliorates Hand Gesture Deficits in Schizophrenia

Social interaction is impaired in schizophrenia, including the use of hand gestures, which is linked to poor social perception and outcome. Brain imaging suggests reduced neural activity in a left-lateralized frontoparietal network during gesture preparation; therefore, gesturing might be improved through facilitation of left hemispheric brain areas or via disruption of interhemispheric inhibition from the right homolog. This study tested whether repetitive transcranial magnetic stimulation (rTMS) protocols would improve gesture performance in schizophrenia. This randomized, placebo-controlled, double-blind, crossover trial applied 3 different protocols of rTMS separated by 48 h. Twenty right-handed schizophrenia patients and 20 matched healthy controls received facilitatory intermittent theta burst stimulation (iTBS) over the left inferior frontal gyrus (IFG), inhibitory continuous theta burst stimulation (cTBS) over right inferior parietal lobe (IPL), and placebo over left IPL in randomized order. Primary outcome was change in the test of upper limb apraxia (TULIA), rated from video recordings of hand gesture performance. Secondary outcome was change in manual dexterity using the coin rotation task. Participants improved on both tasks following rTMS compared with baseline. Only patients improved gesture performance following right IPL cTBS compared with placebo (P = .013). The results of the coin rotation parallel those of the TULIA, with improvements following right IPL cTBS in patients (P = .001). Single sessions of cTBS on the right IPL substantially improved both gesture performance accuracy and manual dexterity. The findings point toward an inhibition of interhemispheric rivalry as a potential mechanism of action.

Attention/memory complaint is correlated with motor speech disorder in Parkinson's disease

BACKGROUND

The mechanisms underlying the online modulation of motor speech in Parkinson's disease (PD) have not been determined. Moreover, medical and rehabilitation interventions for PD-associated motor speech disorder (MSD) have a poor long-term prognosis.

METHODS

To compare risk factors in PD patients with MSD to those without MSD (non-MSD) and determine predictive independent risk factors correlated with the MSD phenotype, we enrolled 314 PD patients, including 250 with and 64 without MSD. We compared demographic, characteristic data, as well as PD-associated evaluations between the MSD group and non-MSD group.

RESULTS

Univariate analysis showed that demographic characteristics, including occupation, educational level, monthly income and speaking background; clinical characteristics, including lesions in the frontal and temporal lobes, and concurrent dysphagia; and PD-associated evaluations, including the activity of daily living (ADL) score, non-motor symptoms scale (NMSS) domain 4 score (perceptual problem), and NMSS domain 5 score (attention/memory) were all significantly different between the MSD and non-MSD group (all P < 0.05). Multivariate logistic regression analysis showed that educational level, frontal lesions, and NMSS domain 5 score (attention/memory) were independent risk factors for PD-associated MSD (all P < 0.005).

CONCLUSIONS

We determined an association between MSD phenotype and cognitive impairment, reflected by low-level education and related clinical profiles. Moreover, attention and memory dysfunction may play key roles in the progression of MSD in PD patients. Further studies are required to detail the mechanism underlying abnormal speech motor modulation in PD patients. Early cognitive intervention may enhance rehabilitation management and motor speech function in patients with PD-associated MSD.

Language, Gesture, and Emotional Communication: An Embodied View of Social Interaction

Spoken language is an innate ability of the human being and represents the most widespread mode of social communication. The ability to share concepts, intentions and feelings, and also to respond to what others are feeling/saying is crucial during social interactions. A growing body of evidence suggests that language evolved from manual gestures, gradually incorporating motor acts with vocal elements. In this evolutionary context, the human mirror mechanism (MM) would permit the passage from “doing something” to “communicating it to someone else.” In this perspective, the MM would mediate semantic processes being involved in both the execution and in the understanding of messages expressed by words or gestures. Thus, the recognition of action related words would activate somatosensory regions, reflecting the semantic grounding of these symbols in action information. Here, the role of the sensorimotor cortex and in general of the human MM on both language perception and understanding is addressed, focusing on recent studies on the integration between symbolic gestures and speech. We conclude documenting some evidence about MM in coding also the emotional aspects conveyed by manual, facial and body signals during communication, and how they act in concert with language to modulate other’s message comprehension and behavior, in line with an “embodied” and integrated view of social interaction.

Transcranial Direct Current Stimulation Improves Semantic Speech-Gesture Matching in Patients With Schizophrenia Spectrum Disorder

BACKGROUND

Patients with schizophrenia spectrum disorders (SSD) have severe deficits in speech and gesture processing that contribute considerably to the burden of this disorder. Brain imaging shows left inferior frontal gyrus involvement for impaired processing of co-verbal gestures in patients with schizophrenia. Recently, transcranial direct current stimulation (tDCS) of the left frontal lobe has been shown to modulate processing of co-verbal gestures in healthy subjects. Although tDCS has been used to reduce symptoms of patients with SSD, the effects of tDCS on gesture processing deficits remain hitherto unexplored.

OBJECTIVE

Here we tested the hypothesis that inhibitory cathodal tDCS of the left frontal lobe decreases pathological dysfunction and improves semantic processing of co-verbal gestures in patients with SSD.

METHODS

We measured ratings and reaction times in a speech-gesture semantic relatedness assessment task during application of frontal, frontoparietal, parietal, and sham tDCS to 20 patients with SSD and 29 healthy controls.

RESULTS

We found a specific effect of tDCS on speech-gesture relatedness ratings of patients. Frontal compared to parietal and sham stimulation significantly improved the differentiation between related and unrelated gestures. Placement of the second electrode (right frontal vs parietal) did not affect the effect of left frontal stimulation, which reduced the preexisting difference between patients and healthy controls.

CONCLUSION

Here we show that left frontal tDCS can improve semantic co-verbal gesture processing in patients with SSD. tDCS could be a viable tool to normalize processing in the left frontal lobe and facilitate direct social communicative functioning in patients with SSD.

Role of Sensorimotor Cortex in Gestural-Verbal Integration

Action comprehension that is related to language or gestural integration has been shown to engage the motor system in the brain, thus providing preliminary evidence for the gestural-verbal embodiment concept. Based on the involvement of the sensorimotor cortex (M1) in language processing, we aimed to further explore its role in the cognitive embodiment necessary for gestural-verbal integration. As such, we applied anodal (excitatory) and sham transcranial direct current stimulation (tDCS) over the left M1 (with reference electrode over the contralateral supraorbital region) during a gestural-verbal integration task where subjects had to make a decision about the semantic congruency of the gesture (prime) and the word (target). We used a cross-over within-subject design in young subjects. Attentional load and simple reaction time (RT) tasks served as control conditions, applied during stimulation (order of three tasks was counterbalanced). Our results showed that anodal (atDCS) compared to sham tDCS (stDCS) reduced RTs in the gestural-verbal integration task, specifically for incongruent pairs of gestures and verbal expressions, with no effect on control task performance. Our findings provide evidence for the involvement of the sensorimotor system in gestural-verbal integration performance. Further, our results suggest that functional modulation induced by sensorimotor tDCS may be specific to gestural-verbal integration. Future studies should now evaluate the modulatory effect of tDCS on semantic congruency by using tDCS over additional brain regions and include assessments of neural connectivity.

Alpha and Beta Oscillations Index Semantic Congruency between Speech and Gestures in Clear and Degraded Speech

Previous work revealed that visual semantic information conveyed by gestures can enhance degraded speech comprehension, but the mechanisms underlying these integration processes under adverse listening conditions remain poorly understood. We used MEG to investigate how oscillatory dynamics support speech–gesture integration when integration load is manipulated by auditory (e.g., speech degradation) and visual semantic (e.g., gesture congruency) factors. Participants were presented with videos of an actress uttering an action verb in clear or degraded speech, accompanied by a matching (mixing gesture + “mixing”) or mismatching (drinking gesture + “walking”) gesture. In clear speech, alpha/beta power was more suppressed in the left inferior frontal gyrus and motor and visual cortices when integration load increased in response to mismatching versus matching gestures. In degraded speech, beta power was less suppressed over posterior STS and medial temporal lobe for mismatching compared with matching gestures, showing that integration load was lowest when speech was degraded and mismatching gestures could not be integrated and disambiguate the degraded signal. Our results thus provide novel insights on how low-frequency oscillatory modulations in different parts of the cortex support the semantic audiovisual integration of gestures in clear and degraded speech: When speech is clear, the left inferior frontal gyrus and motor and visual cortices engage because higher-level semantic information increases semantic integration load. When speech is degraded, posterior STS/middle temporal gyrus and medial temporal lobe are less engaged because integration load is lowest when visual semantic information does not aid lexical retrieval and speech and gestures cannot be integrated.

The Role of Gamma Oscillations During Integration of Metaphoric Gestures and Abstract Speech

Metaphoric (MP) co-speech gestures are commonly used during daily communication. They communicate about abstract information by referring to gestures that are clearly concrete (e.g., raising a hand for “the level of the football game is high”). To understand MP co-speech gestures, a multisensory integration at semantic level is necessary between abstract speech and concrete gestures. While semantic gesture-speech integration has been extensively investigated using functional magnetic resonance imaging, evidence from electroencephalography (EEG) is rare. In the current study, we set out an EEG experiment, investigating the processing of MP vs. iconic (IC) co-speech gestures in different contexts, to reveal the oscillatory signature of MP gesture integration. German participants (n = 20) viewed video clips with an actor performing both types of gestures, accompanied by either comprehensible German or incomprehensible Russian (R) speech, or speaking German sentences without any gestures. Time-frequency analysis of the EEG data showed that, when gestures were accompanied by comprehensible German speech, MP gestures elicited decreased gamma band power (50–70 Hz) between 500 and 700 ms in the parietal electrodes when compared to IC gestures, and the source of this effect was localized to the right middle temporal gyrus. This difference is likely to reflect integration processes, as it was reduced in the R language and no-gesture conditions. Our findings provide the first empirical evidence suggesting the functional relationship between gamma band oscillations and higher-level semantic processes in a multisensory setting.

The connectivity signature of co-speech gesture integration: The superior temporal sulcus modulates connectivity between areas related to visual gesture and auditory speech processing

Humans integrate information communicated by speech and gestures. Functional magnetic resonance imaging (fMRI) studies suggest that the posterior superior temporal sulcus (STS) and adjacent gyri are relevant for multisensory integration. However, a connectivity model representing this essential combinatory process is still missing. Here, we used dynamic causal modeling for fMRI to analyze the effective connectivity pattern between middle temporal gyrus (MTG), occipital cortex (OC) and STS associated with auditory verbal, visual gesture-related, and integrative processing, respectively, to unveil the neural mechanisms underlying integration of intrinsically meaningful gestures (e.g., "Thumbs-up gesture") and corresponding speech. 20 participants were presented videos of an actor either performing intrinsic meaningful gestures in the context of German or Russian sentences, or speaking a German sentence without gesture, while performing a content judgment task. The connectivity analyses resulted in a winning model that included bidirectional intrinsic connectivity between all areas. Furthermore, the model included modulations of both connections to the STS (OC→STS; MTG→STS), and non-linear modulatory effects of the STS on bidirectional connections between MTG and OC. Coupling strength in the occipital pathway (OC→STS) correlated with gesture related advantages in task performance, whereas the temporal pathway (MTG→STS) correlated with performance in the speech only condition. Coupling between MTG and OC correlated negatively with subsequent memory performance for sentences of the Gesture-German condition. Our model provides a first step towards a better understanding of speech-gesture integration on network level. It corroborates the importance of the STS during audio-visual integration by showing that this region inhibits direct auditory-visual coupling.