The Role of Gamma Oscillations During Integration of Metaphoric Gestures and Abstract Speech

Electroencephalographic Effective Connectivity Analysis of the Neural Networks during Gesture and Speech Production Planning in Young Adults

Gestures and speech, as linked communicative expressions, form an integrated system. Previous functional magnetic resonance imaging studies have suggested that neural networks for gesture and spoken word production share similar brain regions consisting of fronto-temporo-parietal brain regions. However, information flow within the neural network may dynamically change during the planning of two communicative expressions and also differ between them. To investigate dynamic information flow in the neural network during the planning of gesture and spoken word generation in this study, participants were presented with spatial images and were required to plan the generation of gestures or spoken words to represent the same spatial situations. The evoked potentials in response to spatial images were recorded to analyze the effective connectivity within the neural network. An independent component analysis of the evoked potentials indicated 12 clusters of independent components, the dipoles of which were located in the bilateral fronto-temporo-parietal brain regions and on the medial wall of the frontal and parietal lobes. Comparison of effective connectivity indicated that information flow from the right middle cingulate gyrus (MCG) to the left supplementary motor area (SMA) and from the left SMA to the left precentral area increased during gesture planning compared with that of word planning. Furthermore, information flow from the right MCG to the left superior frontal gyrus also increased during gesture planning compared with that of word planning. These results suggest that information flow to the brain regions for hand praxis is more strongly activated during gesture planning than during word planning.

The Processing of Semantic Complexity and Cospeech Gestures in Schizophrenia: A Naturalistic, Multimodal fMRI Study

Schizophrenia is marked by aberrant processing of complex speech and gesture, which may contribute functionally to its impaired social communication. To date, extant neuroscientific studies of schizophrenia have largely investigated dysfunctional speech and gesture in isolation, and no prior research has examined how the two communicative channels may interact in more natural contexts. Here, we tested if patients with schizophrenia show aberrant neural processing of semantically complex story segments, and if speech-associated gestures (co-speech gestures) might modulate this effect. In a functional MRI study, we presented to 34 participants (16 patients and 18 matched-controls) an ecologically-valid retelling of a continuous story, performed via speech and spontaneous gestures. We split the entire story into ten-word segments, and measured the semantic complexity for each segment with idea density, a linguistic measure that is commonly used clinically to evaluate aberrant language dysfunction at the semantic level. Per segment, the presence of numbers of gestures varied (n = 0, 1, +2). Our results suggest that, in comparison to controls, patients showed reduced activation for more complex segments in the bilateral middle frontal and inferior parietal regions. Importantly, this neural aberrance was normalized in segments presented with gestures. Thus, for the first time with a naturalistic multimodal stimulation paradigm, we show that gestures reduced group differences when processing a natural story, probably by facilitating the processing of semantically complex segments of the story in schizophrenia.

Modality-specific dysfunctional neural processing of social-abstract and non-social-concrete information in schizophrenia

Schizophrenia is characterized by marked communication dysfunctions encompassing potential impairments in the processing of social-abstract and non-social-concrete information, especially in everyday situations where multiple modalities are present in the form of speech and gesture. To date, the neurobiological basis of these deficits remains elusive. In a functional magnetic resonance imaging (fMRI) study, 17 patients with schizophrenia or schizoaffective disorder, and 18 matched controls watched videos of an actor speaking, gesturing (unimodal), and both speaking and gesturing (bimodal) about social or non-social events in a naturalistic way. Participants were asked to judge whether each video contains person-related (social) or object-related (non-social) information. When processing social-abstract content, patients showed reduced activation in the medial prefrontal cortex (mPFC) only in the gesture but not in the speech condition. For non-social-concrete content, remarkably, reduced neural activation for patients in the left postcentral gyrus and the right insula was observed only in the speech condition. Moreover, in the bimodal conditions, patients displayed improved task performance and comparable activation to controls in both social and non-social content. To conclude, patients with schizophrenia displayed modality-specific aberrant neural processing of social and non-social information, which is not present for the bimodal conditions. This finding provides novel insights into dysfunctional multimodal communication in schizophrenia, and may have potential therapeutic implications.

Age-related effects on the neural processing of semantic complexity in a continuous narrative: Modulation by gestures already present in young to middle-aged adults

The processing of semantically complex speech is a demanding task which can be facilitated by speech-associated arm and hand gestures. However, the role of age concerning the perception of semantic complexity and the influence of gestures in this context remains unclear. The goal of this study was to investigate if age-related differences are already present in early adulthood during the processing of semantic complexity and gestures. To this end, we analyzed fMRI images of a sample of 38 young and middle-aged participants (age-range: 19-55). They had the task to listen and to watch a narrative. The narrative contained segments varying in the degree of semantic complexity, and they were spontaneously accompanied by gestures. The semantic complexity of the story was measured by the idea density. Consistent with previous findings in young adults, we observed increased activation for passages with lower compared to higher complexity in bilateral temporal areas and the precuneus. BOLD signal in the left frontal and left parietal regions correlated during the perception of complex passages with increasing age. This correlation was reduced for passages presented with gestures. Median-split based post-hoc comparisons confirmed that group differences between younger (19-23 years) and older adults within the early adult lifespan (24-55 years) were significantly reduced in passages with gestures. Our results suggest that older adults within early adulthood adapt to the requirements of highly complex passages activating additional regions when no gesture information is available. Gestures might play a facilitative role with increasing age, especially when speech is complex.

Gesture's body orientation modulates the N400 for visual sentences primed by gestures

Body orientation of gesture entails social-communicative intention, and may thus influence how gestures are perceived and comprehended together with auditory speech during face-to-face communication. To date, despite the emergence of neuroscientific literature on the role of body orientation on hand action perception, limited studies have directly investigated the role of body orientation in the interaction between gesture and language. To address this research question, we carried out an electroencephalography (EEG) experiment presenting to participants (n = 21) videos of frontal and lateral communicative hand gestures of 5 s (e.g., raising a hand), followed by visually presented sentences that are either congruent or incongruent with the gesture (e.g., "the mountain is high/low…"). Participants underwent a semantic probe task, judging whether a target word is related or unrelated to the gesture-sentence event. EEG results suggest that, during the perception phase of handgestures, while both frontal and lateral gestures elicited a power decrease in both the alpha (8-12 Hz) and the beta (16-24 Hz) bands, lateral versus frontal gestures elicited reduced power decrease in the beta band, source-located to the medial prefrontal cortex. For sentence comprehension, at the critical word whose meaning is congruent/incongruent with the gesture prime, frontal gestures elicited an N400 effect for gesture-sentence incongruency. More importantly, this incongruency effect was significantly reduced for lateral gestures. These findings suggest that body orientation plays an important role in gesture perception, and that its inferred social-communicative intention may influence gesture-language interaction at semantic level.

Gesture, communication, and adult acquired hearing loss

Nonverbal communication, specifically hand and arm movements (commonly known as gesture), has long been recognized and explored as a significant element in human interaction as well as potential compensatory behavior for individuals with communication difficulties. The use of gesture as a compensatory communication method in expressive and receptive human communication disorders has been the subject of much investigation. Yet within the context of adult acquired hearing loss, gesture has received limited research attention and much remains unknown about patterns of nonverbal behaviors in conversations in which hearing loss is a factor. This paper presents key elements of the background of gesture studies and the theories of gesture function and production followed by a review of research focused on adults with hearing loss and the role of gesture and gaze in rehabilitation. The current examination of the visual resource of co-speech gesture in the context of everyday interactions involving adults with acquired hearing loss suggests the need for the development of an evidence base to effect enhancements and changes in the way in which rehabilitation services are conducted.

Sleep Electroencephalographic Response to Respiratory Events in Patients With Moderate Sleep Apnea-Hypopnea Syndrome

Sleep apnea–hypopnea syndrome is a common breathing disorder that can lead to organic brain injury, prevent memory consolidation, and cause other adverse mental-related complications. Brain activity while sleeping during respiratory events is related to these dysfunctions. In this study, we analyzed variations in electroencephalography (EEG) signals before, during, and after such events. Absolute and relative powers, as well as symbolic transfer entropy (STE) of scalp EEG signals, were calculated to unveil the activity of brain regions and information interactions between them, respectively. During the respiratory events, only low-frequency power increased during rapid eye movement (REM) stage (δ-band absolute and relative power) and N1 (δ- and θ-band absolute power, δ-band relative power) sleep. But absolute power increased in low- and medium-frequency bands (δ, θ, α, and σ bands), and relative power increased mainly in the medium-frequency band (α and σ bands) during stage N2 sleep. After the respiratory events, absolute power increased in all frequency bands and sleep stages, but relative power increased in medium and high frequencies. Regarding information interactions, the β-band STE decreased during and after events. In the γ band, the intrahemispheric STE increased during events and decreased afterward. Moreover, the interhemisphere STE increased after events during REM and stage N1 sleep. The EEG changes throughout respiratory events are supporting evidence for previous EEG knowledge of the impact of sleep apnea on the brain. These findings may provide insights into the influence of the sleep apnea–hypopnea syndrome on cognitive function and neuropsychiatric defects.

Dynamic Causal Modelling suggests impaired effective connectivity in patients with schizophrenia spectrum disorders during gesture-speech integration

Integrating visual and auditory information during gesture-speech integration (GSI) is important for successful social communication, which is often impaired in schizophrenia. Several studies suggested the posterior superior temporal sulcus (pSTS) to be a relevant multisensory integration site. However, intact STS activation patterns were often reported in patients. Thus, here we used Dynamic Causal Modelling (DCM) to analyze whether information processing in schizophrenia spectrum disorders (SSD) is impaired during GSI on network level. We investigated GSI in three different samples. First, we replicated a recently published connectivity model for GSI in a healthy subject group (n = 19). Second, we investigated differences between patients with SSD and a matched healthy control group (n = 17 each). Participants were presented videos of an actor performing intrinsically meaningful gestures accompanied by spoken sentences in German or Russian, or just telling a German sentence without gestures. Across all groups, fMRI analyses revealed similar activation patterns, and DCM analyses resulted in the same winning model for GSI. This finding directly replicates previous results. However, patients revealed significantly reduced connectivity in the verbal pathway (from left middle temporal gyrus (MTG) to left STS). The clinical significance of this connection is supported by its correlations with the severity of concretism and a subscale of negative symptoms (SANS). Our model confirms the importance of the pSTS as integration site during audio-visual integration. Patients showed generally intact connectivity during GSI, but revealed impaired information transfer via the verbal pathway. This might be the basis of interpersonal communication problems in patients with SSD.