A brief self-rating scale for the assessment of individual differences in gesture perception and production

Speech-Gesture Matching and Schizotypal Traits: A Network Approach

BACKGROUND AND HYPOTHESIS

Impaired speech-gesture matching has repeatedly been shown in patients with schizophrenia spectrum disorders. Here, we tested the hypothesis that schizotypal traits in the general population are related to reduced speech-gesture matching performance and reduced self-reports about gesture perception. We further explored the relationships between facets of schizotypy and gesture processing in a network model.

STUDY DESIGN

Participants (1094 mainly healthy adults) were presented with concrete or abstract sentences accompanied with videos showing related or unrelated gestures. For each video, participants evaluated the alignment between speech and gesture. They also completed self-rating scales about the perception and production of gestures (Brief Assessment of Gesture scale) and schizotypal traits (Schizotypal Personality Questionnaire-Brief 22-item version). We analyzed bivariate associations and estimated a non-regularized partial Spearman correlation network. We characterized the network by analyzing bridge centrality and controllability metrics of nodes.

STUDY RESULTS

We found a negative relationship between both concrete and abstract gesture-speech matching performance and overall schizotypy. In the network, disorganization had the highest average controllability and it was negatively related to abstract speech-gesture matching. Bridge centralities indicated that self-reported production of gestures to enhance communication in social interactions connects self-reported gesture perception, schizotypal traits, and gesture processing task performance.

CONCLUSION

The association between impaired abstract speech-gesture matching and disorganization supports a continuum between schizophrenia and schizotypy. Using gestures to facilitate communication connects subjective and objective aspects of gesture processing and schizotypal traits. Future interventional studies in patients should test the potential causal pathways implied by this network model.

Using virtual reality to assess gesture performance deficits in schizophrenia patients

Introduction

Gesture performance deficits are prevalent in schizophrenia patients and are strongly associated with poor social communication skills and community functioning, affecting their overall quality of life. Currently, video-recording technology is widely used in clinical settings to assess gesture production deficits in schizophrenia patients. Nevertheless, the subjective evaluation of video-recordings can encumber task assessment. The present study will aim to use virtual reality to examine its potential use as an alternative tool to objectively measure gesture performance accuracy in schizophrenia patients and healthy controls.

Methods

Gesture performance in the virtual reality setting will be based on the well-established Test of Upper Limb Apraxia. Participants will be immersed in a virtual environment where they will experience themselves being embodied in a collocated virtual body seen from a first-person perspective. Motion trackers will be placed on participants' hands and elbows to track upper body movements in real-time, and to record gesture movement for later analysis. Participants will see a virtual agent sitting across from them, with a virtual table in between. The agent will perform various types of gestures and the participants' task will be to imitate those gestures as accurately as possible. Measurements from the tracking devices will be stored and analyzed to address gesture performance accuracy across groups.

Discussion

This study aims to provide objective measurements of gesture performance accuracy in schizophrenia patients. If successful, the results will provide new knowledge to the gesture literature and offer the potential for novel therapeutic interventions using virtual reality technologies. Such interventions can improve gesturing and thus advance social communication skills in schizophrenia patients.

Exploring the Emotional Functions of Co-Speech Hand Gesture in Language and Communication

Research over the past four decades has built a convincing case that co-speech hand gestures play a powerful role in human cognition . However, this recent focus on the cognitive function of gesture has, to a large extent, overlooked its emotional role-a role that was once central to research on bodily expression. In the present review, we first give a brief summary of the wealth of research demonstrating the cognitive function of co-speech gestures in language acquisition, learning, and thinking. Building on this foundation, we revisit the emotional function of gesture across a wide range of communicative contexts, from clinical to artistic to educational, and spanning diverse fields, from cognitive neuroscience to linguistics to affective science. Bridging the cognitive and emotional functions of gesture highlights promising avenues of research that have varied practical and theoretical implications for human-machine interactions, therapeutic interventions, language evolution, embodied cognition, and more.

Multimodal speech-gesture training in patients with schizophrenia spectrum disorder: Effects on quality of life and neural processing

Dysfunctional social communication is one of the most stable characteristics in patients with schizophrenia spectrum disorder (SSD) that severely affects quality of life. Interpreting abstract speech and integrating nonverbal information is particularly affected. Considering the difficulty to treat communication dysfunctions with usual intervention, we investigated the possibility to apply a multimodal speech-gesture (MSG) training. In the MSG training, we offered 8 sessions (60 min each) including perceptive and expressive tasks as well as meta-learning elements and transfer exercises to 29 patients with SSD. In a within-group crossover design, patients were randomized to a TAU-first (treatment as usual first, then MSG training) group (N = 20) or a MSG-first (MSG training first, then TAU only) group (N = 9), and were compared to healthy controls (N = 17). Outcomes were quality of life and related changes in the neural processing of abstract speech-gesture information, which were measured pre-post training through standardized psychological questionnaires and functional Magnetic Resonance Imaging, respectively. Pre-training, patients showed reduced quality of life as compared to controls but improved significantly during the training. Strikingly, this improvement was correlated with neural activation changes in the middle temporal gyrus for the processing of abstract multimodal content. Improvement during training, self-report measures and ratings of relatives confirmed the MSG-related changes. Together, we provide first promising results of a novel multimodal speech-gesture training for patients with schizophrenia. We could link training induced changes in speech-gesture processing to changes in quality of life, demonstrating the relevance of intact communication skills and gesture processing for well-being.

Brain Stimulation and Group Therapy to Improve Gesture and Social Skills in Schizophrenia-The Study Protocol of a Randomized, Sham-Controlled, Three-Arm, Double-Blind Trial

An important component of nonverbal communication is gesture performance, which is strongly impaired in 2/3 of patients with schizophrenia. Gesture deficits in schizophrenia are linked to poor social functioning and reduced quality of life. Therefore, interventions that can help alleviate these deficits in schizophrenia are crucial. Here, we describe an ongoing randomized, double-blind 3-arm, sham-controlled trial that combines two interventions to reduce gesture deficits in schizophrenia patients. The combined interventions are continuous theta burst stimulation (cTBS) and social cognitive remediation therapy (SCRT). We will randomize 72 patients with schizophrenia spectrum disorders in three different groups of 24 patients. The first group will receive real cTBS and real SCRT, the second group will receive sham cTBS and real SCRT, and finally the third group will receive sham SCRT. Here, the sham treatments are, as per definition, inactive interventions that mimic as closely as possible the real treatments (similar to placebo). In addition, 24 age- and gender-matched controls with no interventions will be added for comparison. Measures of nonverbal communication, social cognition, and multimodal brain imaging will be applied at baseline and after intervention. The main research aim of this project will be to test whether the combination of cTBS and SCRT improves gesture performance and social functioning in schizophrenia patients more than standalone cTBS, SCRT or sham psychotherapy. We hypothesize that the patient group receiving the combined interventions will be superior in improving gesture performance.

CLINICAL TRIAL REGISTRATION

[www.ClinicalTrials.gov], identifier [NCT04106427].

The Processing of Semantic Complexity and Cospeech Gestures in Schizophrenia: A Naturalistic, Multimodal fMRI Study

Schizophrenia is marked by aberrant processing of complex speech and gesture, which may contribute functionally to its impaired social communication. To date, extant neuroscientific studies of schizophrenia have largely investigated dysfunctional speech and gesture in isolation, and no prior research has examined how the two communicative channels may interact in more natural contexts. Here, we tested if patients with schizophrenia show aberrant neural processing of semantically complex story segments, and if speech-associated gestures (co-speech gestures) might modulate this effect. In a functional MRI study, we presented to 34 participants (16 patients and 18 matched-controls) an ecologically-valid retelling of a continuous story, performed via speech and spontaneous gestures. We split the entire story into ten-word segments, and measured the semantic complexity for each segment with idea density, a linguistic measure that is commonly used clinically to evaluate aberrant language dysfunction at the semantic level. Per segment, the presence of numbers of gestures varied (n = 0, 1, +2). Our results suggest that, in comparison to controls, patients showed reduced activation for more complex segments in the bilateral middle frontal and inferior parietal regions. Importantly, this neural aberrance was normalized in segments presented with gestures. Thus, for the first time with a naturalistic multimodal stimulation paradigm, we show that gestures reduced group differences when processing a natural story, probably by facilitating the processing of semantically complex segments of the story in schizophrenia.

Enhancing memory using enactment: does meaning matter in action production?

ABSTRACTEnactment is an encoding strategy in which performing an action related to a target item enhances memory for that word, relative to verbal encoding. Precisely how this motor activity aids recall is unclear. We examined whether the action created during encoding needed to be semantically relevant to the to-be-remembered word, to enhance memory. In Experiment 1, participants were asked to either (a) enact, (b) perform unrelated motoric gestures, or (c) read forty-five action verbs. On a subsequent free-recall test, memory for enacted words was significantly higher relative to words read, or encoded with unrelated gestures. In Experiment 2, to reduce the ambiguity associated with initiating an unrelated gesture, participants were instructed to write target words in the air. Results were similar to Experiment 1. In Experiment 3, we replicated the results of Experiment 2 using video conferencing to record the onset time of action initiation for enacted, unrelated gesture, and read trials. Results showed that planning of meaningful actions may also contribute to the memory performance as evidenced by a longer onset time to initiate an action on enactment relative to gesturing and reading trials. These findings suggest that planning and executing meaningful actions drive the enactment benefit.

Hand gesture performance is impaired in major depressive disorder: A matter of working memory performance?

OBJECTIVE

Individuals with depression exhibit numerous interpersonal deficits. As effective use of gestures is critical for social communication, it is possible that depressed individuals' interpersonal deficits may be due to deficits in gesture performance. The present study thus compared gesture performance of depressed patients and controls and examined whether these deficits relate to cognitive and other domains of dysfunction.

METHODS

Gesture performance was evaluated in 30 depressed patients and 30 controls using the Test of Upper Limb Apraxia (TULIA). Clinical rating scales were assessed to determine if gesture deficits were associated with motor, cognitive or functional outcomes.

RESULTS

Compared to controls, depressed patients exhibited impaired gesture performance with 2/3 of the patients demonstrating gesture deficits. Within depressed patients, gesture performance was highly correlated with working memory abilities. In contrast, no association between gesture performance and gestural knowledge, psychomotor retardation, depression severity, or frontal dysfunction was observed in patients.

LIMITATIONS

This is a cross-sectional study and a larger size would have allowed for confident detection of more subtle, but potentially relevant effects.

CONCLUSION

Gesture performance is impaired in depressed patients, and appears to be related to poor working memory abilities, suggesting a disruption in the retrieval of gestural cues indicative of a distinct clinical phenomenon that might be related to social functioning.

Aging and working memory modulate the ability to benefit from visible speech and iconic gestures during speech-in-noise comprehension

When comprehending speech-in-noise (SiN), younger and older adults benefit from seeing the speaker's mouth, i.e. visible speech. Younger adults additionally benefit from manual iconic co-speech gestures. Here, we investigate to what extent younger and older adults benefit from perceiving both visual articulators while comprehending SiN, and whether this is modulated by working memory and inhibitory control. Twenty-eight younger and 28 older adults performed a word recognition task in three visual contexts: mouth blurred (speech-only), visible speech, or visible speech + iconic gesture. The speech signal was either clear or embedded in multitalker babble. Additionally, there were two visual-only conditions (visible speech, visible speech + gesture). Accuracy levels for both age groups were higher when both visual articulators were present compared to either one or none. However, older adults received a significantly smaller benefit than younger adults, although they performed equally well in speech-only and visual-only word recognition. Individual differences in verbal working memory and inhibitory control partly accounted for age-related performance differences. To conclude, perceiving iconic gestures in addition to visible speech improves younger and older adults' comprehension of SiN. Yet, the ability to benefit from this additional visual information is modulated by age and verbal working memory. Future research will have to show whether these findings extend beyond the single word level.

The role of the left and right inferior frontal gyrus in processing metaphoric and unrelated co-speech gestures

Gestures are an integral part of in-person conversations and complement the meaning of the speech they accompany. The neural processing of co-speech gestures is supported by a mostly left-lateralized network of fronto-temporal regions. However, in contrast to iconic gestures, metaphoric as well as unrelated gestures have been found to more strongly engage the left and right inferior frontal gyrus (IFG), respectively. With this study, we conducted the first systematic comparison of all three types of gestures and resulting potential laterality effects. During collection of functional imaging data, 74 subjects were presented with 5 s videos of abstract speech with related metaphoric gestures, concrete speech with related iconic gestures and concrete speech with unrelated gestures. They were asked to judge whether the content of the speech and gesture matched or not. Differential contrasts revealed that both abstract related and concrete unrelated compared to concrete related stimuli elicited stronger activation of the bilateral IFG. Analyses of lateralization indices for IFG activation further showed a left hemispheric dominance for metaphoric gestures and a right hemispheric dominance for unrelated gestures. Our results give support to the hypothesis that the bilateral IFG is activated specifically when processing load for speech-gesture combinations is high. In addition, laterality effects indicate a stronger involvement of the right IFG in mismatch detection and conflict processing, whereas the left IFG performs the actual integration of information from speech and gesture.

Sensorimotor and Activity Psychosis-Risk (SMAP-R) Scale: An Exploration of Scale Structure With Replication and Validation

BACKGROUND

Sensorimotor abnormalities precede and predict the onset of psychosis. Despite the practical utility of sensorimotor abnormalities for early identification, prediction, and individualized medicine applications, there is currently no dedicated self-report instrument designed to capture these important behaviors. The current study assessed and validated a questionnaire designed for use in individuals at clinical high-risk for psychosis (CHR).

METHODS

The current study included both exploratory (n = 3009) and validation (n = 439) analytic datasets-that included individuals identified as meeting criteria for a CHR syndrome (n = 84)-who completed the novel Sensorimotor Abnormalities and Psychosis-Risk (SMAP-R) Scale, clinical interviews and a finger-tapping task. The structure of the scale and reliability of items were consistent across 2 analytic datasets. The resulting scales were assessed for discriminant validity across CHR, community sample non-psychiatric volunteer, and clinical groups.

RESULTS

The scale showed a consistent structure across 2 analytic datasets subscale structure. The resultant subscale structure was consistent with conceptual models of sensorimotor pathology in psychosis (coordination and dyskinesia) in both the exploratory and the validation analytic dataset. Further, these subscales showed discriminant, predictive, and convergent validity. The sensorimotor abnormality scales discriminated CHR from community sample non-psychiatric controls and clinical samples. Finally, these subscales predicted to risk calculator scores and showed convergent validity with sensorimotor performance on a finger-tapping task.

CONCLUSION

The SMAP-R scale demonstrated good internal, discriminant, predictive, and convergent validity, and subscales mapped on to conceptually relevant sensorimotor circuits. Features of the scale may facilitate widespread incorporation of sensorimotor screening into psychosis-risk research and practice.

Modality-specific dysfunctional neural processing of social-abstract and non-social-concrete information in schizophrenia

Schizophrenia is characterized by marked communication dysfunctions encompassing potential impairments in the processing of social-abstract and non-social-concrete information, especially in everyday situations where multiple modalities are present in the form of speech and gesture. To date, the neurobiological basis of these deficits remains elusive. In a functional magnetic resonance imaging (fMRI) study, 17 patients with schizophrenia or schizoaffective disorder, and 18 matched controls watched videos of an actor speaking, gesturing (unimodal), and both speaking and gesturing (bimodal) about social or non-social events in a naturalistic way. Participants were asked to judge whether each video contains person-related (social) or object-related (non-social) information. When processing social-abstract content, patients showed reduced activation in the medial prefrontal cortex (mPFC) only in the gesture but not in the speech condition. For non-social-concrete content, remarkably, reduced neural activation for patients in the left postcentral gyrus and the right insula was observed only in the speech condition. Moreover, in the bimodal conditions, patients displayed improved task performance and comparable activation to controls in both social and non-social content. To conclude, patients with schizophrenia displayed modality-specific aberrant neural processing of social and non-social information, which is not present for the bimodal conditions. This finding provides novel insights into dysfunctional multimodal communication in schizophrenia, and may have potential therapeutic implications.

Age-related effects on the neural processing of semantic complexity in a continuous narrative: Modulation by gestures already present in young to middle-aged adults

The processing of semantically complex speech is a demanding task which can be facilitated by speech-associated arm and hand gestures. However, the role of age concerning the perception of semantic complexity and the influence of gestures in this context remains unclear. The goal of this study was to investigate if age-related differences are already present in early adulthood during the processing of semantic complexity and gestures. To this end, we analyzed fMRI images of a sample of 38 young and middle-aged participants (age-range: 19-55). They had the task to listen and to watch a narrative. The narrative contained segments varying in the degree of semantic complexity, and they were spontaneously accompanied by gestures. The semantic complexity of the story was measured by the idea density. Consistent with previous findings in young adults, we observed increased activation for passages with lower compared to higher complexity in bilateral temporal areas and the precuneus. BOLD signal in the left frontal and left parietal regions correlated during the perception of complex passages with increasing age. This correlation was reduced for passages presented with gestures. Median-split based post-hoc comparisons confirmed that group differences between younger (19-23 years) and older adults within the early adult lifespan (24-55 years) were significantly reduced in passages with gestures. Our results suggest that older adults within early adulthood adapt to the requirements of highly complex passages activating additional regions when no gesture information is available. Gestures might play a facilitative role with increasing age, especially when speech is complex.

Gesture Use and Processing: A Review on Individual Differences in Cognitive Resources

Speakers use spontaneous hand gestures as they speak and think. These gestures serve many functions for speakers who produce them as well as for listeners who observe them. To date, studies in the gesture literature mostly focused on group-comparisons or the external sources of variation to examine when people use, process, and benefit from using and observing gestures. However, there are also internal sources of variation in gesture use and processing. People differ in how frequently they use gestures, how salient their gestures are, for what purposes they produce gestures, and how much they benefit from using and seeing gestures during comprehension and learning depending on their cognitive dispositions. This review addresses how individual differences in different cognitive skills relate to how people employ gestures in production and comprehension across different ages (from infancy through adulthood to healthy aging) from a functionalist perspective. We conclude that speakers and listeners can use gestures as a compensation tool during communication and thinking that interacts with individuals' cognitive dispositions.

Feeling addressed! The neural processing of social communicative cues in patients with major depression

The feeling of being addressed is the first step in a complex processing stream enabling successful social communication. Social impairments are a relevant characteristic of patients with major depressive disorder (MDD). Here, we investigated a mechanism which—if impaired—might contribute to withdrawal or isolation in MDD, namely, the neural processing of social cues such as body orientation and gesture. During funtional magnetic resonance imaging (fMRI) data acquisition, 33 patients with MDD and 43 healthy control subjects watched video clips of a speaking actor: one version with a gesture accompanying the speech and one without gesture. Videos were filmed simultaneously from two different viewpoints: one with the actor facing the viewer head‐on (frontal) and one side‐view (lateral). After every clip, the participants were instructed to evaluate whether they felt addressed or not. Despite overall comparable addressment ratings and a large overlap in activation patterns in MDD and healthy subjects for gesture processing, the anterior cingulate cortex, bilateral superior/middle frontal cortex, and right angular gyrus were more strongly activated in patients than in healthy subjects for the frontal conditions. Our analyses revealed that patients showed specifically higher activation than healthy subjects for the frontal condition without gesture in regions including the posterior cingulate cortex, left prefrontal cortex, and the left hippocampus. We conclude that MDD patients can recognize and interpret social cues such as gesture or body orientation; however, they seem to require more neural resources. This additional effort might affect successful communication and contribute to social isolation in MDD.

Dynamic Causal Modelling suggests impaired effective connectivity in patients with schizophrenia spectrum disorders during gesture-speech integration

Integrating visual and auditory information during gesture-speech integration (GSI) is important for successful social communication, which is often impaired in schizophrenia. Several studies suggested the posterior superior temporal sulcus (pSTS) to be a relevant multisensory integration site. However, intact STS activation patterns were often reported in patients. Thus, here we used Dynamic Causal Modelling (DCM) to analyze whether information processing in schizophrenia spectrum disorders (SSD) is impaired during GSI on network level. We investigated GSI in three different samples. First, we replicated a recently published connectivity model for GSI in a healthy subject group (n = 19). Second, we investigated differences between patients with SSD and a matched healthy control group (n = 17 each). Participants were presented videos of an actor performing intrinsically meaningful gestures accompanied by spoken sentences in German or Russian, or just telling a German sentence without gestures. Across all groups, fMRI analyses revealed similar activation patterns, and DCM analyses resulted in the same winning model for GSI. This finding directly replicates previous results. However, patients revealed significantly reduced connectivity in the verbal pathway (from left middle temporal gyrus (MTG) to left STS). The clinical significance of this connection is supported by its correlations with the severity of concretism and a subscale of negative symptoms (SANS). Our model confirms the importance of the pSTS as integration site during audio-visual integration. Patients showed generally intact connectivity during GSI, but revealed impaired information transfer via the verbal pathway. This might be the basis of interpersonal communication problems in patients with SSD.

A Multimodal Speech-Gesture Training Intervention for Patients With Schizophrenia and Its Neural Underpinnings - the Study Protocol of a Randomized Controlled Pilot Trial

Dysfunctional social communication is one of the most stable characteristics in patients with schizophrenia that also affects quality of life. Interpreting abstract speech and integrating nonverbal modalities is particularly affected. Considering the impact of communication on social life but failure to treat communication dysfunctions with usual treatment, we will investigate the possibility to improve verbal and non-verbal communication in schizophrenia by applying a multimodal speech-gesture training (MSG training). Here we describe the newly developed MSG training program and the study design for the first clinical investigation. The intervention contains perceptive rating (match/mismatch of sentence and gesture) and memory tasks (n-back tasks), imitation and productive tasks (e.g., SG fluency-similar to verbal fluency where words are accompanied by gesture). In addition, we offer information about gesture as meta-learning element as well as homework for reasons of transfer to everyday life as part of every session. In the MSG training intervention, we offer eight sessions (60 min each) of training. The first pilot study is currently conducted as a single-center, randomized controlled trial of speech-gesture intervention versus wait-list control with a follow-up. Outcomes are measured through pre-post-fMRI and standardized psychological questionnaires comparing two subject groups (30 patients with schizophrenia and 30 healthy controls). Patients and healthy controls are randomized in two intervention groups (with 20 being in the wait-training group and 10 in the training-follow-up group). With our study design we will be able to demonstrate the beneficial effect of the MSG training intervention on behavioral and neural levels.

CLINICAL TRIAL REGISTRATION

DRKS.de, identifier DRKS00015118.

Evidence for gesture-speech mismatch detection impairments in schizophrenia

Patients with schizophrenia suffer from impairments in the perception and production of gestures. The extent to which patients can access the semantic association between speech and co-verbal gestures in concrete or abstract/metaphorical meaning contexts is unknown. We investigated 1) how patients differ from controls in gesture matching performance, 2) how performance differs in the context of abstract versus concrete meaning, and 3) whether formal thought disorder (FTD) symptom severity predicts task impairment. Forty-five patients with schizophrenia spectrum disorders (two subgroups, "mild" and "severe") took part in this study. Participants were presented with video clips, each showing an actor saying a sentence while producing a gesture. Sentences contained either concrete or abstract/metaphorical information, and the accompanying gesture was either semantically related or unrelated to the sentence. Participants indicated via button press whether the gesture matched the content of the verbal utterance. Both patient subgroups demonstrated reduced performance in all comparisons. A significant interaction was found between patient subgroup and sentence abstractness. Task performance was worst for patients with severe positive FTD symptomatology in the abstract condition, while there were no patient subgroup differences in the concrete condition. These data shed new light on gesture-speech mismatch detection impairments in schizophrenia.

The connectivity signature of co-speech gesture integration: The superior temporal sulcus modulates connectivity between areas related to visual gesture and auditory speech processing

Humans integrate information communicated by speech and gestures. Functional magnetic resonance imaging (fMRI) studies suggest that the posterior superior temporal sulcus (STS) and adjacent gyri are relevant for multisensory integration. However, a connectivity model representing this essential combinatory process is still missing. Here, we used dynamic causal modeling for fMRI to analyze the effective connectivity pattern between middle temporal gyrus (MTG), occipital cortex (OC) and STS associated with auditory verbal, visual gesture-related, and integrative processing, respectively, to unveil the neural mechanisms underlying integration of intrinsically meaningful gestures (e.g., "Thumbs-up gesture") and corresponding speech. 20 participants were presented videos of an actor either performing intrinsic meaningful gestures in the context of German or Russian sentences, or speaking a German sentence without gesture, while performing a content judgment task. The connectivity analyses resulted in a winning model that included bidirectional intrinsic connectivity between all areas. Furthermore, the model included modulations of both connections to the STS (OC→STS; MTG→STS), and non-linear modulatory effects of the STS on bidirectional connections between MTG and OC. Coupling strength in the occipital pathway (OC→STS) correlated with gesture related advantages in task performance, whereas the temporal pathway (MTG→STS) correlated with performance in the speech only condition. Coupling between MTG and OC correlated negatively with subsequent memory performance for sentences of the Gesture-German condition. Our model provides a first step towards a better understanding of speech-gesture integration on network level. It corroborates the importance of the STS during audio-visual integration by showing that this region inhibits direct auditory-visual coupling.

Links between Gestures and Multisensory Processing: Individual Differences Suggest a Compensation Mechanism

Speech-associated gestures represent an important communication modality. However, individual differences in the production and perception of gestures are not well understood so far. We hypothesized that the perception of multisensory action consequences might play a crucial role. Verbal communication involves continuous calibration of audio–visual information produced by the speakers. The effective production and perception of gestures supporting this process could depend on the given capacities to perceive multisensory information accurately. We explored the association between the production and perception of gestures and the monitoring of multisensory action consequences in a sample of 31 participants. We applied a recently introduced gesture scale to assess self-reported gesture production and perception in everyday life situations. In the perceptual experiment, we presented unimodal (visual) and bimodal (visual and auditory) sensory outcomes with various delays after a self-initiated (active) or externally generated (passive) button press. Participants had to report whether they detected a delay between the button press and the visual stimulus. We derived psychometric functions for each condition and determined points of subjective equality, reflecting detection thresholds for delays. Results support a robust link between gesture scores and detection thresholds. Individuals with higher detection thresholds (lower performance) reported more frequent gesture production and perception and furthermore profited more from multisensory information in the experimental task. We propose that our findings indicate a compensational function of multisensory processing as a basis for individual differences in both action outcome monitoring and gesture production and perception in everyday life situations.

Gesturing more diminishes recall of abstract words when gesture is allowed and concrete words when it is taboo

Gesture during speech can promote or diminish recall for conversation content. We explored effects of cognitive load on this relationship, manipulating it at two scales: individual-word abstractness and social constraints to prohibit gestures. Prohibited gestures can diminish recall but more so for abstract-word recall. Insofar as movement planning adds to cognitive load, movement amplitude may moderate gesture effects on memory, with greater permitted- and prohibited-gesture movements reducing abstract-word recall and concrete-word recall, respectively. We tested these effects in a dyadic game in which 39 adult participants described words to confederates without naming the word or five related words. Results supported our expectations and indicated that memory effects of gesturing depend on social, cognitive, and motoric aspects of discourse.