The neural basis of hand gesture comprehension: A meta-analysis of functional magnetic resonance imaging studies

Making memories: The gestural misinformation effect in children aged 11-16-years-old with intellectual/developmental difficulties

BACKGROUND

In 2016, global records documented around 1 billion child abuse cases, with higher rates among children with Intellectual and Developmental Disabilities (IDD), and most recorded offenses not proceeding to court. Accurate eyewitness testimony is vital for the justice system. Yet, while children with IDD are known to be influenced by verbal misinformation, the effect of gestures on their testimony is still unknown.

AIMS

The present study assessed the extent to which gesture can mislead children with IDD, alongside comparisons to prior research in typically developing (TD) children.

METHOD

A sample of children with moderate IDD aged 11-16 years (n = 21, M=12.95 years) were recruited from a UK school, and compared to TD 5-6-year-olds (n = 31, M=5.77 years) and 7-8-year-olds (n = 32, M=7.66 years) from previous published research. After watching a video participants underwent an interview containing 12 questions, some of which contained suggestive gestures.

OUTCOMES AND IMPLICATIONS

Results demonstrated that in children with IDD, gesture observation significantly influenced responses given, with 18 of 21 children being misled at least once. Comparisons to TD children indicated no difference in suggestibility. This study is the first to examine how leading gestural information affects children with IDD, broadening previous research to a more representative sample for the justice system. Discussion centres on implications for police interview guidelines.

Language and gesture neural correlates: A meta-analysis of functional magnetic resonance imaging studies

BACKGROUND

Humans often use co-speech gestures to promote effective communication. Attention has been paid to the cortical areas engaged in the processing of co-speech gestures.

AIMS

To investigate the neural network underpinned in the processing of co-speech gestures and to observe whether there is a relationship between areas involved in language and gesture processing.

METHODS & PROCEDURES

We planned to include studies with neurotypical and/or stroke participants who underwent a bimodal task (i.e., processing of co-speech gestures with relative speech) and a unimodal task (i.e., speech or gesture alone) during a functional magnetic resonance imaging (fMRI) session. After a database search, abstract and full-text screening were conducted. Qualitative and quantitative data were extracted, and a meta-analysis was performed with the software GingerALE 3.0.2, performing contrast analyses of uni- and bimodal tasks.

MAIN CONTRIBUTION

The database search produced 1024 records. After the screening process, 27 studies were included in the review. Data from 15 studies were quantitatively analysed through meta-analysis. Meta-analysis found three clusters with a significant activation of the left middle frontal gyrus and inferior frontal gyrus, and bilateral middle occipital gyrus and inferior temporal gyrus.

CONCLUSIONS

There is a close link at the neural level for the semantic processing of auditory and visual information during communication. These findings encourage the integration of the use of co-speech gestures during aphasia treatment as a strategy to foster the possibility to communicate effectively for people with aphasia.

WHAT THIS PAPER ADDS

What is already known on this subject Gestures are an integral part of human communication, and they may have a relationship at neural level with speech processing. What this paper adds to the existing knowledge During processing of bi- and unimodal communication, areas related to semantic processing and multimodal processing are activated, suggesting that there is a close link between co-speech gestures and spoken language at a neural level. What are the potential or actual clinical implications of this work? Knowledge of the functions related to gesture and speech processing neural networks will allow for the adoption of model-based neurorehabilitation programs to foster recovery from aphasia by strengthening the specific functions of these brain networks.

The Neurocognitive Bases of Meaningful Intransitive Gestures: A Systematic Review and Meta-analysis of Neuropsychological Studies

Researchers and clinicians have long used meaningful intransitive (i.e., not tool-related; MFI) gestures to assess apraxia-a complex and frequent motor-cognitive disorder. Nevertheless, the neurocognitive bases of these gestures remain incompletely understood. Models of apraxia have assumed that meaningful intransitive gestures depend on either long-term memory (i.e., semantic memory and action lexicons) stored in the left hemisphere, or social cognition and the right hemisphere. This meta-analysis of 42 studies reports the performance of 2659 patients with either left or right hemisphere damage in tests of meaningful intransitive gestures, as compared to other gestures (i.e., MFT or meaningful transitive and MLI or meaningless intransitive) and cognitive tests. The key findings are as follows: (1) deficits of meaningful intransitive gestures are more frequent and severe after left than right hemisphere lesions, but they have been reported in both groups; (2) we found a transitivity effect in patients with lesions of the left hemisphere (i.e., meaningful transitive gestures more difficult than meaningful intransitive gestures) but a "reverse" transitivity effect in patients with lesions of the right hemisphere (i.e., meaningful transitive gestures easier than meaningful intransitive gestures); (3) there is a strong association between meaningful intransitive and transitive (but not meaningless) gestures; (4) isolated deficits of meaningful intransitive gestures are more frequent in cases with right than left hemisphere lesions; (5) these deficits may occur in the absence of language and semantic memory impairments; (6) meaningful intransitive gesture performance seems to vary according to the emotional content of gestures (i.e., body-centered gestures and emotional valence-intensity). These findings are partially consistent with the social cognition hypothesis. Methodological recommendations are given for future studies.

Gestures as Scaffolding to Learn Vocabulary in a Foreign Language

This paper investigates the influence of gestures on foreign language (FL) vocabulary learning. In this work, we first address the state of the art in the field and then delve into the research conducted in our lab (three experiments already published) in order to finally offer a unified theoretical interpretation of the role of gestures in FL vocabulary learning. In Experiments 1 and 2, we examined the impact of gestures on noun and verb learning. The results revealed that participants exhibited better learning outcomes when FL words were accompanied by congruent gestures compared to those from the no-gesture condition. Conversely, when meaningless or incongruent gestures were presented alongside new FL words, gestures had a detrimental effect on the learning process. Secondly, we addressed the question of whether or not individuals need to physically perform the gestures themselves to observe the effects of gestures on vocabulary learning (Experiment 3). Results indicated that congruent gestures improved FL word recall when learners only observed the instructor's gestures ("see" group) and when they mimicked them ("do" group). Importantly, the adverse effect associated with incongruent gestures was reduced in the "do" compared to that in the "see" experimental group. These findings suggest that iconic gestures can serve as an effective tool for learning vocabulary in an FL, particularly when the gestures align with the meaning of the words. Furthermore, the active performance of gestures helps counteract the negative effects associated with inconsistencies between gestures and word meanings. Consequently, if a choice must be made, an FL learning strategy in which learners acquire words while making gestures congruent with their meaning would be highly desirable.

Major Depression and the Perception of Affective Instrumental and Expressive Gestures: An fMRI Investigation

Major depressive disorder (MDD) is associated with biased perception of human movement. Gesture is important for communication and in this study we investigated neural correlates of gesture perception in MDD. We hypothesised different neural activity between individuals with MDD and typical individuals when viewing instrumental and expressive gestures that were negatively or positively valenced. Differences were expected in brain areas associated with gesture perception, including superior temporal, frontal, and emotion processing regions. We recruited 12 individuals with MDD and 12 typical controls matched on age, gender, and handedness. They viewed gestures displayed by stick figures while functional magnetic resonance imaging (fMRI) was performed. Results of a random effects three-way mixed ANOVA indicated that individuals with MDD had greater activity in the right claustrum compared to controls, regardless of gesture type or valence. Additionally, we observed main effects of gesture type and valence, regardless of group. Perceiving instrumental compared to expressive gestures was associated with greater activity in the left cuneus and left superior temporal gyrus, while perceiving negative compared to positive gestures was associated with greater activity in the right precuneus and right lingual gyrus. We also observed a two-way interaction between gesture type and valence in various brain regions.

The visuo-sensorimotor substrate of co-speech gesture processing

Co-speech gestures are integral to human communication and exhibit diverse forms, each serving a distinct communication function. However, existing literature has focused on individual gesture types, leaving a gap in understanding the comparative neural processing of these diverse forms. To address this, our study investigated the neural processing of two types of iconic gestures: those representing attributes or event knowledge of entity concepts, beat gestures enacting rhythmic manual movements without semantic information, and self-adaptors. During functional magnetic resonance imaging, systematic randomization and attentive observation of video stimuli revealed a general neural substrate for co-speech gesture processing primarily in the bilateral middle temporal and inferior parietal cortices, characterizing visuospatial attention, semantic integration of cross-modal information, and multisensory processing of manual and audiovisual inputs. Specific types of gestures and grooming movements elicited distinct neural responses. Greater activity in the right supramarginal and inferior frontal regions was specific to self-adaptors, and is relevant to the spatiomotor and integrative processing of speech and gestures. The semantic and sensorimotor regions were least active for beat gestures. The processing of attribute gestures was most pronounced in the left posterior middle temporal gyrus upon access to knowledge of entity concepts. This fMRI study illuminated the neural underpinnings of gesture-speech integration and highlighted the differential processing pathways for various co-speech gestures.

Hypoconnectivity networks in schizophrenia patients: A voxel-wise meta-analysis of Rs-fMRI

In recent years several meta-analyses regarding resting-state functional connectivity in patients with schizophrenia have been published. The authors have used different data analysis techniques: regional homogeneity, seed-based data analysis, independent component analysis, and amplitude of low frequencies. Hence, we aim to perform a meta-analysis to identify connectivity networks with different activation patterns between people diagnosed with schizophrenia and healthy controls using voxel-wise analysis.

METHOD

We collected primary studies exploring whole brain connectivity by functional magnetic resonance imaging at rest in patients with schizophrenia compared with healthy controls. We identified 25 studies included high-quality studies that included 1285 patients with schizophrenia and 1279 healthy controls.

RESULTS

The results indicate hypoactivation in the right precentral gyrus and the left superior temporal gyrus of patients with schizophrenia compared with healthy controls.

CONCLUSIONS

These regions have been linked with some clinical symptoms usually present in Plea with schizophrenia, such as auditory verbal hallucinations, formal thought disorder, and the comprehension and production of gestures.

Exploring the Emotional Functions of Co-Speech Hand Gesture in Language and Communication

Research over the past four decades has built a convincing case that co-speech hand gestures play a powerful role in human cognition . However, this recent focus on the cognitive function of gesture has, to a large extent, overlooked its emotional role-a role that was once central to research on bodily expression. In the present review, we first give a brief summary of the wealth of research demonstrating the cognitive function of co-speech gestures in language acquisition, learning, and thinking. Building on this foundation, we revisit the emotional function of gesture across a wide range of communicative contexts, from clinical to artistic to educational, and spanning diverse fields, from cognitive neuroscience to linguistics to affective science. Bridging the cognitive and emotional functions of gesture highlights promising avenues of research that have varied practical and theoretical implications for human-machine interactions, therapeutic interventions, language evolution, embodied cognition, and more.

No gesture too small: An investigation into the ability of gestural information to mislead eyewitness accounts by 5- to 8-year-olds

The accuracy of eyewitness interviews has legal and clinical implications within the criminal justice system. Leading verbal suggestions have been shown to give rise to false memories and inaccurate testimonies in children, but only a small body of research exists regarding non-verbal communication. The present study examined whether 5- to 8-year-olds in the UK could be misled about their memory of an event through exposure to leading gestural information, which suggested an incorrect response, using a variety of question and gesture types. Results showed that leading gestures significantly corrupted participants' memory compared to the control group (MD = 0.60, p < 0.001), with participants being misled by at least one question nearly three-quarters of the time. Questions about peripheral details, and gestures that were more visible and expressive, increased false memory further, with even subtle gestures demonstrating a strong misleading influence. We discuss the implications of these findings for the guidelines governing eyewitness interviews.

Using dynamic point light display stimuli to assess gesture deficits in schizophrenia

Background

Gesture deficits are ubiquitous in schizophrenia patients contributing to poor social communication and functional outcome. Given the dynamic nature of social communications, the current study aimed to explore the underlying socio-cognitive processes associated with point-light-displays (PLDs) of communicative gestures in the absence of any other confounding visual characteristics, and compare them to other well-established stimuli of gestures such as pictures by examining their association with symptom severity and motor-cognitive modalities.

Methods

We included 39-stable schizophrenia outpatients and 27-age-gender matched controls and assessed gesture processing using two tasks. The first task used static stimuli of pictures of a person performing a gesture. The limbs executing the gesture were missing and participants' task was to choose the correct gesture from three-options provided. The second task included videos of dynamic PLDs interacting with each other. One PLD performed communicative gestures, while the other PLD imitated/followed these performed gestures. Participants had to indicate, which of the two PLDs was imitating/following the other. Additionally, we evaluated symptom severity, as well as, motor and cognitive parameters.

Results

Patients underperformed in both gesture tasks compared to controls. Task performance for static stimuli was associated with blunted affect, motor coordination and sequencing domains, while PLD performance was associated with expressive gestures and sensory integration processes.

Discussion

Gesture representations of static and dynamic stimuli are associated with distinct processes contributing to poor social communication in schizophrenia, requiring novel therapeutic interventions. Such stimuli can easily be applied remotely for screening socio-cognitive deficits in schizophrenia.

Brain Dynamics of Action Monitoring in Higher-Order Motor Control Disorders: The Case of Apraxia

Limb apraxia (LA) refers to a high-order motor disorder characterized by the inability to reproduce transitive actions on commands or after observation. Studies demonstrate that action observation and action execution activate the same networks in the human brain, and provides an onlooker’s motor system with appropriate cognitive, motor and sensory-motor cues to flexibly implementing action-sequences and gestures. Tellingly, the temporal dynamics of action monitoring has never been explored in people suffering from LA. To fill this gap, we studied the electro-cortical signatures of error observation in human participants suffering from acquired left-brain lesions with (LA+) and without (LA–) LA, and in a group of healthy controls (H). EEG was acquired while participants observed from a first-person perspective (1PP) an avatar performing correct or incorrect reach-to-grasp a glass action in an immersive-virtual environment. Alterations of typical EEG signatures of error observation in time (early error positivity; Pe) and time-frequency domain (theta band-power) were found reduced in LA+ compared with H. Connectivity analyses showed that LA+ exhibited a decreased theta phase synchronization of both the frontoparietal and frontofrontal network, compared with H and LA–. Moreover, linear regression analysis revealed that the severity of LA [test of upper LA (TULIA) scores] was predicted by mid-frontal error-related theta activity, suggesting a link between error monitoring capacity and apraxic phenotypes. These results provide novel neurophysiological evidence of altered neurophysiological dynamics of action monitoring in individuals with LA and shed light on the performance monitoring changes occurring in this disorder.

Neural-level associations of non-verbal pragmatic comprehension in young Finnish autistic adults

This video-based study examines the pragmatic non-verbal comprehension skills and corresponding neural-level findings in young Finnish autistic adults, and controls. Items from the Assessment Battery of Communication (ABaCo) were chosen to evaluate the comprehension of non-verbal communication. Inter-subject correlation (ISC) analysis of the functional magnetic resonance imaging data was used to reveal the synchrony of brain activation across participants during the viewing of pragmatically complex scenes of ABaCo videos. The results showed a significant difference between the ISC maps of the autistic and control groups in tasks involving the comprehension of non-verbal communication, thereby revealing several brain regions where correlation of brain activity was greater within the control group. The results suggest a possible weaker modulation of brain states in response to the pragmatic non-verbal communicative situations in autistic participants. Although there was no difference between the groups in behavioural responses to ABaCo items, there was more variability in the accuracy of the responses in the autistic group. Furthermore, mean answering and reaction times correlated with the severity of autistic traits. The results indicate that even if young autistic adults may have learned to use compensatory resources in their communicative-pragmatic comprehension, pragmatic processing in naturalistic situations still requires additional effort.

Making Sense of the Hands and Mouth: The Role of "Secondary" Cues to Meaning in British Sign Language and English

Successful face-to-face communication involves multiple channels, notably hand gestures in addition to speech for spoken language, and mouth patterns in addition to manual signs for sign language. In four experiments, we assess the extent to which comprehenders of British Sign Language (BSL) and English rely, respectively, on cues from the hands and the mouth in accessing meaning. We created congruent and incongruent combinations of BSL manual signs and mouthings and English speech and gesture by video manipulation and asked participants to carry out a picture-matching task. When participants were instructed to pay attention only to the primary channel, incongruent "secondary" cues still affected performance, showing that these are reliably used for comprehension. When both cues were relevant, the languages diverged: Hand gestures continued to be used in English, but mouth movements did not in BSL. Moreover, non-fluent speakers and signers varied in the use of these cues: Gestures were found to be more important for non-native than native speakers; mouth movements were found to be less important for non-fluent signers. We discuss the results in terms of the information provided by different communicative channels, which combine to provide meaningful information.

Event related spectral perturbations of gesture congruity: Visuospatial resources are recruited for multimodal discourse comprehension

Here we examine the role of visuospatial working memory (WM) during the comprehension of multimodal discourse with co-speech iconic gestures. EEG was recorded as healthy adults encoded either a sequence of one (low load) or four (high load) dot locations on a grid and rehearsed them until a free recall response was collected later in the trial. During the rehearsal period of the WM task, participants observed videos of a speaker describing objects in which half of the trials included semantically related co-speech gestures (congruent), and the other half included semantically unrelated gestures (incongruent). Discourse processing was indexed by oscillatory EEG activity in the alpha and beta bands during the videos. Across all participants, effects of speech and gesture incongruity were more evident in low load trials than in high load trials. Effects were also modulated by individual differences in visuospatial WM capacity. These data suggest visuospatial WM resources are recruited in the comprehension of multimodal discourse.

Processing of pragmatic communication in ASD: a video-based brain imaging study

Social and pragmatic difficulties in autism spectrum disorder (ASD) are widely recognized, although their underlying neural level processing is not well understood. The aim of this study was to examine the activity of the brain network components linked to social and pragmatic understanding in order to reveal whether complex socio-pragmatic events evoke differences in brain activity between the ASD and control groups. Nineteen young adults (mean age 23.6 years) with ASD and 19 controls (mean age 22.7 years) were recruited for the study. The stimulus data consisted of video clips showing complex social events that demanded processing of pragmatic communication. In the analysis, the functional magnetic resonance imaging signal responses of the selected brain network components linked to social and pragmatic information processing were compared. Although the processing of the young adults with ASD was similar to that of the control group during the majority of the social scenes, differences between the groups were found in the activity of the social brain network components when the participants were observing situations with concurrent verbal and non-verbal communication events. The results suggest that the ASD group had challenges in processing concurrent multimodal cues in complex pragmatic communication situations.

Verbal working memory and co-speech gesture processing

Multimodal discourse requires an assembly of cognitive processes that are uniquely recruited for language comprehension in social contexts. In this study, we investigated the role of verbal working memory for the online integration of speech and iconic gestures. Participants memorized and rehearsed a series of auditorily presented digits in low (one digit) or high (four digits) memory load conditions. To observe how verbal working memory load impacts online discourse comprehension, ERPs were recorded while participants watched discourse videos containing either congruent or incongruent speech-gesture combinations during the maintenance portion of the memory task. While expected speech-gesture congruity effects were found in the low memory load condition, high memory load trials elicited enhanced frontal positivities that indicated a unique interaction between online speech-gesture integration and the availability of verbal working memory resources. This work contributes to an understanding of discourse comprehension by demonstrating that language processing in a multimodal context is subject to the relationship between cognitive resource availability and the degree of controlled processing required for task performance. We suggest that verbal working memory is less important for speech-gesture integration than it is for mediating speech processing under high task demands.

Dysbalanced Resting-State Functional Connectivity Within the Praxis Network Is Linked to Gesture Deficits in Schizophrenia

Patients with schizophrenia frequently present deficits in gesture production and interpretation, greatly affecting their communication skills. As these gesture deficits can be found early in the course of illness and as they can predict later outcomes, exploring their neural basis may lead to a better understanding of schizophrenia. While gesturing has been reported to rely on a left lateralized network of brain regions, termed praxis network, in healthy subjects and lesioned patients, studies in patients with schizophrenia are sparse. It is currently unclear whether within-network connectivity at rest is linked to gesture deficit. Here, we compared the functional connectivity between regions of the praxis network at rest between 46 patients and 44 healthy controls. All participants completed a validated test of hand gesture performance before resting-state functional magnetic resonance imaging (fMRI) was acquired. Patients performed gestures poorer than controls in all categories and domains. In patients, we also found significantly higher resting-state functional connectivity between left precentral gyrus and bilateral superior and inferior parietal lobule. Likewise, patients had higher connectivity from right precentral gyrus to left inferior and bilateral superior parietal lobule (SPL). In contrast, they exhibited lower connectivity between bilateral superior temporal gyrus (STG). Connectivity between right precentral gyrus and left SPL, as well as connectivity between bilateral STG, correlated with gesture performance in healthy controls. We failed to detect similar correlations in patients. We suggest that altered resting-state functional connectivity within the praxis network perturbs correct gesture planning in patients, reflecting the gesture deficit often seen in schizophrenia.

Functional magnetic resonance imaging (fMRI) item analysis of empathy and theory of mind

In contrast to conventional functional magnetic resonance imaging (fMRI) analysis across participants, item analysis allows generalizing the observed neural response patterns from a specific stimulus set to the entire population of stimuli. In the present study, we perform an item analysis on an fMRI paradigm (EmpaToM) that measures the neural correlates of empathy and Theory of Mind (ToM). The task includes a large stimulus set (240 emotional vs. neutral videos to probe empathic responding and 240 ToM or factual reasoning questions to probe ToM), which we tested in two large participant samples (N = 178, N = 130). Both, the empathy‐related network comprising anterior insula, anterior cingulate/dorsomedial prefrontal cortex, inferior frontal gyrus, and dorsal temporoparietal junction/supramarginal gyrus (TPJ) and the ToM related network including ventral TPJ, superior temporal gyrus, temporal poles, and anterior and posterior midline regions, were observed across participants and items. Regression analyses confirmed that these activations are predicted by the empathy or ToM condition of the stimuli, but not by low‐level features such as video length, number of words, syllables or syntactic complexity. The item analysis also allowed for the selection of the most effective items to create optimized stimulus sets that provide the most stable and reproducible results. Finally, reproducibility was shown in the replication of all analyses in the second participant sample. The data demonstrate (a) the generalizability of empathy and ToM related neural activity and (b) the reproducibility of the EmpaToM task and its applicability in intervention and clinical imaging studies.

A description of verbal and gestural communication during postictal aphasia

Patients suffering from drug-resistant temporal lobe epilepsy show substantial language deficits (i.e., anomia) during their seizures and in the postictal period (postictal aphasia). Verbal impairments observed during the postictal period may be studied to help localizing the epileptogenic zone. These explorations have been essentially based on simple tasks focused on speech, thus disregarding the multimodal nature of verbal communication, particularly the fact that, when speakers want to communicate, they often produce gestures of various kinds. Here, we propose an innovative procedure for testing postictal language and communication abilities, including the assessment of co-speech gestures. We provide a preliminary description of the changes induced on communication during postictal aphasia. We studied 21 seizures that induced postictal aphasia from 12 patients with drug-refractory epilepsy, including left temporal and left frontal seizures. The experimental task required patients to memorize a highly detailed picture and, briefly after, to describe what they had seen, thus eliciting a communicative meaningful monologue. This allowed comparing verbal communication in postictal and interictal conditions within the same individuals. Co-speech gestures were coded according to two categories: "Rhythmic" gestures, thought to be produced in support of speech building, and "illustrative" gestures, thought to be produced to complement the speech content. When postictal and interictal conditions were compared, there was decreased speech flow along with an increase of rhythmic gesture production at the expense of illustrative gesture production. The communication patterns did not differ significantly after temporal and frontal seizures, yet they were illustrated separately, owing to the clinical importance of the distinction, along with considerations of interindividual variability. A contrast between rhythmic and illustrative gestures production is congruent with previous literature in which rhythmic gestures have been linked to lexical retrieval processes. If confirmed in further studies, such evidence for a facilitative role of co-speech gestures in language difficulties could be put to use in the context of multimodal language therapies.

Gestures convey different physiological responses when performed toward and away from the body

We assessed the sympathetic and parasympathetic activation associated to the observation of Pantomime (i.e. the mime of the use of a tool) and Intransitive gestures (i.e. expressive) performed toward (e.g. a comb and “thinking”) and away from the body (e.g. key and “come here”) in a group of healthy participants while both pupil dilation (N = 31) and heart rate variability (N = 33; HF-HRV) were recorded. Large pupil dilation was observed in both Pantomime and Intransitive gestures toward the body; whereas an increase of the vagal suppression was observed in Intransitive gestures away from the body but not in those toward the body. Our results suggest that the space where people act when performing a gesture has an impact on the physiological responses of the observer in relation to the type of social communicative information that the gesture direction conveys, from a more intimate (toward the body) to a more interactive one (away from the body).

Speech-accompanying gestures are not processed by the language-processing mechanisms

Speech-accompanying gestures constitute one information channel during communication. Some have argued that processing gestures engages the brain regions that support language comprehension. However, studies that have been used as evidence for shared mechanisms suffer from one or more of the following limitations: they (a) have not directly compared activations for gesture and language processing in the same study and relied on the fallacious reverse inference (Poldrack, 2006) for interpretation, (b) relied on traditional group analyses, which are bound to overestimate overlap (e.g., Nieto-Castañon and Fedorenko, 2012), (c) failed to directly compare the magnitudes of response (e.g., Chen et al., 2017), and (d) focused on gestures that may have activated the corresponding linguistic representations (e.g., "emblems"). To circumvent these limitations, we used fMRI to examine responses to gesture processing in language regions defined functionally in individual participants (e.g., Fedorenko et al., 2010), including directly comparing effect sizes, and covering a broad range of spontaneously generated co-speech gestures. Whenever speech was present, language regions responded robustly (and to a similar degree regardless of whether the video contained gestures or grooming movements). In contrast, and critically, responses in the language regions were low - at or slightly above the fixation baseline - when silent videos were processed (again, regardless of whether they contained gestures or grooming movements). Brain regions outside of the language network, including some in close proximity to its regions, differentiated between gestures and grooming movements, ruling out the possibility that the gesture/grooming manipulation was too subtle. Behavioral studies on the critical video materials further showed robust differentiation between the gesture and grooming conditions. In summary, contra prior claims, language-processing regions do not respond to co-speech gestures in the absence of speech, suggesting that these regions are selectively driven by linguistic input (e.g., Fedorenko et al., 2011). Although co-speech gestures are uncontroversially important in communication, they appear to be processed in brain regions distinct from those that support language comprehension, similar to other extra-linguistic communicative signals, like facial expressions and prosody.

Does TMS Disruption of the Left Primary Motor Cortex Affect Verb Retrieval Following Exposure to Pantomimed Gestures?

Previous research suggests that meaning-laden gestures, even when produced in the absence of language (i.e., pantomimed gestures), influence lexical retrieval. Yet, little is known about the neural mechanisms that underlie this process. Based on embodied cognition theories, many studies have demonstrated motor cortex involvement in the representation of action verbs and in the understanding of actions. The present study aimed to investigate whether the motor system plays a critical role in the behavioral influence of pantomimed gestures on action naming. Continuous theta burst stimulation (cTBS) was applied over the hand area of the left primary motor cortex and to a control site (occipital cortex). An action-picture naming task followed cTBS. In the naming task, participants named action pictures that were preceded by videos of congruent pantomimed gestures, unrelated pantomimed gestures or a control video with no movement (as a neutral, non-gestural condition). In addition to behavioral measures of performance, cTBS-induced changes in corticospinal activity were assessed. We replicated previous finding that exposure to congruent pantomimed gestures facilitates word production, compared to unrelated or neutral primes. However, we found no evidence that the left primary motor area is crucially involved in the mechanism underlying behavioral facilitation effects of gesture on verb production. Although, at the group level, cTBS induced motor cortex suppression, at the individual level we found remarkable variability of cTBS effects on the motor cortex. We found cTBS induction of both inhibition of corticospinal activity (with slower behavioral of responses) and enhancement (with faster behavioral responses). Our findings cast doubt on assumptions that the motor cortex is causally involved in the impact of gestures on action-word processing. Our results also highlight the importance of careful consideration of interindividual variability for the interpretation of cTBS effects.

High-level language processing regions are not engaged in action observation or imitation

A set of left frontal, temporal, and parietal brain regions respond robustly during language comprehension and production (e.g., Fedorenko E, Hsieh PJ, Nieto-Castañón A, Whitfield-Gabrieli S, Kanwisher N. J Neurophysiol 104: 1177-1194, 2010; Menenti L, Gierhan SM, Segaert K, Hagoort P. Psychol Sci 22: 1173-1182, 2011). These regions have been further shown to be selective for language relative to other cognitive processes, including arithmetic, aspects of executive function, and music perception (e.g., Fedorenko E, Behr MK, Kanwisher N. Proc Natl Acad Sci USA 108: 16428-16433, 2011; Monti MM, Osherson DN. Brain Res 1428: 33-42, 2012). However, one claim about overlap between language and nonlinguistic cognition remains prominent. In particular, some have argued that language processing shares computational demands with action observation and/or execution (e.g., Rizzolatti G, Arbib MA. Trends Neurosci 21: 188-194, 1998; Koechlin E, Jubault T. Neuron 50: 963-974, 2006; Tettamanti M, Weniger D. Cortex 42: 491-494, 2006). However, the evidence for these claims is indirect, based on observing activation for language and action tasks within the same broad anatomical areas (e.g., on the lateral surface of the left frontal lobe). To test whether language indeed shares machinery with action observation/execution, we examined the responses of language brain regions, defined functionally in each individual participant (Fedorenko E, Hsieh PJ, Nieto-Castañón A, Whitfield-Gabrieli S, Kanwisher N. J Neurophysiol 104: 1177-1194, 2010) to action observation ( experiments 1, 2, and 3a) and action imitation ( experiment 3b). With the exception of the language region in the angular gyrus, all language regions, including those in the inferior frontal gyrus (within "Broca's area"), showed little or no response during action observation/imitation. These results add to the growing body of literature suggesting that high-level language regions are highly selective for language processing (see Fedorenko E, Varley R. Ann NY Acad Sci 1369: 132-153, 2016 for a review). NEW & NOTEWORTHY Many have argued for overlap in the machinery used to interpret language and others' actions, either because action observation was a precursor to linguistic communication or because both require interpreting hierarchically-structured stimuli. However, existing evidence is indirect, relying on group analyses or reverse inference. We examined responses to action observation in language regions defined functionally in individual participants and found no response. Thus language comprehension and action observation recruit distinct circuits in the modern brain.

Interpretation of Social Interactions: Functional Imaging of Cognitive-Semiotic Categories During Naturalistic Viewing

Social interactions arise from patterns of communicative signs, whose perception and interpretation require a multitude of cognitive functions. The semiotic framework of Peirce's Universal Categories (UCs) laid ground for a novel cognitive-semiotic typology of social interactions. During functional magnetic resonance imaging (fMRI), 16 volunteers watched a movie narrative encompassing verbal and non-verbal social interactions. Three types of non-verbal interactions were coded ("unresolved," "non-habitual," and "habitual") based on a typology reflecting Peirce's UCs. As expected, the auditory cortex responded to verbal interactions, but non-verbal interactions modulated temporal areas as well. Conceivably, when speech was lacking, ambiguous visual information (unresolved interactions) primed auditory processing in contrast to learned behavioral patterns (habitual interactions). The latter recruited a parahippocampal-occipital network supporting conceptual processing and associative memory retrieval. Requesting semiotic contextualization, non-habitual interactions activated visuo-spatial and contextual rule-learning areas such as the temporo-parietal junction and right lateral prefrontal cortex. In summary, the cognitive-semiotic typology reflected distinct sensory and association networks underlying the interpretation of observed non-verbal social interactions.

Transcranial Magnetic Stimulation over Left Inferior Frontal and Posterior Temporal Cortex Disrupts Gesture-Speech Integration

Language and action naturally occur together in the form of cospeech gestures, and there is now convincing evidence that listeners display a strong tendency to integrate semantic information from both domains during comprehension. A contentious question, however, has been which brain areas are causally involved in this integration process. In previous neuroimaging studies, left inferior frontal gyrus (IFG) and posterior middle temporal gyrus (pMTG) have emerged as candidate areas; however, it is currently not clear whether these areas are causally or merely epiphenomenally involved in gesture-speech integration. In the present series of experiments, we directly tested for a potential critical role of IFG and pMTG by observing the effect of disrupting activity in these areas using transcranial magnetic stimulation in a mixed gender sample of healthy human volunteers. The outcome measure was performance on a Stroop-like gesture task (Kelly et al., 2010a), which provides a behavioral index of gesture-speech integration. Our results provide clear evidence that disrupting activity in IFG and pMTG selectively impairs gesture-speech integration, suggesting that both areas are causally involved in the process. These findings are consistent with the idea that these areas play a joint role in gesture-speech integration, with IFG regulating strategic semantic access via top-down signals acting upon temporal storage areas.SIGNIFICANCE STATEMENT Previous neuroimaging studies suggest an involvement of inferior frontal gyrus and posterior middle temporal gyrus in gesture-speech integration, but findings have been mixed and due to methodological constraints did not allow inferences of causality. By adopting a virtual lesion approach involving transcranial magnetic stimulation, the present study provides clear evidence that both areas are causally involved in combining semantic information arising from gesture and speech. These findings support the view that, rather than being separate entities, gesture and speech are part of an integrated multimodal language system, with inferior frontal gyrus and posterior middle temporal gyrus serving as critical nodes of the cortical network underpinning this system.

Perceived Conventionality in Co-speech Gestures Involves the Fronto-Temporal Language Network

Face-to-face communication is multimodal; it encompasses spoken words, facial expressions, gaze, and co-speech gestures. In contrast to linguistic symbols (e.g., spoken words or signs in sign language) relying on mostly explicit conventions, gestures vary in their degree of conventionality. Bodily signs may have a general accepted or conventionalized meaning (e.g., a head shake) or less so (e.g., self-grooming). We hypothesized that subjective perception of conventionality in co-speech gestures relies on the classical language network, i.e., the left hemispheric inferior frontal gyrus (IFG, Broca's area) and the posterior superior temporal gyrus (pSTG, Wernicke's area) and studied 36 subjects watching video-recorded story retellings during a behavioral and an functional magnetic resonance imaging (fMRI) experiment. It is well documented that neural correlates of such naturalistic videos emerge as intersubject covariance (ISC) in fMRI even without involving a stimulus (model-free analysis). The subjects attended either to perceived conventionality or to a control condition (any hand movements or gesture-speech relations). Such tasks modulate ISC in contributing neural structures and thus we studied ISC changes to task demands in language networks. Indeed, the conventionality task significantly increased covariance of the button press time series and neuronal synchronization in the left IFG over the comparison with other tasks. In the left IFG, synchronous activity was observed during the conventionality task only. In contrast, the left pSTG exhibited correlated activation patterns during all conditions with an increase in the conventionality task at the trend level only. Conceivably, the left IFG can be considered a core region for the processing of perceived conventionality in co-speech gestures similar to spoken language. In general, the interpretation of conventionalized signs may rely on neural mechanisms that engage during language comprehension.

The cortical signature of impaired gesturing: Findings from schizophrenia

Schizophrenia is characterized by deficits in gesturing that is important for nonverbal communication. Research in healthy participants and brain-damaged patients revealed a left-lateralized fronto-parieto-temporal network underlying gesture performance. First evidence from structural imaging studies in schizophrenia corroborates these results. However, as of yet, it is unclear if cortical thickness abnormalities contribute to impairments in gesture performance. We hypothesized that patients with deficits in gesture production show cortical thinning in 12 regions of interest (ROIs) of a gesture network relevant for gesture performance and recognition. Forty patients with schizophrenia and 41 healthy controls performed hand and finger gestures as either imitation or pantomime. Group differences in cortical thickness between patients with deficits, patients without deficits, and controls were explored using a multivariate analysis of covariance. In addition, the relationship between gesture recognition and cortical thickness was investigated. Patients with deficits in gesture production had reduced cortical thickness in eight ROIs, including the pars opercularis of the inferior frontal gyrus, the superior and inferior parietal lobes, and the superior and middle temporal gyri. Gesture recognition correlated with cortical thickness in fewer, but mainly the same, ROIs within the patient sample. In conclusion, our results show that impaired gesture production and recognition in schizophrenia is associated with cortical thinning in distinct areas of the gesture network.

•

Impairments in gesture production and recognition in schizophrenia are related to altered brain structure.

•

Brain alterations in schizophrenia are located in areas that are generally damaged in apraxia.

•

Schizophrenia patients with gesture deficits show cortical thinning of several regions in the gesture network.

•

Deficits of gesture production and recognition are both related to a fronto-parieto-temporal gesture network.

Mapping effective connectivity within cortical networks with diffuse optical tomography

Understanding how cortical networks interact in response to task demands is important both for providing insight into the brain's processing architecture and for managing neurological diseases and mental disorders. High-density diffuse optical tomography (HD-DOT) is a neuroimaging technique that offers the significant advantages of having a naturalistic, acoustically controllable environment and being compatible with metal implants, neither of which is possible with functional magnetic resonance imaging. We used HD-DOT to study the effective connectivity and assess the modulatory effects of speech intelligibility and syntactic complexity on functional connections within the cortical speech network. To accomplish this, we extend the use of a generalized psychophysiological interaction (PPI) analysis framework. In particular, we apply PPI methods to event-related HD-DOT recordings of cortical oxyhemoglobin activity during auditory sentence processing. We evaluate multiple approaches for selecting cortical regions of interest and for modeling interactions among these regions. Our results show that using subject-based regions has minimal effect on group-level connectivity maps. We also demonstrate that incorporating an interaction model based on estimated neural activity results in significantly stronger effective connectivity. Taken together our findings support the use of HD-DOT with PPI methods for noninvasively studying task-related modulations of functional connectivity.

Multimodal integration of spontaneously produced representational co-speech gestures: an fMRI study

To examine whether more ecologically valid co-speech gesture stimuli elicit brain responses consistent with those found by studies that relied on scripted stimuli, we presented participants with spontaneously produced, meaningful co-speech gesture during fMRI scanning (n = 28). Speech presented with gesture (versus either presented alone) elicited heightened activity in bilateral posterior superior temporal, premotor, and inferior frontal regions. Within left temporal and premotor, but not inferior frontal regions, we identified small clusters with superadditive responses, suggesting that these discrete regions support both sensory and semantic integration. In contrast, surrounding areas and the inferior frontal gyrus may support either sensory or semantic integration. Reduced activation for speech with gesture in language-related regions indicates allocation of fewer neural resources when meaningful gestures accompany speech. Sign language experience did not affect co-speech gesture activation. Overall, our results indicate that scripted stimuli have minimal confounding influences; however, they may miss subtle superadditive effects.

Sources of mu activity and their functional connectivity in perceiving complexities in reciprocal social interactive motion: An exploratory study using the 'Namaste' task

Cognitive processes underlying reciprocal social interactions are understood by the mechanism of embodiment, which is closely related to the mirror neuron system. Electroencephalographic (EEG) mu activity is a neural marker of the mirror neuron system. This study investigated the mu activity, localization of its sources and functional connectivity, which was induced while watching reciprocal social interactive motion across various degrees of complexity. Eighteen healthy participants underwent high-resolution EEG recording using 256-channels while they watched a specifically designed, culture specific, video task that showed two persons interacting socially using body gestures. Task complexity was determined by (1) whether there was an identical gestural response or a non-identical one; (2) whether the participant watched two persons interacting or was virtually involved in the interaction. Source localization and functional connectivity analysis was conducted for mu activity across various tasks. We also correlated mu activity and functional connectivity measures with serum BDNF. We found that spectral densities in various brain sources of mu activity and their increased functional connectivity distinguished identical and non-identical reciprocal expression observations, while mu suppression alone did not discriminate various degrees of complexities. These findings might have important implications in the understanding of mechanisms underlying mirror neuron dysfunction in various psychiatric disorders.

A touch with words: Dynamic synergies between manual actions and language

Manual actions are a hallmark of humanness. Their underlying neural circuitry gives rise to species-specific skills and interacts with language processes. In particular, multiple studies show that hand-related expressions - verbal units evoking manual activity - variously affect concurrent manual actions, yielding apparently controversial results (interference, facilitation, or null effects) in varied time windows. Through a systematic review of 108 experiments, we show that such effects are driven by several factors, such as the level of verbal processing, action complexity, and the time-lag between linguistic and motor processes. We reconcile key empirical patterns by introducing the Hand-Action-Network Dynamic Language Embodiment (HANDLE) model, an integrative framework based on neural coupling dynamics and predictive-coding principles. To conclude, we assess HANDLE against the backdrop of other action-cognition theories, illustrate its potential applications to understand high-level deficits in motor disorders, and discuss key challenges for further development. In sum, our work aligns with the 'pragmatic turn', moving away from passive and static representationalist perspectives to a more dynamic, enactive, and embodied conceptualization of cognitive processes.

Iconic Gestures for Robot Avatars, Recognition and Integration with Speech

Co-verbal gestures are an important part of human communication, improving its efficiency and efficacy for information conveyance. One possible means by which such multi-modal communication might be realized remotely is through the use of a tele-operated humanoid robot avatar. Such avatars have been previously shown to enhance social presence and operator salience. We present a motion tracking based tele-operation system for the NAO robot platform that allows direct transmission of speech and gestures produced by the operator. To assess the capabilities of this system for transmitting multi-modal communication, we have conducted a user study that investigated if robot-produced iconic gestures are comprehensible, and are integrated with speech. Robot performed gesture outcomes were compared directly to those for gestures produced by a human actor, using a within participant experimental design. We show that iconic gestures produced by a tele-operated robot are understood by participants when presented alone, almost as well as when produced by a human. More importantly, we show that gestures are integrated with speech when presented as part of a multi-modal communication equally well for human and robot performances.