Neural integration of iconic and unrelated coverbal gestures: a functional MRI study

Feeling addressed! The role of body orientation and co-speech gesture in social communication

During face-to-face communication, body orientation and coverbal gestures influence how information is conveyed. The neural pathways underpinning the comprehension of such nonverbal social cues in everyday interaction are to some part still unknown. During fMRI data acquisition, 37 participants were presented with video clips showing an actor speaking short sentences. The actor produced speech-associated iconic gestures (IC) or no gestures (NG) while he was visible either from an egocentric (ego) or from an allocentric (allo) position. Participants were asked to indicate via button press whether they felt addressed or not. We found a significant interaction of body orientation and gesture in addressment evaluations, indicating that participants evaluated IC-ego conditions as most addressing. The anterior cingulate cortex (ACC) and left fusiform gyrus were stronger activated for egocentric versus allocentric actor position in gesture context. Activation increase in the ACC for IC-ego>IC-allo further correlated positively with increased addressment ratings in the egocentric gesture condition. Gesture-related activation increase in the supplementary motor area, left inferior frontal gyrus and right insula correlated positively with gesture-related increase of addressment evaluations in the egocentric context. Results indicate that gesture use and body-orientation contribute to the feeling of being addressed and together influence neural processing in brain regions involved in motor simulation, empathy and mentalizing.

Hearing and seeing meaning in speech and gesture: insights from brain and behaviour

As we speak, we use not only the arbitrary form-meaning mappings of the speech channel but also motivated form-meaning correspondences, i.e. iconic gestures that accompany speech (e.g. inverted V-shaped hand wiggling across gesture space to demonstrate walking). This article reviews what we know about processing of semantic information from speech and iconic gestures in spoken languages during comprehension of such composite utterances. Several studies have shown that comprehension of iconic gestures involves brain activations known to be involved in semantic processing of speech: i.e. modulation of the electrophysiological recording component N400, which is sensitive to the ease of semantic integration of a word to previous context, and recruitment of the left-lateralized frontal-posterior temporal network (left inferior frontal gyrus (IFG), medial temporal gyrus (MTG) and superior temporal gyrus/sulcus (STG/S)). Furthermore, we integrate the information coming from both channels recruiting brain areas such as left IFG, posterior superior temporal sulcus (STS)/MTG and even motor cortex. Finally, this integration is flexible: the temporal synchrony between the iconic gesture and the speech segment, as well as the perceived communicative intent of the speaker, modulate the integration process. Whether these findings are special to gestures or are shared with actions or other visual accompaniments to speech (e.g. lips) or other visual symbols such as pictures are discussed, as well as the implications for a multimodal view of language.

Keeping an eye on the conductor: neural correlates of visuo-motor synchronization and musical experience

For orchestra musicians, synchronized playing under a conductor’s direction is necessary to achieve optimal performance. Previous studies using simple auditory/visual stimuli have reported cortico-subcortical networks underlying synchronization and that training improves the accuracy of synchronization. However, it is unclear whether people who played regularly under a conductor and non-musicians activate the same networks when synchronizing with a conductor’s gestures. We conducted a functional magnetic resonance imaging (fMRI) experiment testing nonmusicians and musicians who regularly play music under a conductor. Participants were required to tap the rhythm they perceived from silent movies displaying either conductor’s gestures or a swinging metronome. Musicians performed tapping under a conductor with more precision than nonmusicians. Results from fMRI measurement showed greater activity in the anterior part of the left superior frontal gyrus (SFG) in musicians with more frequent practice under a conductor. Conversely, tapping with the metronome did not show any difference between musicians and nonmusicians, indicating that the expertize effect in tapping under the conductor does not result in a general increase in tapping performance for musicians. These results suggest that orchestra musicians have developed an advanced ability to predict conductor’s next action from the gestures.

Superior temporal sulcus disconnectivity during processing of metaphoric gestures in schizophrenia

The left superior temporal sulcus (STS) plays an important role in integrating audiovisual information and is functionally connected to disparate regions of the brain. For the integration of gesture information in an abstract sentence context (metaphoric gestures), intact connectivity between the left STS and the inferior frontal gyrus (IFG) should be important. Patients with schizophrenia have problems with the processing of metaphors (concretism) and show aberrant structural connectivity of long fiber bundles. Thus, we tested the hypothesis that patients with schizophrenia differ in the functional connectivity of the left STS to the IFG for the processing of metaphoric gestures. During functional magnetic resonance imaging data acquisition, 16 patients with schizophrenia (P) and a healthy control group (C) were shown videos of an actor performing gestures in a concrete (iconic, IC) and abstract (metaphoric, MP) sentence context. A psychophysiological interaction analysis based on the seed region from a previous analysis in the left STS was performed. In both groups we found common positive connectivity for IC and MP of the STS seed region to the left middle temporal gyrus (MTG) and left ventral IFG. The interaction of group (C>P) and gesture condition (MP>IC) revealed effects in the connectivity to the bilateral IFG and the left MTG with patients exhibiting lower connectivity for the MP condition. In schizophrenia the left STS is misconnected to the IFG, particularly during the processing of MP gestures. Dysfunctional integration of gestures in an abstract sentence context might be the basis of certain interpersonal communication problems in the patients.

Semantic relation vs. surprise: the differential effects of related and unrelated co-verbal gestures on neural encoding and subsequent recognition

Speech-associated gesturing leads to memory advantages for spoken sentences. However, unexpected or surprising events are also likely to be remembered. With this study we test the hypothesis that different neural mechanisms (semantic elaboration and surprise) lead to memory advantages for iconic and unrelated gestures. During fMRI-data acquisition participants were presented with video clips of an actor verbalising concrete sentences accompanied by iconic gestures (IG; e.g., circular gesture; sentence: "The man is sitting at the round table"), unrelated free gestures (FG; e.g., unrelated up down movements; same sentence) and no gestures (NG; same sentence). After scanning, recognition performance for the three conditions was tested. Videos were evaluated regarding semantic relation and surprise by a different group of participants. The semantic relationship between speech and gesture was rated higher for IG (IG>FG), whereas surprise was rated higher for FG (FG>IG). Activation of the hippocampus correlated with subsequent memory performance of both gesture conditions (IG+FG>NG). For the IG condition we found activation in the left temporal pole and middle cingulate cortex (MCC; IG>FG). In contrast, for the FG condition posterior thalamic structures (FG>IG) as well as anterior and posterior cingulate cortices were activated (FG>NG). Our behavioral and fMRI-data suggest different mechanisms for processing related and unrelated co-verbal gestures, both of them leading to enhanced memory performance. Whereas activation in MCC and left temporal pole for iconic co-verbal gestures may reflect semantic memory processes, memory enhancement for unrelated gestures relies on the surprise response, mediated by anterior/posterior cingulate cortex and thalamico-hippocampal structures.

Eye'm talking to you: speakers' gaze direction modulates co-speech gesture processing in the right MTG

Recipients process information from speech and co-speech gestures, but it is currently unknown how this processing is influenced by the presence of other important social cues, especially gaze direction, a marker of communicative intent. Such cues may modulate neural activity in regions associated either with the processing of ostensive cues, such as eye gaze, or with the processing of semantic information, provided by speech and gesture. Participants were scanned (fMRI) while taking part in triadic communication involving two recipients and a speaker. The speaker uttered sentences that were and were not accompanied by complementary iconic gestures. Crucially, the speaker alternated her gaze direction, thus creating two recipient roles: addressed (direct gaze) vs unaddressed (averted gaze) recipient. The comprehension of Speech&Gesture relative to SpeechOnly utterances recruited middle occipital, middle temporal and inferior frontal gyri, bilaterally. The calcarine sulcus and posterior cingulate cortex were sensitive to differences between direct and averted gaze. Most importantly, Speech&Gesture utterances, but not SpeechOnly utterances, produced additional activity in the right middle temporal gyrus when participants were addressed. Marking communicative intent with gaze direction modulates the processing of speech-gesture utterances in cerebral areas typically associated with the semantic processing of multi-modal communicative acts.

Externalizing the private experience of pain: a role for co-speech gestures in pain communication?

Despite the importance of effective pain communication, talking about pain represents a major challenge for patients and clinicians because pain is a private and subjective experience. Focusing primarily on acute pain, this article considers the limitations of current methods of obtaining information about the sensory characteristics of pain and suggests that spontaneously produced "co-speech hand gestures" may constitute an important source of information here. Although this is a relatively new area of research, we present recent empirical evidence that reveals that co-speech gestures contain important information about pain that can both add to and clarify speech. Following this, we discuss how these findings might eventually lead to a greater understanding of the sensory characteristics of pain, and to improvements in treatment and support for pain sufferers. We hope that this article will stimulate further research and discussion of this previously overlooked dimension of pain communication.

Alcohol affects neuronal substrates of response inhibition but not of perceptual processing of stimuli signalling a stop response

Alcohol impairs inhibitory control, including the ability to terminate an initiated action. While there is increasing knowledge about neural mechanisms involved in response inhibition, the level at which alcohol impairs such mechanisms remains poorly understood. Thirty-nine healthy social drinkers received either 0.4g/kg or 0.8g/kg of alcohol, or placebo, and performed two variants of a Visual Stop-signal task during acquisition of functional magnetic resonance imaging (fMRI) data. The two task variants differed only in their instructions: in the classic variant (VSST), participants inhibited their response to a “Go-stimulus” when it was followed by a “Stop-stimulus”. In the control variant (VSST_C), participants responded to the “Go-stimulus” even if it was followed by a “Stop-stimulus”. Comparison of successful Stop-trials (Sstop)>Go, and unsuccessful Stop-trials (Ustop)>Sstop between the three beverage groups enabled the identification of alcohol effects on functional neural circuits supporting inhibitory behaviour and error processing. Alcohol impaired inhibitory control as measured by the Stop-signal reaction time, but did not affect other aspects of VSST performance, nor performance on the VSST_C. The low alcohol dose evoked changes in neural activity within prefrontal, temporal, occipital and motor cortices. The high alcohol dose evoked changes in activity in areas affected by the low dose but importantly induced changes in activity within subcortical centres including the globus pallidus and thalamus. Alcohol did not affect neural correlates of perceptual processing of infrequent cues, as revealed by conjunction analyses of VSST and VSST_C tasks. Alcohol ingestion compromises the inhibitory control of action by modulating cortical regions supporting attentional, sensorimotor and action-planning processes. At higher doses the impact of alcohol also extends to affect subcortical nodes of fronto-basal ganglia- thalamo-cortical motor circuits. In contrast, alcohol appears to have little impact on the early visual processing of infrequent perceptual cues. These observations clarify clinically-important effects of alcohol on behaviour.

Supramodal neural processing of abstract information conveyed by speech and gesture

Abstractness and modality of interpersonal communication have a considerable impact on comprehension. They are relevant for determining thoughts and constituting internal models of the environment. Whereas concrete object-related information can be represented in mind irrespective of language, abstract concepts require a representation in speech. Consequently, modality-independent processing of abstract information can be expected. Here we investigated the neural correlates of abstractness (abstract vs. concrete) and modality (speech vs. gestures), to identify an abstractness-specific supramodal neural network. During fMRI data acquisition 20 participants were presented with videos of an actor either speaking sentences with an abstract-social [AS] or concrete-object-related content [CS], or performing meaningful abstract-social emblematic [AG] or concrete-object-related tool-use gestures [CG]. Gestures were accompanied by a foreign language to increase the comparability between conditions and to frame the communication context of the gesture videos. Participants performed a content judgment task referring to the person vs. object-relatedness of the utterances. The behavioral data suggest a comparable comprehension of contents communicated by speech or gesture. Furthermore, we found common neural processing for abstract information independent of modality (AS > CS ∩ AG > CG) in a left hemispheric network including the left inferior frontal gyrus (IFG), temporal pole, and medial frontal cortex. Modality specific activations were found in bilateral occipital, parietal, and temporal as well as right inferior frontal brain regions for gesture (G > S) and in left anterior temporal regions and the left angular gyrus for the processing of speech semantics (S > G). These data support the idea that abstract concepts are represented in a supramodal manner. Consequently, gestures referring to abstract concepts are processed in a predominantly left hemispheric language related neural network.

The neurobiology of receptive-expressive language interdependence

With a focus on receptive language, we examine the neurobiological evidence for the interdependence of receptive and expressive language processes. While we agree that there is compelling evidence for such interdependence, we suggest that Pickering & Garrod's (P&G's) account would be enhanced by considering more-specific situations in which their model does, and does not, apply.

Neural integration of speech and gesture in schizophrenia: evidence for differential processing of metaphoric gestures

Gestures are an important component of interpersonal communication. Especially, complex multimodal communication is assumed to be disrupted in patients with schizophrenia. In healthy subjects, differential neural integration processes for gestures in the context of concrete [iconic (IC) gestures] and abstract sentence contents [metaphoric (MP) gestures] had been demonstrated. With this study we wanted to investigate neural integration processes for both gesture types in patients with schizophrenia. During functional magnetic resonance imaging-data acquisition, 16 patients with schizophrenia (P) and a healthy control group (C) were shown videos of an actor performing IC and MP gestures and associated sentences. An isolated gesture (G) and isolated sentence condition (S) were included to separate unimodal from bimodal effects at the neural level. During IC conditions (IC > G ∩ IC > S) we found increased activity in the left posterior middle temporal gyrus (pMTG) in both groups. Whereas in the control group the left pMTG and the inferior frontal gyrus (IFG) were activated for the MP conditions (MP > G ∩ MP > S), no significant activation was found for the identical contrast in patients. The interaction of group (P/C) and gesture condition (MP/IC) revealed activation in the bilateral hippocampus, the left middle/superior temporal and IFG. Activation of the pMTG for the IC condition in both groups indicates intact neural integration of IC gestures in schizophrenia. However, failure to activate the left pMTG and IFG for MP co-verbal gestures suggests a disturbed integration of gestures embedded in an abstract sentence context. This study provides new insight into the neural integration of co-verbal gestures in patients with schizophrenia.

Acute alcohol effects on attentional bias are mediated by subcortical areas associated with arousal and salience attribution

Acute alcohol ingestion increases attentional bias to alcohol-related stimuli; however, the underlying cognitive and brain mechanisms remain unknown. We combined functional magnetic resonance imaging (fMRI) with performance of a dual task that probed attentional distraction by alcohol-related stimuli during 'conflict' processing: the Concurrent Flanker/Alcohol-Attentional bias task (CFAAT). In this task, an Eriksen Flanker task is superimposed on task-unrelated background pictures with alcohol-associated or neutral content. Participants respond to the direction of a central 'target' arrow and ignore adjacent congruent (low cognitive load) or incongruent (high cognitive load) 'flanking' arrows. Using a between-subject design, 40 healthy moderate-to-heavy social drinkers received either no alcohol (placebo), 0.4 g/kg (low dose), or 0.8 g/kg (high dose) of alcohol, and underwent fMRI while performing the CFAAT. The low alcohol dose, relative to placebo, increased response latencies on trials with alcohol-associated backgrounds and, under low cognitive load, increased the activity evoked by these pictures within a medial hypothalamic region. Under high cognitive load, the low alcohol dose, relative to placebo, elicited greater activity within a more lateral hypothalamic region, and reduced activity within frontal motor areas. The high alcohol dose, relative to placebo, did not reliably affect response latencies or neural responses to background images, but reduced overall accuracy under high cognitive load. This effect correlated with changes in reactivity within medial and dorsal prefrontal cortices. These data suggest that alcohol at a low dose primes attentional bias to alcohol-associated stimuli, an effect mediated by activation of subcortical hypothalamic areas implicated in arousal and salience attribution.

Differential role of the Mentalizing and the Mirror Neuron system in the imitation of communicative gestures

Successful social interaction requires recognising the intention of another person's communicative gestures. At a neural level, this process may involve neural activity in different systems, such as the mentalizing system (MS) and the mirror neuron system (MNS). The aim of the present study was to explore the neural correlates of communicative gestures during observation and execution of these gestures. Twenty participants watched video clips of an actor executing social gestures (S), non-social gestures (NS) and meaningless gestures (ML). During fMRI data acquisition, participants were asked to observe (O) and subsequently to execute (E) one of two tasks: imitate the gesture presented (IMI) or perform a motor control task (CT). For the contrast IMI>CT we found activations in the core areas of the MNS [inferior parietal lobule (IPL) and inferior frontal cortex, the posterior part of pars opercularis], as well as in areas related to the MS [superior temporal sulcus (STS) and middle cingulate cortex]. For S>NS, we found activations in the left medial orbitofrontal cortex (mOFC), right superior frontal cortex and middle cingulate cortex. The interaction of stimulus condition (S vs NS) and task (IMI vs CT) revealed activation in the right IPL. For the interaction between observation vs execution (O vs E), task (IMI vs CT) and stimulus condition (S vs NS) we found activation in the right mOFC. Our data suggest that imitation is differentially processed in the MNS as well as in the MS. The activation in IPL is enhanced during the processing of social gestures most likely due to their communicative intention. The activation of IPL together with medial frontal areas may contribute to mentalizing processes. The interaction in the mOFC suggests an involvement of self-referential processes in the processing of social gesture.

Neural correlates of gesture processing across human development

Co-speech gesture facilitates learning to a greater degree in children than in adults, suggesting that the mechanisms underlying the processing of co-speech gesture differ as a function of development. We suggest that this may be partially due to children's lack of experience producing gesture, leading to differences in the recruitment of sensorimotor networks when comparing adults to children. Here, we investigated the neural substrates of gesture processing in a cross-sectional sample of 5-, 7.5-, and 10-year-old children and adults and focused on relative recruitment of a sensorimotor system that included the precentral gyrus (PCG) and the posterior middle temporal gyrus (pMTG). Children and adults were presented with videos in which communication occurred through different combinations of speech and gesture during a functional magnetic resonance imaging (fMRI) session. Results demonstrated that the PCG and pMTG were recruited to different extents in the two populations. We interpret these novel findings as supporting the idea that gesture perception (pMTG) is affected by a history of gesture production (PCG), revealing the importance of considering gesture processing as a sensorimotor process.

Brain function overlaps when people observe emblems, speech, and grasping

A hand grasping a cup or gesturing "thumbs-up", while both manual actions, have different purposes and effects. Grasping directly affects the cup, whereas gesturing "thumbs-up" has an effect through an implied verbal (symbolic) meaning. Because grasping and emblematic gestures ("emblems") are both goal-oriented hand actions, we pursued the hypothesis that observing each should evoke similar activity in neural regions implicated in processing goal-oriented hand actions. However, because emblems express symbolic meaning, observing them should also evoke activity in regions implicated in interpreting meaning, which is most commonly expressed in language. Using fMRI to test this hypothesis, we had participants watch videos of an actor performing emblems, speaking utterances matched in meaning to the emblems, and grasping objects. Our results show that lateral temporal and inferior frontal regions respond to symbolic meaning, even when it is expressed by a single hand action. In particular, we found that left inferior frontal and right lateral temporal regions are strongly engaged when people observe either emblems or speech. In contrast, we also replicate and extend previous work that implicates parietal and premotor responses in observing goal-oriented hand actions. For hand actions, we found that bilateral parietal and premotor regions are strongly engaged when people observe either emblems or grasping. These findings thus characterize converging brain responses to shared features (e.g., symbolic or manual), despite their encoding and presentation in different stimulus modalities.

Frontal and temporal contributions to understanding the iconic co-speech gestures that accompany speech

In everyday conversation, listeners often rely on a speaker's gestures to clarify any ambiguities in the verbal message. Using fMRI during naturalistic story comprehension, we examined which brain regions in the listener are sensitive to speakers' iconic gestures. We focused on iconic gestures that contribute information not found in the speaker's talk, compared with those that convey information redundant with the speaker's talk. We found that three regions-left inferior frontal gyrus triangular (IFGTr) and opercular (IFGOp) portions, and left posterior middle temporal gyrus (MTGp)--responded more strongly when gestures added information to nonspecific language, compared with when they conveyed the same information in more specific language; in other words, when gesture disambiguated speech as opposed to reinforced it. An increased BOLD response was not found in these regions when the nonspecific language was produced without gesture, suggesting that IFGTr, IFGOp, and MTGp are involved in integrating semantic information across gesture and speech. In addition, we found that activity in the posterior superior temporal sulcus (STSp), previously thought to be involved in gesture-speech integration, was not sensitive to the gesture-speech relation. Together, these findings clarify the neurobiology of gesture-speech integration and contribute to an emerging picture of how listeners glean meaning from gestures that accompany speech.

A possible brain network for representation of cooperative behavior and its implications for the psychopathology of schizophrenia

It has been shown that social deficits contribute to psychopathology in schizophrenia, such as the bleulerian autism. A possible dysfunction in the mirror neuron system may be the reason for these deficits in the disorder. We wanted to better characterize the neural networks involved in the perception of social behavior. Fifteen healthy participants were presented with video clips of 8 seconds' duration depicting either (1) one actor manipulating an object, (2) two actors with only one manipulating an object or (3) two actors cooperating in manipulating an object and 2 other control conditions. Functional magnetic resonance imaging data were acquired during watching these videos. We found the perception of social cooperation is supported by a neural network comprising the precuneus, the temporoparietal junction (supramarginal gyrus, angular gyrus, BA 39/40), the middle temporal gyrus (including superior temporal sulcus) and frontal regions (medial frontal gyrus, inferior frontal gyrus). These areas form a complex network also being activated during theory of mind and cooperative behavior tasks. Its nodes overlap with those of the mirror neuron system. Consequently, both theory of mind abilities and mirror mechanisms are relevant in the perception and understanding of social cooperative behavior. We outline the consequences of these results for a further understanding of schizophrenic psychopathology with respect to social deficits and ego disturbances.

Learning by doing? The effect of gestures on implicit retrieval of newly acquired words

Meaningful gestures enhance speech comprehensibility. However, their role during novel-word acquisition remains elusive. Here we investigate how meaningful versus meaningless gestures impact on novel-word learning and contrast these conditions to a purely verbal training. After training, neuronal processing of the novel words was assessed by blood-oxygen-level-dependent functional magnetic resonance imaging (BOLD-fMRI), disclosing that networks affording retrieval differ depending on the training condition. Over 3 days participants learned pseudowords for common objects (e.g., /klira/ -cap). For training they repeated the novel word while performing (i) an iconic, (ii) a grooming or (iii) no gesture. For the two conditions involving gestures, these were either actively repeated or passively observed during training. Behaviorally no substantial differences between the five different training conditions were found while fMRI disclosed differential networks affording implicit retrieval of the learned pseudowords depending on the training procedure. Most notably training with actively performed iconic gestures yielded larger activation in a semantic network comprising left inferior frontal (BA47) and inferior temporal gyri. Additionally hippocampal activation was stronger for all trained compared to unknown pseudowords of identical structure. The behavioral results challenge the generality of an 'enactment-effect' for single word learning. Imaging results, however, suggest that actively performed meaningful gestures lead to a deeper semantic encoding of novel words. The findings are discussed regarding their implications for theoretical accounts and for empirical approaches of gesture-based strategies in language (re)learning.

A supramodal neural network for speech and gesture semantics: an fMRI study

In a natural setting, speech is often accompanied by gestures. As language, speech-accompanying iconic gestures to some extent convey semantic information. However, if comprehension of the information contained in both the auditory and visual modality depends on same or different brain-networks is quite unknown. In this fMRI study, we aimed at identifying the cortical areas engaged in supramodal processing of semantic information. BOLD changes were recorded in 18 healthy right-handed male subjects watching video clips showing an actor who either performed speech (S, acoustic) or gestures (G, visual) in more (+) or less (−) meaningful varieties. In the experimental conditions familiar speech or isolated iconic gestures were presented; during the visual control condition the volunteers watched meaningless gestures (G−), while during the acoustic control condition a foreign language was presented (S−). The conjunction of the visual and acoustic semantic processing revealed activations extending from the left inferior frontal gyrus to the precentral gyrus, and included bilateral posterior temporal regions. We conclude that proclaiming this frontotemporal network the brain's core language system is to take too narrow a view. Our results rather indicate that these regions constitute a supramodal semantic processing network.

An overview of the neuro-cognitive processes involved in the encoding, consolidation, and retrieval of true and false memories

Perception and memory are imperfect reconstructions of reality. These reconstructions are prone to be influenced by several factors, which may result in false memories. A false memory is the recollection of an event, or details of an episode, that did not actually occur. Memory formation comprises at least three different sub-processes: encoding, consolidation and the retrieval of the learned material. All of these sub-processes are vulnerable for specific errors and consequently may result in false memories. Whereas, processes like imagery, self-referential encoding or spreading activation can lead to the formation of false memories at encoding, semantic generalization during sleep and updating processes due to misleading post event information, in particular, are relevant at the consolidation stage. Finally at the retrieval stage, monitoring processes, which are assumed to be essential to reject false memories, are of specific importance. Different neuro-cognitive processes have been linked to the formation of true and false memories. Most consistently the medial temporal lobe and the medial and lateral prefrontal cortex have been reported with regard to the formation of true and false memories. Despite the fact that all phases entailing memory formation, consolidation of stored information and retrieval processes, are relevant for the forming of false memories, most studies focused on either memory encoding or retrieval. Thus, future studies should try to integrate data from all phases to give a more comprehensive view on systematic memory distortions. An initial outline is developed within this review to connect the different memory stages and research strategies.

Gesture's Neural Language

When people talk to each other, they often make arm and hand movements that accompany what they say. These manual movements, called “co-speech gestures,” can convey meaning by way of their interaction with the oral message. Another class of manual gestures, called “emblematic gestures” or “emblems,” also conveys meaning, but in contrast to co-speech gestures, they can do so directly and independent of speech. There is currently significant interest in the behavioral and biological relationships between action and language. Since co-speech gestures are actions that rely on spoken language, and emblems convey meaning to the effect that they can sometimes substitute for speech, these actions may be important, and potentially informative, examples of language–motor interactions. Researchers have recently been examining how the brain processes these actions. The current results of this work do not yet give a clear understanding of gesture processing at the neural level. For the most part, however, it seems that two complimentary sets of brain areas respond when people see gestures, reflecting their role in disambiguating meaning. These include areas thought to be important for understanding actions and areas ordinarily related to processing language. The shared and distinct responses across these two sets of areas during communication are just beginning to emerge. In this review, we talk about the ways that the brain responds when people see gestures, how these responses relate to brain activity when people process language, and how these might relate in normal, everyday communication.

The differentiation of iconic and metaphoric gestures: common and unique integration processes

Recent research on the neural integration of speech and gesture has examined either gesture in the context of concrete [iconic (IC) gestures] or abstract sentence content [metaphoric (MP) gestures]. However, there has not yet been a direct comparison of the processing of both gesture types. This study tested the theory that left posterior temporal and inferior frontal brain regions are each uniquely involved in the integration of IC and MP gestures. During fMRI-data acquisition, participants were shown videos of an actor performing IC and MP gestures and associated sentences. An isolated gesture (G) and isolated sentence condition (S) were included to separate unimodal from bimodal effects at the neural level. During IC conditions, we found increased activity in the left posterior middle temporal gyrus and its right hemispheric homologue. The same regions in addition to the left inferior frontal gyrus (IFG) were activated during MP conditions in contrast to the isolated conditions (G&S). These findings support the hypothesis that there are distinct integration processes for IC and MP gestures. In line with recent claims of the semantic unification theory, there seems to be a division between perceptual-matching processes within the posterior temporal lobe and higher-order relational processes within the IFG.

Speech comprehension aided by multiple modalities: behavioural and neural interactions

Speech comprehension is a complex human skill, the performance of which requires the perceiver to combine information from several sources - e.g. voice, face, gesture, linguistic context - to achieve an intelligible and interpretable percept. We describe a functional imaging investigation of how auditory, visual and linguistic information interact to facilitate comprehension. Our specific aims were to investigate the neural responses to these different information sources, alone and in interaction, and further to use behavioural speech comprehension scores to address sites of intelligibility-related activation in multifactorial speech comprehension. In fMRI, participants passively watched videos of spoken sentences, in which we varied Auditory Clarity (with noise-vocoding), Visual Clarity (with Gaussian blurring) and Linguistic Predictability. Main effects of enhanced signal with increased auditory and visual clarity were observed in overlapping regions of posterior STS. Two-way interactions of the factors (auditory × visual, auditory × predictability) in the neural data were observed outside temporal cortex, where positive signal change in response to clearer facial information and greater semantic predictability was greatest at intermediate levels of auditory clarity. Overall changes in stimulus intelligibility by condition (as determined using an independent behavioural experiment) were reflected in the neural data by increased activation predominantly in bilateral dorsolateral temporal cortex, as well as inferior frontal cortex and left fusiform gyrus. Specific investigation of intelligibility changes at intermediate auditory clarity revealed a set of regions, including posterior STS and fusiform gyrus, showing enhanced responses to both visual and linguistic information. Finally, an individual differences analysis showed that greater comprehension performance in the scanning participants (measured in a post-scan behavioural test) were associated with increased activation in left inferior frontal gyrus and left posterior STS. The current multimodal speech comprehension paradigm demonstrates recruitment of a wide comprehension network in the brain, in which posterior STS and fusiform gyrus form sites for convergence of auditory, visual and linguistic information, while left-dominant sites in temporal and frontal cortex support successful comprehension.

Gesture in the developing brain

Speakers convey meaning not only through words, but also through gestures. Although children are exposed to co-speech gestures from birth, we do not know how the developing brain comes to connect meaning conveyed in gesture with speech. We used functional magnetic resonance imaging (fMRI) to address this question and scanned 8- to 11-year-old children and adults listening to stories accompanied by hand movements, either meaningful co-speech gestures or meaningless self-adaptors. When listening to stories accompanied by both types of hand movement, both children and adults recruited inferior frontal, inferior parietal, and posterior temporal brain regions known to be involved in processing language not accompanied by hand movements. There were, however, age-related differences in activity in posterior superior temporal sulcus (STSp), inferior frontal gyrus, pars triangularis (IFGTr), and posterior middle temporal gyrus (MTGp) regions previously implicated in processing gesture. Both children and adults showed sensitivity to the meaning of hand movements in IFGTr and MTGp, but in different ways. Finally, we found that hand movement meaning modulates interactions between STSp and other posterior temporal and inferior parietal regions for adults, but not for children. These results shed light on the developing neural substrate for understanding meaning contributed by co-speech gesture.

Unique brain areas associated with abstinence control are damaged in multiply detoxified alcoholics

BACKGROUND

The ability to abstain from drinking, despite incentives to imbibe, is essential to recovery from alcoholism.

METHODS

We used an incentive conflict task to investigate ability to abstain from responding during presentations of incentive cues. Both alcoholic (n = 23) and healthy subjects (n = 22) were required to withhold responding during the simultaneous presentation of two visual stimuli in which the individual presentation allowed responding for monetary reward. Brain structures activated during performance of the task were studied using functional magnetic resonance imaging in healthy volunteers (n = 8), and changes in gray matter volume were studied in a separate group of patients (n = 29) compared with control subjects (n = 31) in regions of interest identified on functional magnetic resonance imaging.

RESULTS

Abstinent alcoholic patients were severely impaired on the incentive conflict task. The impairment was greater in patients with experience of several versus a single detoxification. Healthy volunteers, during the same incentive conflict task, showed distinct patterns of brain activation (including gyrus rectus, ventromedial prefrontal cortex, and superior frontal gyrus). Reduction of gray matter volume in ventromedial prefrontal cortex and superior frontal gyrus of patients was more extensive in those with multiple detoxifications.

CONCLUSIONS

Performance deficits in alcoholics are associated with withdrawal-induced impairments in prefrontal subfields, which are exacerbated following repeated episodes of detoxification. Detoxification thus compromises functional and structural integrity of prefrontal cortex and may thus impair the ability to control future drinking. Performance in the incentive conflict task is a sensitive biomarker for such deficits.

Intention Processing in Communication: A Common Brain Network for Language and Gestures

Human communicative competence is based on the ability to process a specific class of mental states, namely, communicative intention. The present fMRI study aims to analyze whether intention processing in communication is affected by the expressive means through which a communicative intention is conveyed, that is, the linguistic or extralinguistic gestural means. Combined factorial and conjunction analyses were used to test two sets of predictions: first, that a common brain network is recruited for the comprehension of communicative intentions independently of the modality through which they are conveyed; second, that additional brain areas are specifically recruited depending on the communicative modality used, reflecting distinct sensorimotor gateways. Our results clearly showed that a common neural network is engaged in communicative intention processing independently of the modality used. This network includes the precuneus, the left and right posterior STS and TPJ, and the medial pFC. Additional brain areas outside those involved in intention processing are specifically engaged by the particular communicative modality, that is, a peri-sylvian language network for the linguistic modality and a sensorimotor network for the extralinguistic modality. Thus, common representation of communicative intention may be accessed by modality-specific gateways, which are distinct for linguistic versus extralinguistic expressive means. Taken together, our results indicate that the information acquired by different communicative modalities is equivalent from a mental processing standpoint, in particular, at the point at which the actor's communicative intention has to be reconstructed.

What Iconic Gesture Fragments Reveal about Gesture–Speech Integration: When Synchrony Is Lost, Memory Can Help

The present series of experiments explores several issues related to gesture–speech integration and synchrony during sentence processing. To be able to more precisely manipulate gesture–speech synchrony, we used gesture fragments instead of complete gestures, thereby avoiding the usual long temporal overlap of gestures with their coexpressive speech. In a pretest, the minimal duration of an iconic gesture fragment needed to disambiguate a homonym (i.e., disambiguation point) was therefore identified. In three subsequent ERP experiments, we then investigated whether the gesture information available at the disambiguation point has immediate as well as delayed consequences on the processing of a temporarily ambiguous spoken sentence, and whether these gesture–speech integration processes are susceptible to temporal synchrony. Experiment 1, which used asynchronous stimuli as well as an explicit task, showed clear N400 effects at the homonym as well as at the target word presented further downstream, suggesting that asynchrony does not prevent integration under explicit task conditions. No such effects were found when asynchronous stimuli were presented using a more shallow task (Experiment 2). Finally, when gesture fragment and homonym were synchronous, similar results as in Experiment 1 were found, even under shallow task conditions (Experiment 3). We conclude that when iconic gesture fragments and speech are in synchrony, their interaction is more or less automatic. When they are not, more controlled, active memory processes are necessary to be able to combine the gesture fragment and speech context in such a way that the homonym is disambiguated correctly.

Neural correlates of intentional communication

We know a great deal about the neurophysiological mechanisms supporting instrumental actions, i.e., actions designed to alter the physical state of the environment. In contrast, little is known about our ability to select communicative actions, i.e., actions directly designed to modify the mental state of another agent. We have recently provided novel empirical evidence for a mechanism in which a communicator selects his actions on the basis of a prediction of the communicative intentions that an addressee is most likely to attribute to those actions. The main novelty of those findings was that this prediction of intention recognition is cerebrally implemented within the intention recognition system of the communicator, is modulated by the ambiguity in meaning of the communicative acts, and not by their sensorimotor complexity. The characteristics of this predictive mechanism support the notion that human communicative abilities are distinct from both sensorimotor and linguistic processes.

Encoding social interactions: the neural correlates of true and false memories

In social situations, we encounter information transferred in firsthand (egocentric) and secondhand (allocentric) communication contexts. However, the mechanism by which an individual distinguishes whether a past interaction occurred in an egocentric versus allocentric situation is poorly understood. This study examined the neural bases for encoding memories of social interactions through experimentally manipulating the communication context. During fMRI data acquisition, participants watched video clips of an actor speaking and gesturing directly toward them (egocentric context) or toward an unseen third person (allocentric context). After scanning, a recognition task gauged participants' ability to recognize the sentences they had just seen and to recall the context in which the sentences had been spoken. We found no differences between the recognition of sentences spoken in egocentric and allocentric contexts. However, when asked about the communication context ("Had the actor directly spoken to you?"), participants tended to believe falsely that the actor had directly spoken to them during allocentric conditions. Greater activity in the hippocampus was related to correct context memory, whereas the ventral ACC was activated for subsequent inaccurate context memory. For the interaction between encoding context and context memory, we observed increased activation for egocentric remembered items in the bilateral and medial frontal cortex, the BG, and the left parietal and temporal lobe. Our data indicate that memories of social interactions are biased to be remembered egocentrically. Self-referential encoding processes reflected in increased frontal activation and decreased hippocampal activation might be the basis of correct item but false context memory of social interactions.

Social cues, mentalizing and the neural processing of speech accompanied by gestures

Body orientation and eye gaze influence how information is conveyed during face-to-face communication. However, the neural pathways underpinning the comprehension of social cues in everyday interaction are not known. In this study we investigated the influence of addressing vs. non-addressing body orientation on the neural processing of speech accompanied by gestures. While in an fMRI scanner, participants viewed short video clips of an actor speaking sentences with object- (O; e.g., shape) or person-related content (P; e.g., saying goodbye) accompanied by iconic (e.g., circle) or emblematic gestures (e.g., waving), respectively. The actor's body was oriented either toward the participant (frontal, F) or toward a third person (lateral, L) not visible. For frontal vs. lateral actor orientation (F>L), we observed activation of bilateral occipital, inferior frontal, medial frontal, right anterior temporal and left parietal brain regions. Additionally, we observed activity in the occipital and anterior temporal lobes due to an interaction effect between actor orientation and content of the communication (PF>PL)>(OF>OL). Our findings indicate that social cues influence the neural processing of speech-gesture utterances. Mentalizing (the process of inferring the mental state of another individual) could be responsible for these effects. In particular, socially relevant cues seem to activate regions of the anterior temporal lobes if abstract person-related content is communicated by speech and gesture. These new findings illustrate the complexity of interpersonal communication, as our data demonstrate that multisensory information pathways interact at both perceptual and semantic levels.