Encoding social interactions: the neural correlates of true and false memories

Gesture's body orientation modulates the N400 for visual sentences primed by gestures

Body orientation of gesture entails social-communicative intention, and may thus influence how gestures are perceived and comprehended together with auditory speech during face-to-face communication. To date, despite the emergence of neuroscientific literature on the role of body orientation on hand action perception, limited studies have directly investigated the role of body orientation in the interaction between gesture and language. To address this research question, we carried out an electroencephalography (EEG) experiment presenting to participants (n = 21) videos of frontal and lateral communicative hand gestures of 5 s (e.g., raising a hand), followed by visually presented sentences that are either congruent or incongruent with the gesture (e.g., "the mountain is high/low…"). Participants underwent a semantic probe task, judging whether a target word is related or unrelated to the gesture-sentence event. EEG results suggest that, during the perception phase of handgestures, while both frontal and lateral gestures elicited a power decrease in both the alpha (8-12 Hz) and the beta (16-24 Hz) bands, lateral versus frontal gestures elicited reduced power decrease in the beta band, source-located to the medial prefrontal cortex. For sentence comprehension, at the critical word whose meaning is congruent/incongruent with the gesture prime, frontal gestures elicited an N400 effect for gesture-sentence incongruency. More importantly, this incongruency effect was significantly reduced for lateral gestures. These findings suggest that body orientation plays an important role in gesture perception, and that its inferred social-communicative intention may influence gesture-language interaction at semantic level.

How does social competition affect true and false recognition?

Memory is highly susceptible to distortions, which can exert serious consequences in daily life. Despite this, we still know little about the role of factors that comprise social contexts in which memory processes occur. In the present study, we attempted to address this issue by examining how social competition influences true and false recognition. Participants performed a version of the Deese-Roediger-McDermott (DRM) paradigm designed to lure them into producing both true and false recognition either in competition against or independently of another person. We found that participants in the competition group showed lower levels of true and false recognition than those in the control group. Signal-detection analyses revealed that participants in both groups showed equivalent memory sensitivity for true recognition, while those in the competition group exhibited a decreased sensitivity for false recognition, which implies enhanced item-specific encoding during social competition. Moreover, participants in the competition group showed a more conservative response bias for both true and false recognition at retrieval than those in the control group, indicating a shift towards conservatism in decision strategy for both true and false recognition during social competition. The results provide compelling evidence for a decision-based reduction of true recognition and both encoding-based and decision-based reductions of false recognition under competitive contexts. Therefore, these novel findings may have implications both for understanding the powerful role of social competition on true and false memories and for understanding the potential role of social competition on other aspects of memory processes.

Feeling addressed! The neural processing of social communicative cues in patients with major depression

The feeling of being addressed is the first step in a complex processing stream enabling successful social communication. Social impairments are a relevant characteristic of patients with major depressive disorder (MDD). Here, we investigated a mechanism which—if impaired—might contribute to withdrawal or isolation in MDD, namely, the neural processing of social cues such as body orientation and gesture. During funtional magnetic resonance imaging (fMRI) data acquisition, 33 patients with MDD and 43 healthy control subjects watched video clips of a speaking actor: one version with a gesture accompanying the speech and one without gesture. Videos were filmed simultaneously from two different viewpoints: one with the actor facing the viewer head‐on (frontal) and one side‐view (lateral). After every clip, the participants were instructed to evaluate whether they felt addressed or not. Despite overall comparable addressment ratings and a large overlap in activation patterns in MDD and healthy subjects for gesture processing, the anterior cingulate cortex, bilateral superior/middle frontal cortex, and right angular gyrus were more strongly activated in patients than in healthy subjects for the frontal conditions. Our analyses revealed that patients showed specifically higher activation than healthy subjects for the frontal condition without gesture in regions including the posterior cingulate cortex, left prefrontal cortex, and the left hippocampus. We conclude that MDD patients can recognize and interpret social cues such as gesture or body orientation; however, they seem to require more neural resources. This additional effort might affect successful communication and contribute to social isolation in MDD.

A Multimodal Speech-Gesture Training Intervention for Patients With Schizophrenia and Its Neural Underpinnings - the Study Protocol of a Randomized Controlled Pilot Trial

Dysfunctional social communication is one of the most stable characteristics in patients with schizophrenia that also affects quality of life. Interpreting abstract speech and integrating nonverbal modalities is particularly affected. Considering the impact of communication on social life but failure to treat communication dysfunctions with usual treatment, we will investigate the possibility to improve verbal and non-verbal communication in schizophrenia by applying a multimodal speech-gesture training (MSG training). Here we describe the newly developed MSG training program and the study design for the first clinical investigation. The intervention contains perceptive rating (match/mismatch of sentence and gesture) and memory tasks (n-back tasks), imitation and productive tasks (e.g., SG fluency-similar to verbal fluency where words are accompanied by gesture). In addition, we offer information about gesture as meta-learning element as well as homework for reasons of transfer to everyday life as part of every session. In the MSG training intervention, we offer eight sessions (60 min each) of training. The first pilot study is currently conducted as a single-center, randomized controlled trial of speech-gesture intervention versus wait-list control with a follow-up. Outcomes are measured through pre-post-fMRI and standardized psychological questionnaires comparing two subject groups (30 patients with schizophrenia and 30 healthy controls). Patients and healthy controls are randomized in two intervention groups (with 20 being in the wait-training group and 10 in the training-follow-up group). With our study design we will be able to demonstrate the beneficial effect of the MSG training intervention on behavioral and neural levels.

CLINICAL TRIAL REGISTRATION

DRKS.de, identifier DRKS00015118.

The connectivity signature of co-speech gesture integration: The superior temporal sulcus modulates connectivity between areas related to visual gesture and auditory speech processing

Humans integrate information communicated by speech and gestures. Functional magnetic resonance imaging (fMRI) studies suggest that the posterior superior temporal sulcus (STS) and adjacent gyri are relevant for multisensory integration. However, a connectivity model representing this essential combinatory process is still missing. Here, we used dynamic causal modeling for fMRI to analyze the effective connectivity pattern between middle temporal gyrus (MTG), occipital cortex (OC) and STS associated with auditory verbal, visual gesture-related, and integrative processing, respectively, to unveil the neural mechanisms underlying integration of intrinsically meaningful gestures (e.g., "Thumbs-up gesture") and corresponding speech. 20 participants were presented videos of an actor either performing intrinsic meaningful gestures in the context of German or Russian sentences, or speaking a German sentence without gesture, while performing a content judgment task. The connectivity analyses resulted in a winning model that included bidirectional intrinsic connectivity between all areas. Furthermore, the model included modulations of both connections to the STS (OC→STS; MTG→STS), and non-linear modulatory effects of the STS on bidirectional connections between MTG and OC. Coupling strength in the occipital pathway (OC→STS) correlated with gesture related advantages in task performance, whereas the temporal pathway (MTG→STS) correlated with performance in the speech only condition. Coupling between MTG and OC correlated negatively with subsequent memory performance for sentences of the Gesture-German condition. Our model provides a first step towards a better understanding of speech-gesture integration on network level. It corroborates the importance of the STS during audio-visual integration by showing that this region inhibits direct auditory-visual coupling.

Hetereogeneity in Neuronal Intrinsic Properties: A Possible Mechanism for Hub-Like Properties of the Rat Anterior Cingulate Cortex during Network Activity

The anterior cingulate cortex (ACC) is vital for a range of brain functions requiring cognitive control and has highly divergent inputs and outputs, thus manifesting as a hub in connectomic analyses. Studies show diverse functional interactions within the ACC are associated with network oscillations in the β (20–30 Hz) and γ (30-80 Hz) frequency range. Oscillations permit dynamic routing of information within cortex, a function that depends on bandpass filter–like behavior to selectively respond to specific inputs. However, a putative hub region such as ACC needs to be able to combine inputs from multiple sources rather than select a single input at the expense of others. To address this potential functional dichotomy, we modeled local ACC network dynamics in the rat in vitro. Modal peak oscillation frequencies in the β- and γ-frequency band corresponded to GABA_Aergic synaptic kinetics as seen in other regions; however, the intrinsic properties of ACC principal neurons were highly diverse. Computational modeling predicted that this neuronal response diversity broadened the bandwidth for filtering rhythmic inputs and supported combination—rather than selection—of different frequencies within the canonical γ and β electroencephalograph bands. These findings suggest that oscillating neuronal populations can support either response selection (routing) or combination, depending on the interplay between the kinetics of synaptic inhibition and the degree of heterogeneity of principal cell intrinsic conductances.

Predicting the Multisensory Consequences of One's Own Action: BOLD Suppression in Auditory and Visual Cortices

Predictive mechanisms are essential to successfully interact with the environment and to compensate for delays in the transmission of neural signals. However, whether and how we predict multisensory action outcomes remains largely unknown. Here we investigated the existence of multisensory predictive mechanisms in a context where actions have outcomes in different modalities. During fMRI data acquisition auditory, visual and auditory-visual stimuli were presented in active and passive conditions. In the active condition, a self-initiated button press elicited the stimuli with variable short delays (0-417ms) between action and outcome, and participants had to detect the presence of a delay for auditory or visual outcome (task modality). In the passive condition, stimuli appeared automatically, and participants had to detect the number of stimulus modalities (unimodal/bimodal). For action consequences compared to identical but unpredictable control stimuli we observed suppression of the blood oxygen level depended (BOLD) response in a broad network including bilateral auditory and visual cortices. This effect was independent of task modality or stimulus modality and strongest for trials where no delay was detected (undetected<detected). In bimodal vs. unimodal conditions we found activation differences in the left cerebellum for detected vs. undetected trials and an increased cerebellar-sensory cortex connectivity. Thus, action-related predictive mechanisms lead to BOLD suppression in multiple sensory brain regions. These findings support the hypothesis of multisensory predictive mechanisms, which are probably conducted in the left cerebellum.

Play along: effects of music and social interaction on word learning

Learning new words is an increasingly common necessity in everyday life. External factors, among which music and social interaction are particularly debated, are claimed to facilitate this task. Due to their influence on the learner's temporal behavior, these stimuli are able to drive the learner's attention to the correct referent of new words at the correct point in time. However, do music and social interaction impact learning behavior in the same way? The current study aims to answer this question. Native German speakers (N = 80) were requested to learn new words (pseudo-words) during a contextual learning game. This learning task was performed alone with a computer or with a partner, with or without music. Results showed that music and social interaction had a different impact on the learner's behavior: Participants tended to temporally coordinate their behavior more with a partner than with music, and in both cases more than with a computer. However, when both music and social interaction were present, this temporal coordination was hindered. These results suggest that while music and social interaction do influence participants' learning behavior, they have a different impact. Moreover, impaired behavior when both music and a partner are present suggests that different mechanisms are employed to coordinate with the two types of stimuli. Whether one or the other approach is more efficient for word learning, however, is a question still requiring further investigation, as no differences were observed between conditions in a retrieval phase, which took place immediately after the learning session. This study contributes to the literature on word learning in adults by investigating two possible facilitating factors, and has important implications for situations such as music therapy, in which music and social interaction are present at the same time.

Feeling addressed! The role of body orientation and co-speech gesture in social communication

During face-to-face communication, body orientation and coverbal gestures influence how information is conveyed. The neural pathways underpinning the comprehension of such nonverbal social cues in everyday interaction are to some part still unknown. During fMRI data acquisition, 37 participants were presented with video clips showing an actor speaking short sentences. The actor produced speech-associated iconic gestures (IC) or no gestures (NG) while he was visible either from an egocentric (ego) or from an allocentric (allo) position. Participants were asked to indicate via button press whether they felt addressed or not. We found a significant interaction of body orientation and gesture in addressment evaluations, indicating that participants evaluated IC-ego conditions as most addressing. The anterior cingulate cortex (ACC) and left fusiform gyrus were stronger activated for egocentric versus allocentric actor position in gesture context. Activation increase in the ACC for IC-ego>IC-allo further correlated positively with increased addressment ratings in the egocentric gesture condition. Gesture-related activation increase in the supplementary motor area, left inferior frontal gyrus and right insula correlated positively with gesture-related increase of addressment evaluations in the egocentric context. Results indicate that gesture use and body-orientation contribute to the feeling of being addressed and together influence neural processing in brain regions involved in motor simulation, empathy and mentalizing.

Superior temporal sulcus disconnectivity during processing of metaphoric gestures in schizophrenia

The left superior temporal sulcus (STS) plays an important role in integrating audiovisual information and is functionally connected to disparate regions of the brain. For the integration of gesture information in an abstract sentence context (metaphoric gestures), intact connectivity between the left STS and the inferior frontal gyrus (IFG) should be important. Patients with schizophrenia have problems with the processing of metaphors (concretism) and show aberrant structural connectivity of long fiber bundles. Thus, we tested the hypothesis that patients with schizophrenia differ in the functional connectivity of the left STS to the IFG for the processing of metaphoric gestures. During functional magnetic resonance imaging data acquisition, 16 patients with schizophrenia (P) and a healthy control group (C) were shown videos of an actor performing gestures in a concrete (iconic, IC) and abstract (metaphoric, MP) sentence context. A psychophysiological interaction analysis based on the seed region from a previous analysis in the left STS was performed. In both groups we found common positive connectivity for IC and MP of the STS seed region to the left middle temporal gyrus (MTG) and left ventral IFG. The interaction of group (C>P) and gesture condition (MP>IC) revealed effects in the connectivity to the bilateral IFG and the left MTG with patients exhibiting lower connectivity for the MP condition. In schizophrenia the left STS is misconnected to the IFG, particularly during the processing of MP gestures. Dysfunctional integration of gestures in an abstract sentence context might be the basis of certain interpersonal communication problems in the patients.

Semantic relation vs. surprise: the differential effects of related and unrelated co-verbal gestures on neural encoding and subsequent recognition

Speech-associated gesturing leads to memory advantages for spoken sentences. However, unexpected or surprising events are also likely to be remembered. With this study we test the hypothesis that different neural mechanisms (semantic elaboration and surprise) lead to memory advantages for iconic and unrelated gestures. During fMRI-data acquisition participants were presented with video clips of an actor verbalising concrete sentences accompanied by iconic gestures (IG; e.g., circular gesture; sentence: "The man is sitting at the round table"), unrelated free gestures (FG; e.g., unrelated up down movements; same sentence) and no gestures (NG; same sentence). After scanning, recognition performance for the three conditions was tested. Videos were evaluated regarding semantic relation and surprise by a different group of participants. The semantic relationship between speech and gesture was rated higher for IG (IG>FG), whereas surprise was rated higher for FG (FG>IG). Activation of the hippocampus correlated with subsequent memory performance of both gesture conditions (IG+FG>NG). For the IG condition we found activation in the left temporal pole and middle cingulate cortex (MCC; IG>FG). In contrast, for the FG condition posterior thalamic structures (FG>IG) as well as anterior and posterior cingulate cortices were activated (FG>NG). Our behavioral and fMRI-data suggest different mechanisms for processing related and unrelated co-verbal gestures, both of them leading to enhanced memory performance. Whereas activation in MCC and left temporal pole for iconic co-verbal gestures may reflect semantic memory processes, memory enhancement for unrelated gestures relies on the surprise response, mediated by anterior/posterior cingulate cortex and thalamico-hippocampal structures.

Eye'm talking to you: speakers' gaze direction modulates co-speech gesture processing in the right MTG

Recipients process information from speech and co-speech gestures, but it is currently unknown how this processing is influenced by the presence of other important social cues, especially gaze direction, a marker of communicative intent. Such cues may modulate neural activity in regions associated either with the processing of ostensive cues, such as eye gaze, or with the processing of semantic information, provided by speech and gesture. Participants were scanned (fMRI) while taking part in triadic communication involving two recipients and a speaker. The speaker uttered sentences that were and were not accompanied by complementary iconic gestures. Crucially, the speaker alternated her gaze direction, thus creating two recipient roles: addressed (direct gaze) vs unaddressed (averted gaze) recipient. The comprehension of Speech&Gesture relative to SpeechOnly utterances recruited middle occipital, middle temporal and inferior frontal gyri, bilaterally. The calcarine sulcus and posterior cingulate cortex were sensitive to differences between direct and averted gaze. Most importantly, Speech&Gesture utterances, but not SpeechOnly utterances, produced additional activity in the right middle temporal gyrus when participants were addressed. Marking communicative intent with gaze direction modulates the processing of speech-gesture utterances in cerebral areas typically associated with the semantic processing of multi-modal communicative acts.

Cognitive mapping deficits in schizophrenia: a critical overview

Hippocampal deficits are an established feature of schizophrenia and are complementary with recent evidences of marked allocentric processing deficits being reported in this disorder. By "Cognitive mapping" we intend to refer to the concepts from the seminal works of O'Keefe and Nadel (1978) that led to the development of cognitive map theory of hippocampal function. In this review, we summarize emerging evidences and issues that indicate that "Cognitive mapping deficits" form one of the important cognitive aberrations in schizophrenia. The importance has been placed upon hippocampally mediated allocentric processing deficits and their role in pathology of schizophrenia, for spatial/representational cognitive deficits and positive symptoms in particular. It is modestly summarized that emerging evidences point toward a web of spatial and cognitive representation errors concurrent with pronounced hippocampal dysfunction. In general, it can be stated that there are clear and consistent evidences that favor the cognitive mapping theory in explaining certain deficits of schizophrenia and for drawing out a possible and promising endophenotype/biomarkers. Further research in this regard demands attention.

Supramodal neural processing of abstract information conveyed by speech and gesture

Abstractness and modality of interpersonal communication have a considerable impact on comprehension. They are relevant for determining thoughts and constituting internal models of the environment. Whereas concrete object-related information can be represented in mind irrespective of language, abstract concepts require a representation in speech. Consequently, modality-independent processing of abstract information can be expected. Here we investigated the neural correlates of abstractness (abstract vs. concrete) and modality (speech vs. gestures), to identify an abstractness-specific supramodal neural network. During fMRI data acquisition 20 participants were presented with videos of an actor either speaking sentences with an abstract-social [AS] or concrete-object-related content [CS], or performing meaningful abstract-social emblematic [AG] or concrete-object-related tool-use gestures [CG]. Gestures were accompanied by a foreign language to increase the comparability between conditions and to frame the communication context of the gesture videos. Participants performed a content judgment task referring to the person vs. object-relatedness of the utterances. The behavioral data suggest a comparable comprehension of contents communicated by speech or gesture. Furthermore, we found common neural processing for abstract information independent of modality (AS > CS ∩ AG > CG) in a left hemispheric network including the left inferior frontal gyrus (IFG), temporal pole, and medial frontal cortex. Modality specific activations were found in bilateral occipital, parietal, and temporal as well as right inferior frontal brain regions for gesture (G > S) and in left anterior temporal regions and the left angular gyrus for the processing of speech semantics (S > G). These data support the idea that abstract concepts are represented in a supramodal manner. Consequently, gestures referring to abstract concepts are processed in a predominantly left hemispheric language related neural network.

Neural integration of speech and gesture in schizophrenia: evidence for differential processing of metaphoric gestures

Gestures are an important component of interpersonal communication. Especially, complex multimodal communication is assumed to be disrupted in patients with schizophrenia. In healthy subjects, differential neural integration processes for gestures in the context of concrete [iconic (IC) gestures] and abstract sentence contents [metaphoric (MP) gestures] had been demonstrated. With this study we wanted to investigate neural integration processes for both gesture types in patients with schizophrenia. During functional magnetic resonance imaging-data acquisition, 16 patients with schizophrenia (P) and a healthy control group (C) were shown videos of an actor performing IC and MP gestures and associated sentences. An isolated gesture (G) and isolated sentence condition (S) were included to separate unimodal from bimodal effects at the neural level. During IC conditions (IC > G ∩ IC > S) we found increased activity in the left posterior middle temporal gyrus (pMTG) in both groups. Whereas in the control group the left pMTG and the inferior frontal gyrus (IFG) were activated for the MP conditions (MP > G ∩ MP > S), no significant activation was found for the identical contrast in patients. The interaction of group (P/C) and gesture condition (MP/IC) revealed activation in the bilateral hippocampus, the left middle/superior temporal and IFG. Activation of the pMTG for the IC condition in both groups indicates intact neural integration of IC gestures in schizophrenia. However, failure to activate the left pMTG and IFG for MP co-verbal gestures suggests a disturbed integration of gestures embedded in an abstract sentence context. This study provides new insight into the neural integration of co-verbal gestures in patients with schizophrenia.

Differential role of the Mentalizing and the Mirror Neuron system in the imitation of communicative gestures

Successful social interaction requires recognising the intention of another person's communicative gestures. At a neural level, this process may involve neural activity in different systems, such as the mentalizing system (MS) and the mirror neuron system (MNS). The aim of the present study was to explore the neural correlates of communicative gestures during observation and execution of these gestures. Twenty participants watched video clips of an actor executing social gestures (S), non-social gestures (NS) and meaningless gestures (ML). During fMRI data acquisition, participants were asked to observe (O) and subsequently to execute (E) one of two tasks: imitate the gesture presented (IMI) or perform a motor control task (CT). For the contrast IMI>CT we found activations in the core areas of the MNS [inferior parietal lobule (IPL) and inferior frontal cortex, the posterior part of pars opercularis], as well as in areas related to the MS [superior temporal sulcus (STS) and middle cingulate cortex]. For S>NS, we found activations in the left medial orbitofrontal cortex (mOFC), right superior frontal cortex and middle cingulate cortex. The interaction of stimulus condition (S vs NS) and task (IMI vs CT) revealed activation in the right IPL. For the interaction between observation vs execution (O vs E), task (IMI vs CT) and stimulus condition (S vs NS) we found activation in the right mOFC. Our data suggest that imitation is differentially processed in the MNS as well as in the MS. The activation in IPL is enhanced during the processing of social gestures most likely due to their communicative intention. The activation of IPL together with medial frontal areas may contribute to mentalizing processes. The interaction in the mOFC suggests an involvement of self-referential processes in the processing of social gesture.

A supramodal neural network for speech and gesture semantics: an fMRI study

In a natural setting, speech is often accompanied by gestures. As language, speech-accompanying iconic gestures to some extent convey semantic information. However, if comprehension of the information contained in both the auditory and visual modality depends on same or different brain-networks is quite unknown. In this fMRI study, we aimed at identifying the cortical areas engaged in supramodal processing of semantic information. BOLD changes were recorded in 18 healthy right-handed male subjects watching video clips showing an actor who either performed speech (S, acoustic) or gestures (G, visual) in more (+) or less (−) meaningful varieties. In the experimental conditions familiar speech or isolated iconic gestures were presented; during the visual control condition the volunteers watched meaningless gestures (G−), while during the acoustic control condition a foreign language was presented (S−). The conjunction of the visual and acoustic semantic processing revealed activations extending from the left inferior frontal gyrus to the precentral gyrus, and included bilateral posterior temporal regions. We conclude that proclaiming this frontotemporal network the brain's core language system is to take too narrow a view. Our results rather indicate that these regions constitute a supramodal semantic processing network.

An overview of the neuro-cognitive processes involved in the encoding, consolidation, and retrieval of true and false memories

Perception and memory are imperfect reconstructions of reality. These reconstructions are prone to be influenced by several factors, which may result in false memories. A false memory is the recollection of an event, or details of an episode, that did not actually occur. Memory formation comprises at least three different sub-processes: encoding, consolidation and the retrieval of the learned material. All of these sub-processes are vulnerable for specific errors and consequently may result in false memories. Whereas, processes like imagery, self-referential encoding or spreading activation can lead to the formation of false memories at encoding, semantic generalization during sleep and updating processes due to misleading post event information, in particular, are relevant at the consolidation stage. Finally at the retrieval stage, monitoring processes, which are assumed to be essential to reject false memories, are of specific importance. Different neuro-cognitive processes have been linked to the formation of true and false memories. Most consistently the medial temporal lobe and the medial and lateral prefrontal cortex have been reported with regard to the formation of true and false memories. Despite the fact that all phases entailing memory formation, consolidation of stored information and retrieval processes, are relevant for the forming of false memories, most studies focused on either memory encoding or retrieval. Thus, future studies should try to integrate data from all phases to give a more comprehensive view on systematic memory distortions. An initial outline is developed within this review to connect the different memory stages and research strategies.