Communication with emblematic gestures: shared and distinct neural correlates of expression and reception

Mapping relational links between motor imagery, action observation, action-related language, and action execution

Actions can be physically executed, observed, imagined, or simply thought about. Unifying mental processes, such as simulation, emulation, or predictive processing, are thought to underlie different action types, whether they are mental states, as in the case of motor imagery and action observation, or involve physical execution. While overlapping brain activity is typically observed across different actions which indicates commonalities, research interest is also concerned with investigating the distinct functional components of these action types. Unfortunately, untangling subtleties associated with the neurocognitive bases of different action types is a complex endeavour due to the high dimensional nature of their neural substrate (e.g., any action process is likely to activate multiple brain regions thereby having multiple dimensions to consider when comparing across them). This has impeded progress in action-related theorising and application. The present study addresses this challenge by using the novel approach of multidimensional modeling to reduce the high-dimensional neural substrate of four action-related behaviours (motor imagery, action observation, action-related language, and action execution), find the least number of dimensions that distinguish or relate these action types, and characterise their neurocognitive relational links. Data for the model comprised brain activations for action types from whole-brain analyses reported in 53 published articles. Eighty-two dimensions (i.e., 82 brain regions) for the action types were reduced to a three-dimensional model, that mapped action types in ordination space where the greater the distance between the action types, the more dissimilar they are. A series of one-way ANOVAs and post-hoc comparisons performed on the mean coordinates for each action type in the model showed that across all action types, action execution and concurrent action observation (AO)-motor imagery (MI) were most neurocognitively similar, while action execution and AO were most dissimilar. Most action types were similar on at least one neurocognitive dimension, the exception to this being action-related language. The import of the findings are discussed in terms of future research and implications for application.

The Design and Observed Effects of Robot-Performed Manual Gestures: A Systematic Review

Communication using manual (hand) gestures is considered a defining property of social robots, and their physical embodiment and presence, therefore we see a need for a comprehensive overview of the state of the art in social robots that use gestures. This systematic literature review aims to address this need by (1) describing the gesture production process of a social robot, including the design and planning steps, and (2) providing a survey of the effects of robot-performed gestures on human-robot interactions in a multitude of domains. We identify patterns and themes from the existing body of literature, resulting in nine outstanding questions for research on robot-performed gestures regarding: developments in sensor technology and AI, structuring the gesture design and evaluation process, the relationship between physical appearance and gestures, the effects of planning on the overall interaction, standardizing measurements of gesture ‘quality’, individual differences, gesture mirroring, whether human-likeness is desirable, and universal accessibility of robots. We also reflect on current methodological practices in studies of robot-performed gestures, and suggest improvements regarding replicability, external validity, measurement instruments used, and connections with other disciplines. These outstanding questions and methodological suggestions can guide future work in this field of research.

Cerebellar contribution to emotional body language perception: a TMS study

Consistent evidence suggests that the cerebellum contributes to the processing of emotional facial expressions. However, it is not yet known whether the cerebellum is recruited when emotions are expressed by body postures or movements, or whether it is recruited differently for positive and negative emotions. In this study, we asked healthy participants to discriminate between body postures (with masked face) expressing emotions of opposite valence (happiness vs anger, Experiment 1), or of the same valence (negative: anger vs sadness; positive: happiness vs surprise, Experiment 2). While performing the task, participants received online transcranial magnetic stimulation (TMS) over a region of the posterior left cerebellum and over two control sites (early visual cortex and vertex). We found that TMS over the cerebellum affected participants' ability to discriminate emotional body postures, but only when one of the emotions was negatively valenced (i.e. anger). These findings suggest that the cerebellar region we stimulated is involved in processing the emotional content conveyed by body postures and gestures. Our findings complement prior evidence on the role of the cerebellum in emotional face processing and have important implications from a clinical perspective, where non-invasive cerebellar stimulation is a promising tool for the treatment of motor, cognitive and affective deficits.

The engaging nature of interactive gestures

The social interactions that we experience from early infancy often involve actions that are not strictly instrumental but engage the recipient by eliciting a (complementary) response. Interactive gestures may have privileged access to our perceptual and motor systems either because of their intrinsically engaging nature or as a result of extensive social learning. We compared these two hypotheses in a series of behavioral experiments by presenting individuals with interactive gestures that call for motor responses to complement the interaction ('hand shaking', 'requesting', 'high-five') and with communicative gestures that are equally socially relevant and salient, but do not strictly require a response from the recipient ('Ok', 'Thumbs up', 'Peace'). By means of a spatial compatibility task, we measured the interfering power of these task-irrelevant stimuli on the behavioral responses of individuals asked to respond to a target. Across three experiments, our results showed that the interactive gestures impact on response selection and reduce spatial compatibility effects as compared to the communicative (non-interactive) gestures. Importantly, this effect was independent of the activation of specific social scripts that may interfere with response selection. Overall, our results show that interactive gestures have privileged access to our perceptual and motor systems, possibly because they entail an automatic preparation to respond that involuntary engages the motor system of the observers. We discuss the implications from a developmental and neurophysiological point of view.

A meta-analysis of neuroimaging studies on pain empathy: investigating the role of visual information and observers' perspective

Empathy relies on brain systems that support the interaction between an observer's mental state and cues about the others' experience. Beyond the core brain areas typically activated in pain empathy studies (insular and anterior cingulate cortices), the diversity of paradigms used may reveal secondary networks that subserve other more specific processes. A coordinate-based meta-analysis of fMRI experiments on pain empathy was conducted to obtain activation likelihood estimates along three factors and seven conditions: visual cues (body parts, facial expressions), visuospatial (first-person, thirdperson), and cognitive (self-, stimuli-, other-oriented tasks) perspectives. The core network was found across cues and perspectives, and common activation was observed in higher-order visual areas. Body-parts distinctly activated areas related with sensorimotor processing (superior and inferior parietal lobules, anterior insula) while facial expression distinctly involved the inferior frontal gyrus. Self- compared to other-perspective produced distinct activations in the left insula while stimulus- versus other-perspective produced distinctive responses in the inferior frontal and parietal lobules, precentral gyrus, and cerebellum. Pain empathy relies on a core network which is modulated by several secondary networks. The involvement of the latter seems to depend on the visual cues available and the observer's mental state that can be influenced by specific instructions.

From Believing to Belief: A General Theoretical Model

Cognitive neuroscience research has begun to explore the mental processes underlying what a belief and what believing are. Recent evidence suggests that believing involves fundamental brain functions that result in meaningful probabilistic representations, called beliefs. When relatively stable, these beliefs allow for guidance of behavior in individuals and social groups. However, they are also fluid and can be modified by new relevant information, interpersonal contact, social pressure, and situational demands. We present a theoretical model of believing that can account for the formation of both empirically grounded and metaphysical beliefs.

Systems Neuroscience of Psychosis: Mapping Schizophrenia Symptoms onto Brain Systems

Schizophrenia research has been in a deadlock for many decades. Despite important advances in clinical treatment, there are still major concerns regarding long-term psychosocial reintegration and disease management, biological heterogeneity, unsatisfactory predictors of individual course and treatment strategies, and a confusing variety of controversial theories about its etiology and pathophysiological mechanisms. In the present perspective on schizophrenia research, we first discuss a methodological pitfall in contemporary schizophrenia research inherent in the attempt to link mental phenomena with the brain: we claim that the time-honored phenomenological method of defining mental symptoms should not be contaminated with the naturalistic approach of modern neuroscience. We then describe our Systems Neuroscience of Psychosis (SyNoPsis) project, which aims to overcome this intrinsic problem of psychiatric research. Considering schizophrenia primarily as a disorder of interindividual communication, we developed a neurobiologically informed semiotics of psychotic disorders, as well as an operational clinical rating scale. The novel psychopathology allows disentangling the clinical manifestations of schizophrenia into behavioral domains matching the functions of three well-described higher-order corticobasal brain systems involved in interindividual human communication, namely, the limbic, associative, and motor loops, including their corticocortical sensorimotor connections. The results of several empirical studies support the hypothesis that the proposed three-dimensional symptom structure, segregated into the affective, the language, and the motor domain, can be specifically mapped onto structural and functional abnormalities of the respective brain systems. New pathophysiological hypotheses derived from this brain system-oriented approach have helped to develop and improve novel treatment strategies with noninvasive brain stimulation and practicable clinical parameters. In clinical practice, the novel psychopathology allows confining the communication deficits of the individual patient, shifting attention from the symptoms to the intact resources. We have studied this approach and observed important advantages for therapeutic alliances, personalized treatment, and de-escalation strategies. Future studies will further conjoin clinical definitions of psychotic symptoms with brain structures and functions, and disentangle structural and functional deficit patterns within these systems to identify neurobiologically distinct subsyndromes. Neurobiologically homogeneous patient groups may provide new momentum for treatment research. Finally, lessons learned from schizophrenia research may contribute to developing a comprehensive perspective on human experience and behavior that integrates methodologically distinct, but internally consistent, insights from humanities and neuroscience.

Perceived Conventionality in Co-speech Gestures Involves the Fronto-Temporal Language Network

Face-to-face communication is multimodal; it encompasses spoken words, facial expressions, gaze, and co-speech gestures. In contrast to linguistic symbols (e.g., spoken words or signs in sign language) relying on mostly explicit conventions, gestures vary in their degree of conventionality. Bodily signs may have a general accepted or conventionalized meaning (e.g., a head shake) or less so (e.g., self-grooming). We hypothesized that subjective perception of conventionality in co-speech gestures relies on the classical language network, i.e., the left hemispheric inferior frontal gyrus (IFG, Broca's area) and the posterior superior temporal gyrus (pSTG, Wernicke's area) and studied 36 subjects watching video-recorded story retellings during a behavioral and an functional magnetic resonance imaging (fMRI) experiment. It is well documented that neural correlates of such naturalistic videos emerge as intersubject covariance (ISC) in fMRI even without involving a stimulus (model-free analysis). The subjects attended either to perceived conventionality or to a control condition (any hand movements or gesture-speech relations). Such tasks modulate ISC in contributing neural structures and thus we studied ISC changes to task demands in language networks. Indeed, the conventionality task significantly increased covariance of the button press time series and neuronal synchronization in the left IFG over the comparison with other tasks. In the left IFG, synchronous activity was observed during the conventionality task only. In contrast, the left pSTG exhibited correlated activation patterns during all conditions with an increase in the conventionality task at the trend level only. Conceivably, the left IFG can be considered a core region for the processing of perceived conventionality in co-speech gestures similar to spoken language. In general, the interpretation of conventionalized signs may rely on neural mechanisms that engage during language comprehension.

Impaired error processing in late-phase psychosis: Four-year stability and relationships with negative symptoms

Error processing is impaired in psychosis, and numerous event-related potential studies have found reductions in the error-related negativity (ERN) and, more recently, the error positivity (Pe). The stability of reduced ERN/Pe in psychosis, however, is unknown. In a previous cross-sectional report, reduced ERN was associated with negative symptom severity and reduced Pe with a diagnosis of schizophrenia versus other psychosis. Here, we test the stability of impaired error processing over a four-year follow-up and relationships with subdimensions of negative symptoms. The ERN and Pe were recorded from individuals with psychotic disorders twice: 79 individuals were assessed 15years after first hospitalization, and 69 were assessed at 19years; 59 (26 with schizophrenia, 33 with other psychotic disorders) had data at both assessments. At 19years the Pe was blunted in schizophrenia. The ERN and Pe exhibited temporal stability over the four years (r=0.59 and 0.60, respectively). Reduced ERN and Pe correlated with the negative symptom subdimensions of inexpressivity and avolition, respectively, and not with psychotic or disorganized symptoms. Moreover, 15-year ERN predicted an increase in inexpressivity by year 19. No evidence was found for the reverse: negative symptoms did not predict change in ERN/Pe. Similar to non-clinical samples, the ERN and Pe show impressive four-year stability in late-phase psychosis. The ERN and Pe are promising neural measures for capturing individual differences in psychotic disorders, particularly with regard to negative symptomatology. They may prove to be useful clinically for forecasting illness course and as treatment targets.

Neural correlates of text-based emoticons: a preliminary fMRI study

INTRODUCTION

Like nonverbal cues in oral interactions, text-based emoticons, which are textual portrayals of a writer's facial expressions, are commonly used in electronic device-mediated communication. Little is known, however, about how text-based emoticons are processed in the human brain. With this study, we investigated whether the text-based emoticons are processed as face expressions using fMRI.

METHODS

During fMRI scan, subjects were asked to respond by pressing a button, indicating whether text-based emoticons represented positive or negative emotions. Voxel-wise analyses were performed to compare the responses and contrasted with emotional versus scrambled emoticons and among emoticons with different emotions. To explore processing strategies for text-based emoticons, brain activity in the bilateral occipital and fusiform face areas were compared.

RESULTS

In the voxel-wise analysis, both emotional and scrambled emoticons were processed mainly in the bilateral fusiform gyri, inferior division of lateral occipital cortex, inferior frontal gyri, dorsolateral prefrontal cortex (DLPFC), dorsal anterior cingulate cortex (dACC), and parietal cortex. In a percent signal change analysis, the right occipital and fusiform face areas showed significantly higher activation than left ones. In comparisons among emoticons, sad one showed significant BOLD signal decrease in the dACC, the left AIC, the bilateral thalamus, and the precuneus as compared with other conditions.

CONCLUSION

The results of this study imply that people recognize text-based emoticons as pictures representing face expressions. Even though text-based emoticons contain emotional meaning, they are not associated with the amygdala while previous studies using emotional stimuli documented amygdala activation.

Role of the first and second person perspective for control of behaviour: Understanding other people's facial expressions

Humans typically make probabilistic inferences about another person's affective state based on her/his bodily movements such as emotional facial expressions, emblematic gestures and whole body movements. Furthermore, humans deduce tentative predictions about the other person's intentions. Thus, the first person perspective of a subject is supplemented by the second person perspective involving theory of mind and empathy. Neuroimaging investigations have shown that the medial and lateral frontal cortex are critical nodes in the circuits underlying theory of mind, empathy, as well as intention of action. It is suggested that personal perspective taking in social interactions is paradigmatic for the capability of humans to generate probabilistic accounts of the outside world that underlie a person's control of behaviour.

Beat and metaphoric gestures are differentially associated with regional cerebellar and cortical volumes

Gestures represent an integral aspect of interpersonal communication, and they are closely linked with language and thought. Brain regions for language processing overlap with those for gesture processing. Two types of gesticulation, beat gestures and metaphoric gestures are particularly important for understanding the taxonomy of co-speech gestures. Here, we investigated gesture production during taped interviews with respect to regional brain volume. First, we were interested in whether beat gesture production is associated with similar regions as metaphoric gesture. Second, we investigated whether cortical regions associated with metaphoric gesture processing are linked to gesture production based on correlations with brain volumes. We found that beat gestures are uniquely related to regional volume in cerebellar regions previously implicated in discrete motor timing. We suggest that these gestures may be an artifact of the timing processes of the cerebellum that are important for the timing of vocalizations. Second, our findings indicate that brain volumes in regions of the left hemisphere previously implicated in metaphoric gesture processing are positively correlated with metaphoric gesture production. Together, this novel work extends our understanding of left hemisphere regions associated with gesture to indicate their importance in gesture production, and also suggests that beat gestures may be especially unique. This provides important insight into the taxonomy of co-speech gestures, and also further insight into the general role of the cerebellum in language.

Should I stay or should I go? Conceptual underpinnings of goal-directed actions

All actions, even the simplest like moving an arm to grasp a pen, are associated with energy costs. Thus all mobile organisms possess the ability to evaluate resources and select those behaviors that are most likely to lead to the greatest accrual of valuable items (reward) in the near or, especially in the case of humans, distant future. The evaluation process is performed at all possible stages of the series of decisions that lead to the building of a goal-directed action or to its suppression. This is because all animals have a limited amount of energy and resources; to survive and be able to reproduce they have to minimize the costs and maximize the outcomes of their actions. These computations are at the root of behavioral flexibility. Two executive functions play a major role in generating flexible behaviors: (i) the ability to predict future outcomes of goal-directed actions; and (ii) the ability to cancel them when they are unlikely to accomplish valuable results. These two processes operate continuously during the entire course of a movement: during its genesis, its planning and even its execution, so that the motor output can be modulated or suppressed at any time before its execution. In this review, functional interactions of the extended neural network subserving generation and inhibition of goal-directed movements will be outlined, leading to the intriguing hypothesis that the performance of actions and their suppression are not specified by independent sets of brain regions. Rather, it will be proposed that acting and stopping are functions emerging from specific interactions between largely overlapping brain regions, whose activity is intimately linked (directly or indirectly) to the evaluations of pros and cons of an action. Such mechanism would allow the brain to perform as a highly efficient and flexible system, as different functions could be computed exploiting the same components operating in different configurations.

Anterior and posterior subareas of the dorsolateral frontal cortex in socially relevant decisions based on masked affect expressions

Socially-relevant decisions are based on clearly recognizable but also not consciously accessible affective stimuli. We studied the role of the dorsolateral frontal cortex (DLFC) in decision-making on masked affect expressions using functional magnetic resonance imaging. Our paradigm permitted us to capture brain activity during a pre-decision phase when the subjects viewed emotional expressions below the threshold of subjective awareness, and during the decision phase, which was based on verbal descriptions as the choice criterion. Using meta-analytic connectivity modeling, we found that the preparatory phase of the decision was associated with activity in a right-posterior portion of the DLFC featuring co-activations in the left-inferior frontal cortex. During the subsequent decision a right-anterior and more dorsal portion of the DLFC became activated, exhibiting a different co-activation pattern. These results provide evidence for partially independent sub-regions within the DLFC, supporting the notion of dual associative processes in intuitive judgments.

Anterior and posterior subareas of the dorsolateral frontal cortex in socially relevant decisions based on masked affect expressions

Socially-relevant decisions are based on clearly recognizable but also not consciously accessible affective stimuli. We studied the role of the dorsolateral frontal cortex (DLFC) in decision-making on masked affect expressions using functional magnetic resonance imaging. Our paradigm permitted us to capture brain activity during a pre-decision phase when the subjects viewed emotional expressions below the threshold of subjective awareness, and during the decision phase, which was based on verbal descriptions as the choice criterion. Using meta-analytic connectivity modeling, we found that the preparatory phase of the decision was associated with activity in a right-posterior portion of the DLFC featuring co-activations in the left-inferior frontal cortex. During the subsequent decision a right-anterior and more dorsal portion of the DLFC became activated, exhibiting a different co-activation pattern. These results provide evidence for partially independent sub-regions within the DLFC, supporting the notion of dual associative processes in intuitive judgments.

Spatiotemporal dynamics of early cortical gesture processing

Gesture processing has been consistently shown to be associated with activation of the inferior parietal lobe (IPL); however, little is known about the integration of IPL activation into the temporal dynamics of early sensory areas. Using a temporally graded repetition suppression paradigm, we examined the activation and time course of brain areas involved in hand gesture processing. We recorded event-related potentials in response to stimulus pairs of static hand images forming gestures of the popular rock-paper-scissors game and estimated their neuronal generators. We identified two main components associated with adaptive patterns related to stimulus repetition. The N190 component elicited at temporo-parietal sites adapted to repetitions of the same gesture and was associated with right-hemispheric extrastriate body area activation. A later component at parieto-occipital sites demonstrated temporally graded adaptation effects for all gestures with a left-hemispheric dominance. Source localization revealed concurrent activations of the right extrastriate body area, fusiform gyri bilaterally, and the left IPL at about 250 ms. The adaptation pattern derived from the graded repetition suppression paradigm demonstrates the functional sensitivity of these sources to gesture processing. Given the literature on IPL contribution to imitation, action recognition, and action execution, IPL activation at about 250 ms may represent the access into specific cognitive routes for gesture processing and may thus be involved in integrating sensory information from cortical body areas into subsequent visuo-motor transformation processes.

The eye contact effect in request and emblematic hand gestures

Request and emblematic gestures, despite being both communicative gestures, do differ in terms of social valence. Indeed, only the former are used to initiate/maintain/terminate an actual interaction. If such a difference is at stake, a relevant social cue, i.e. eye contact, should have different impacts on the neuronal underpinnings of the two types of gesture. We measured blood oxygen level-dependent signals, using functional magnetic resonance imaging, while participants watched videos of an actor, either blindfolded or not, performing emblems, request gestures, or meaningless control movements. A left-lateralized network was more activated by both types of communicative gestures than by meaningless movements, regardless of the accessibility of the actor's eyes. Strikingly, when eye contact was taken into account as a factor, a right-lateralized network was more strongly activated by emblematic gestures performed by the non-blindfolded actor than by those performed by the blindfolded actor. Such modulation possibly reflects the integration of information conveyed by the eyes with the representation of emblems. Conversely, a wider right-lateralized network was more strongly activated by request gestures performed by the blindfolded than by those performed by the non-blindfolded actor. This probably reflects the effect of the conflict between the observed action and its associated contextual information, in which relevant social cues are missing.

Brain function overlaps when people observe emblems, speech, and grasping

A hand grasping a cup or gesturing "thumbs-up", while both manual actions, have different purposes and effects. Grasping directly affects the cup, whereas gesturing "thumbs-up" has an effect through an implied verbal (symbolic) meaning. Because grasping and emblematic gestures ("emblems") are both goal-oriented hand actions, we pursued the hypothesis that observing each should evoke similar activity in neural regions implicated in processing goal-oriented hand actions. However, because emblems express symbolic meaning, observing them should also evoke activity in regions implicated in interpreting meaning, which is most commonly expressed in language. Using fMRI to test this hypothesis, we had participants watch videos of an actor performing emblems, speaking utterances matched in meaning to the emblems, and grasping objects. Our results show that lateral temporal and inferior frontal regions respond to symbolic meaning, even when it is expressed by a single hand action. In particular, we found that left inferior frontal and right lateral temporal regions are strongly engaged when people observe either emblems or speech. In contrast, we also replicate and extend previous work that implicates parietal and premotor responses in observing goal-oriented hand actions. For hand actions, we found that bilateral parietal and premotor regions are strongly engaged when people observe either emblems or grasping. These findings thus characterize converging brain responses to shared features (e.g., symbolic or manual), despite their encoding and presentation in different stimulus modalities.

Social communication in young children with traumatic brain injury: relations with corpus callosum morphometry

The purpose of the present investigation was to characterize the relations of specific social communication behaviors, including joint attention, gestures, and verbalization, with surface area of midsagittal corpus callosum (CC) subregions in children who sustained traumatic brain injury (TBI) before 7 years of age. Participants sustained mild (n=10) or moderate-severe (n=26) noninflicted TBI. The mean age at injury was 33.6 months; mean age at MRI was 44.4 months. The CC was divided into seven subregions. Relative to young children with mild TBI, those with moderate-severe TBI had smaller surface area of the isthmus. A semi-structured sequence of social interactions between the child and an examiner was videotaped and coded for specific social initiation and response behaviors. Social responses were similar across severity groups. Even though the complexity of their language was similar, children with moderate-severe TBI used more gestures than those with mild TBI to initiate social overtures; this may indicate a developmental lag or deficit as the use of gestural communication typically diminishes after age 2. After controlling for age at scan and for total brain volume, the correlation of social interaction response and initiation scores with the midsagittal surface area of the CC regions was examined. For the total group, responding to a social overture using joint attention was significantly and positively correlated with surface area of all regions, except the rostrum. Initiating joint attention was specifically and negatively correlated with surface area of the anterior midbody. Use of gestures to initiate a social interaction correlated significantly and positively with surface area of the anterior and posterior midbody. Social response and initiation behaviors were selectively related to regional callosal surface areas in young children with TBI. Specific brainbehavior relations indicate early regional specialization of anterior and posterior CC for social communication.