Visual bodily signals as core devices for coordinating minds in interaction

Unraveling dyadic psycho-physiology of social presence between strangers during an audio drama - a signal-analysis approach

A mere co-presence of an unfamiliar person may modulate an individual's attentive engagement with specific events or situations to a significant degree. To understand better how such social presence affects experiences, we recorded a set of parallel multimodal facial and psychophysiological data with subjects (N = 36) who listened to dramatic audio scenes alone or when facing an unfamiliar person. Both a selection of 6 s affective sound clips (IADS-2) followed by a 27 min soundtrack extracted from a Finnish episode film depicted familiar and often intense social situations familiar from the everyday world. Considering the systemic complexity of both the chosen naturalistic stimuli and expected variations in the experimental social situation, we applied a novel combination of signal analysis methods using inter-subject correlation (ISC) analysis, Representational Similarity Analysis (RSA) and Recurrence Quantification Analysis (RQA) followed by gradient boosting classification. We report our findings concerning three facial signals, gaze, eyebrow and smile that can be linked to socially motivated facial movements. We found that ISC values of pairs, whether calculated on true pairs or any two individuals who had a partner, were lower than the group with single individuals. Thus, audio stimuli induced more unique responses in those subjects who were listening to it in the presence of another person, while individual listeners tended to yield a more uniform response as it was driven by dramatized audio stimulus alone. Furthermore, our classifiers models trained using recurrence properties of gaze, eyebrows and smile signals demonstrated distinctive differences in the recurrence dynamics of signals from paired subjects and revealed the impact of individual differences on the latter. We showed that the presence of an unfamiliar co-listener that modifies social dynamics of dyadic listening tasks can be detected reliably from visible facial modalities. By applying our analysis framework to a broader range of psycho-physiological data, together with annotations of the content, and subjective reports of participants, we expected more detailed dyadic dependencies to be revealed. Our work contributes towards modeling and predicting human social behaviors to specific types of audio-visually mediated, virtual, and live social situations.

Human-like interactions prompt people to take a robot's perspective

The increasing need for human-robot interaction requires not only robots to understand how humans think, but also humans to understand robots. Interestingly, little attention has been given to how humans interpret robots' behaviors. In this study, we adopted a social mental rotation task and investigated whether socially engaging behaviors could influence how people take a robot's perspectives. In a real lab, two android robots with neutral appearance sat opposite each other by a table with conflicting perspectives. Before the participant started the experiment, one of the robots behaved more interactively than the other by showing more socially engaging behaviors. Then the participant was required to identify rotated normal or mirrored digits presented inbetween the two robots. Results revealed a significant interactive effect between the digits type (normal; mirrored) and robot type (interactive; noninteractive). When digits were oriented to the interactive robot, we found a larger RT difference between normal and mirrored digits. In general, these findings suggested that robots' interactive behaviors could influence how people spontaneously consider the robot's perspective. Future studies may further consider how interactive behaviors can shape human-robot relationships and facilitate human-robot interaction.

Mouth and facial informativeness norms for 2276 English words

Mouth and facial movements are part and parcel of face-to-face communication. The primary way of assessing their role in speech perception has been by manipulating their presence (e.g., by blurring the area of a speaker's lips) or by looking at how informative different mouth patterns are for the corresponding phonemes (or visemes; e.g., /b/ is visually more salient than /g/). However, moving beyond informativeness of single phonemes is challenging due to coarticulation and language variations (to name just a few factors). Here, we present mouth and facial informativeness (MaFI) for words, i.e., how visually informative words are based on their corresponding mouth and facial movements. MaFI was quantified for 2276 English words, varying in length, frequency, and age of acquisition, using phonological distance between a word and participants' speechreading guesses. The results showed that MaFI norms capture well the dynamic nature of mouth and facial movements per word, with words containing phonemes with roundness and frontness features, as well as visemes characterized by lower lip tuck, lip rounding, and lip closure being visually more informative. We also showed that the more of these features there are in a word, the more informative it is based on mouth and facial movements. Finally, we demonstrated that the MaFI norms generalize across different variants of English language. The norms are freely accessible via Open Science Framework ( https://osf.io/mna8j/ ) and can benefit any language researcher using audiovisual stimuli (e.g., to control for the effect of speech-linked mouth and facial movements).

Face2face: advancing the science of social interaction

Face-to-face interaction is core to human sociality and its evolution, and provides the environment in which most of human communication occurs. Research into the full complexities that define face-to-face interaction requires a multi-disciplinary, multi-level approach, illuminating from different perspectives how we and other species interact. This special issue showcases a wide range of approaches, bringing together detailed studies of naturalistic social-interactional behaviour with larger scale analyses for generalization, and investigations of socially contextualized cognitive and neural processes that underpin the behaviour we observe. We suggest that this integrative approach will allow us to propel forwards the science of face-to-face interaction by leading us to new paradigms and novel, more ecologically grounded and comprehensive insights into how we interact with one another and with artificial agents, how differences in psychological profiles might affect interaction, and how the capacity to socially interact develops and has evolved in the human and other species. This theme issue makes a first step into this direction, with the aim to break down disciplinary boundaries and emphasizing the value of illuminating the many facets of face-to-face interaction. This article is part of a discussion meeting issue 'Face2face: advancing the science of social interaction'.

Multimodal processing in face-to-face interactions: A bridging link between psycholinguistics and sensory neuroscience

In face-to-face communication, humans are faced with multiple layers of discontinuous multimodal signals, such as head, face, hand gestures, speech and non-speech sounds, which need to be interpreted as coherent and unified communicative actions. This implies a fundamental computational challenge: optimally binding only signals belonging to the same communicative action while segregating signals that are not connected by the communicative content. How do we achieve such an extraordinary feat, reliably, and efficiently? To address this question, we need to further move the study of human communication beyond speech-centred perspectives and promote a multimodal approach combined with interdisciplinary cooperation. Accordingly, we seek to reconcile two explanatory frameworks recently proposed in psycholinguistics and sensory neuroscience into a neurocognitive model of multimodal face-to-face communication. First, we introduce a psycholinguistic framework that characterises face-to-face communication at three parallel processing levels: multiplex signals, multimodal gestalts and multilevel predictions. Second, we consider the recent proposal of a lateral neural visual pathway specifically dedicated to the dynamic aspects of social perception and reconceive it from a multimodal perspective ("lateral processing pathway"). Third, we reconcile the two frameworks into a neurocognitive model that proposes how multiplex signals, multimodal gestalts, and multilevel predictions may be implemented along the lateral processing pathway. Finally, we advocate a multimodal and multidisciplinary research approach, combining state-of-the-art imaging techniques, computational modelling and artificial intelligence for future empirical testing of our model.

Verbal Communication with the Patient Is Not Enough: The Six Languages of the Sick

Evidence shows that verbal communication is just one of the ways patients indicate their wishes. For a sufficiently careful communication, we should also grasp other five unusual though evident languages: (a) body language, (b) the way patients manage their environment, (c) unconscious language, (d) lab-evidenced language, and (e) the way they master technology. So, we have six languages that should be intertwined to understand the real language of the sick. Grasping these languages helps health professionals frame the patient’s mood, their level of suffering or mental growth, and understand what words alone cannot express. Words cannot express completely what a patient senses: for subjection, shyness, because some patients are still non-verbal or because verbal communication is just a useful way of freezing concept but has not the same fluidity and liberty of the other above-described languages. It is mandatory for caregivers to wonder how many of these languages they are actually decrypting during an interview with the patient. On the other hand, caregivers unconsciously communicate much through two unexpected languages: the architectural language and the language of medical procedures. The way they welcome or obstruct the patient, their hesitations across a treatment, or in showing a serene collegiality are forms of subtle communication. A paradigmatic scenario where all these languages should be implemented is the “informed consent” process, which should be turned into a “shared therapeutic pathway”, summing up all the communicative modes illustrated in the text.

Showing and giving: from incipient to conventional forms

Understanding humans' motivation and capacity for social interaction requires understanding communicative gestures. Gestures are one of the earliest means that infants employ to communicate with others, and showing and giving are among the earliest-emerging gestures. However, there are limited data on the processes that lead up to the emergence of conventional showing and giving gestures. This study aimed to provide such data. Twenty-five infants were assessed longitudinally at monthly intervals from 6 to 10 months of age using a variety of methods (elicitation procedures, free play observations and maternal interviews), as well as via questionnaires conducted at 11-12 months. A particular focus was on pre-conventional, incipient gestures, behaviours that involved some components of conventional gestures, but lacked other important components. We present observational evidence that at least some of these behaviours (observed as early as 7 months of age) were communicative and make the case for how conventional showing and giving may emerge gradually in the context of social interactions. We also discuss the influence of maternal interpretations of these early behaviours on their development. Overall, the study seeks to draw attention to the importance of understanding the cognitive, motor and interactional processes that lead to the emergence of infants' earliest communicative gestures. This article is part of the theme issue 'Revisiting the human 'interaction engine': comparative approaches to social action coordination'.

Coordinating social action: a primer for the cross-species investigation of communicative repair

Human joint action is inherently cooperative, manifested in the collaborative efforts of participants to minimize communicative trouble through interactive repair. Although interactive repair requires sophisticated cognitive abilities, it can be dissected into basic building blocks shared with non-human animal species. A review of the primate literature shows that interactionally contingent signal sequences are at least common among species of non-human great apes, suggesting a gradual evolution of repair. To pioneer a cross-species assessment of repair this paper aims at (i) identifying necessary precursors of human interactive repair; (ii) proposing a coding framework for its comparative study in humans and non-human species; and (iii) using this framework to analyse examples of interactions of humans (adults/children) and non-human great apes. We hope this paper will serve as a primer for cross-species comparisons of communicative breakdowns and how they are repaired. This article is part of the theme issue 'Revisiting the human 'interaction engine': comparative approaches to social action coordination'.

Revisiting the human ‘interaction engine': comparative approaches to social action coordination

The evolution of language was likely facilitated by a special predisposition for social interaction, involving a set of communicative and cognitive skills summarized as the ‘interaction engine'. This assemblage seems to emerge early in development, to be found universally across cultures, and to enable participation in sophisticated joint action through the addition of spoken language. Yet, new evidence on social action coordination and communication in nonhuman primates warrants an update of the interaction engine hypothesis, particularly with respect to the evolutionary origins of its specific ingredients. However, one enduring problem for comparative research results from a conceptual gulf between disciplines, rendering it difficult to test concepts derived from human interaction research in nonhuman animals. The goal of this theme issue is to make such concepts accessible for comparative research, to promote a fruitful interdisciplinary debate on social action coordination as a new arena of research, and to enable mutual fertilization between human and nonhuman interaction research. In consequence, we here consider relevant theoretical and empirical research within and beyond this theme issue to revisit the interaction engine's shared, convergently derived and uniquely derived ingredients preceding (or perhaps in the last case, succeeding) human language.

This article is part of the theme issue ‘Revisiting the human ‘interaction engine’: comparative approaches to social action coordination’.

The interaction engine: cuteness selection and the evolution of the interactional base for language

The deep structural diversity of languages suggests that our language capacities are not based on any single template but rather on an underlying ability and motivation for infants to acquire a culturally transmitted system. The hypothesis is that this ability has an interactional base that has discernable precursors in other primates. In this paper, I explore a specific evolutionary route for the most puzzling aspect of this interactional base in humans, namely the development of an empathetic intentional stance. The route involves a generalization of mother-infant interaction patterns to all adults via a process (cuteness selection) analogous to, but distinct from, RA Fisher's runaway sexual selection. This provides a cornerstone for the carrying capacity for language. This article is part of the theme issue 'Revisiting the human 'interaction engine': comparative approaches to social action coordination'.