Perception of social interactions for spatially scrambled biological motion

Evidence for a role of synchrony but not common fate in the perception of biological group movements

Extensive research has shown that observers are able to efficiently extract summary information from groups of people. However, little is known about the cues that determine whether multiple people are represented as a social group or as independent individuals. Initial research on this topic has primarily focused on the role of static cues. Here, we instead investigate the role of dynamic cues. In two experiments with male and female human participants, we use EEG frequency tagging to investigate the influence of two fundamental Gestalt principles - synchrony and common fate - on the grouping of biological movements. In Experiment 1, we find that brain responses coupled to four point-light figures walking together are enhanced when they move in sync vs. out of sync, but only when they are presented upright. In contrast, we found no effect of movement direction (i.e., common fate). In Experiment 2, we rule out that synchrony takes precedence over common fate by replicating the null effect of movement direction while keeping synchrony constant. These results suggest that synchrony plays an important role in the processing of biological group movements. In contrast, the role of common fate is less clear and will require further research.

Post-migration living difficulties and poor mental health associated with increased interpretation bias for threat

Previous research has found associations between mental health difficulties and interpretation biases, including heightened interpretation of threat from neutral or ambiguous stimuli. Building on this research, we explored associations between interpretation biases (positive and negative) and three constructs that have been linked to migrant experience: mental health symptoms (Global Severity Index [GSI]), Post-Migration Living Difficulties (PMLD), and Perceived Ethnic Discrimination Questionnaire (PEDQ). Two hundred thirty students who identified as first- (n = 94) or second-generation ethnic minority migrants (n = 68), and first-generation White migrants (n = 68) completed measures of GSI, PEDQ, and PMLD. They also performed an interpretation bias task using Point Light Walkers (PLW), dynamic stimuli with reduced visual input that are easily perceived as humans performing an action. Five categories of PLW were used: four that clearly depicted human forms undertaking positive, neutral, negative, or ambiguous actions, and a fifth that involved scrambled animations with no clear action or form. Participants were asked to imagine their interaction with the stimuli and rate their friendliness (positive interpretation bias) and aggressiveness (interpretation bias for threat). We found that the three groups differed on PEDQ and PMLD, with no significant differences in GSI, and the three measured were positively correlated. Poorer mental health and increased PMLD were associated with a heightened interpretation for threat of scrambled animations only. These findings have implications for understanding of the role of threat biases in mental health and the migrant experience.

Seeing social interactions

Seeing the interactions between other people is a critical part of our everyday visual experience, but recognizing the social interactions of others is often considered outside the scope of vision and grouped with higher-level social cognition like theory of mind. Recent work, however, has revealed that recognition of social interactions is efficient and automatic, is well modeled by bottom-up computational algorithms, and occurs in visually-selective regions of the brain. We review recent evidence from these three methodologies (behavioral, computational, and neural) that converge to suggest the core of social interaction perception is visual. We propose a computational framework for how this process is carried out in the brain and offer directions for future interdisciplinary investigations of social perception.

When the timing is right: The link between temporal coupling in dyadic interactions and emotion recognition

Affective states can be understood as dynamic interpersonal processes developing over time and space. When we observe emotional interactions performed by other individuals, our visual system anticipates how the action will unfold. Thus, it has been proposed that the process of emotion perception is not only a simulative but also a predictive process - a phenomenon described as interpersonal predictive coding. The present study investigated whether the recognition of emotions from dyadic interactions depends on a fixed spatiotemporal coupling of the agents. We used an emotion recognition task to manipulate the actions of two interacting point-light figures by implementing different temporal offsets that delayed the onset of one of the agent's actions (+0 ms, +500 ms, +1000 ms or + 2000 ms). Participants had to determine both the subjective valence and the emotion category (happiness, anger, sadness, affection) of the interaction. Results showed that temporal decoupling had a critical effect on both emotion recognition and the subjective impression of valence intensity: Both measures decreased with increasing temporal offset. However, these effects were dependent on which emotion was displayed. Whereas affection and anger sequences were impacted by the temporal manipulation, happiness and sadness were not. To further investigate these effects, we conducted post-hoc exploratory analyses of interpersonal movement parameters. Our findings complement and extend previous evidence by showing that the complex, noncoincidental coordination of actions within dyadic interactions results in a meaningful movement pattern and might serve as a fundamental factor in both detecting and understanding complex actions during human interaction.

Decoding spatiotemporal features of emotional body language in social interactions

How are emotions perceived through human body language in social interactions? This study used point-light displays of human interactions portraying emotional scenes (1) to examine quantitative intrapersonal kinematic and postural body configurations, (2) to calculate interaction-specific parameters of these interactions, and (3) to analyze how far both contribute to the perception of an emotion category (i.e. anger, sadness, happiness or affection) as well as to the perception of emotional valence. By using ANOVA and classification trees, we investigated emotion-specific differences in the calculated parameters. We further applied representational similarity analyses to determine how perceptual ratings relate to intra- and interpersonal features of the observed scene. Results showed that within an interaction, intrapersonal kinematic cues corresponded to emotion category ratings, whereas postural cues reflected valence ratings. Perception of emotion category was also driven by interpersonal orientation, proxemics, the time spent in the personal space of the counterpart, and the motion–energy balance between interacting people. Furthermore, motion–energy balance and orientation relate to valence ratings. Thus, features of emotional body language are connected with the emotional content of an observed scene and people make use of the observed emotionally expressive body language and interpersonal coordination to infer emotional content of interactions.

Is the social chunking of agent actions in working memory resource-demanding?

Retaining social interactions in working memory (WM) for further social activities is vital for a successful social life. Researchers have noted a social chunking phenomenon in WM: WM involuntarily uses the social interaction cues embedded in the individual actions and chunks them as one unit. Our study is the first to examine whether the social chunking in WM is an automatic process, by asking whether social chunking of agent actions in WM is resource-demanding, a key hallmark of automaticity. We achieved this by probing whether retaining agent interactions in WM as a chunk required more attention than retaining actions without interaction. We employed a WM change-detection task with actions containing social interaction cues as memory stimuli, and required participants only memorizing individual actions. As domain-general attention and object-based attention are suggested playing a key role in retaining chunks in WM, a secondary task was inserted in the WM maintenance phase to consume these two types of attention. We reestablished the fact that the social chunking in WM required no voluntary control (Experiments 1 and 2). Critically, we demonstrated substantial evidence that social chunking in WM did not require extra domain-general attention (Experiment 1) or object-based attention (Experiment 2). These findings imply that the social chunking of agent actions in WM is not resource-demanding, supporting an automatic view of social chunking in WM.

A Dynamical Generative Model of Social Interactions

The ability to make accurate social inferences makes humans able to navigate and act in their social environment effortlessly. Converging evidence shows that motion is one of the most informative cues in shaping the perception of social interactions. However, the scarcity of parameterized generative models for the generation of highly-controlled stimuli has slowed down both the identification of the most critical motion features and the understanding of the computational mechanisms underlying their extraction and processing from rich visual inputs. In this work, we introduce a novel generative model for the automatic generation of an arbitrarily large number of videos of socially interacting agents for comprehensive studies of social perception. The proposed framework, validated with three psychophysical experiments, allows generating as many as 15 distinct interaction classes. The model builds on classical dynamical system models of biological navigation and is able to generate visual stimuli that are parametrically controlled and representative of a heterogeneous set of social interaction classes. The proposed method represents thus an important tool for experiments aimed at unveiling the computational mechanisms mediating the perception of social interactions. The ability to generate highly-controlled stimuli makes the model valuable not only to conduct behavioral and neuroimaging studies, but also to develop and validate neural models of social inference, and machine vision systems for the automatic recognition of social interactions. In fact, contrasting human and model responses to a heterogeneous set of highly-controlled stimuli can help to identify critical computational steps in the processing of social interaction stimuli.

Review and Challenges of Technologies for Real-Time Human Behavior Monitoring

A person's behavior significantly influences their health and well-being. It also contributes to the social environment in which humans interact, with cascading impacts to the health and behaviors of others. During social interactions, our understanding and awareness of vital nonverbal messages expressing beliefs, emotions, and intentions can be obstructed by a variety of factors including greatly flawed self-awareness. For these reasons, human behavior is a very important topic to study using the most advanced technology. Moreover, technology offers a breakthrough opportunity to improve people's social awareness and self-awareness through machine-enhanced recognition and interpretation of human behaviors. This paper reviews (1) the social psychology theories that have established the framework to study human behaviors and their manifestations during social interactions and (2) the technologies that have contributed to the monitoring of human behaviors. State-of-the-art in sensors, signal features, and computational models are categorized, summarized, and evaluated from a comprehensive transdisciplinary perspective. This review focuses on assessing technologies most suitable for real-time monitoring while highlighting their challenges and opportunities in near-future applications. Although social behavior monitoring has been highly reported in psychology and engineering literature, this paper uniquely aims to serve as a disciplinary convergence bridge and a guide for engineers capable of bringing new technologies to bear against the current challenges in real-time human behavior monitoring.

Exploring biological motion perception in two-stream convolutional neural networks

Visual recognition of biological motion recruits form and motion processes supported by both dorsal and ventral pathways. This neural architecture inspired the two-stream convolutional neural network (CNN) model, which includes a spatial CNN to process appearance information in a sequence of image frames, a temporal CNN to process optical flow information, and a fusion network to integrate the features extracted by the two CNNs and make final decisions about action recognition. In five simulations, we compared the CNN model's performance with classical findings in biological motion perception. The CNNs trained with raw RGB action videos showed weak performance in recognizing point-light actions. Additional transfer training with actions shown in other display formats (e.g., skeletal) was necessary for CNNs to recognize point-light actions. The CNN models exhibited largely viewpoint-dependent recognition of actions, with a limited ability to generalize to viewpoints close to the training views. The CNNs predicted the inversion effect in the presence of global body configuration, but failed to predict the inversion effect driven solely by local motion signals. The CNNs provided a qualitative account of some behavioral results observed in human biological motion perception for fine discrimination tasks with noisy inputs, such as point-light actions with disrupted local motion signals, and walking actions with temporally misaligned motion cues. However, these successes are limited by the CNNs' lack of adaptive integration for form and motion processes, and failure to incorporate specialized mechanisms (e.g., a life detector) as well as top-down influences on biological motion perception.

Understanding the visual perception of awkward body movements: How interactions go awry

Dyadic interactions can sometimes elicit a disconcerting response from viewers, generating a sense of "awkwardness." Despite the ubiquity of awkward social interactions in daily life, it remains unknown what visual cues signal the oddity of human interactions and yield the subjective impression of awkwardness. In the present experiments, we focused on a range of greeting behaviors (handshake, fist bump, high five) to examine both the inherent objectivity and impact of contextual and kinematic information in the social evaluation of awkwardness. In Experiment 1, participants were asked to discriminate whether greeting behaviors presented in raw videos were awkward or natural, and if judged as awkward, participants provided verbal descriptions regarding the awkward greeting behaviors. Participants showed consensus in judging awkwardness from raw videos, with a high proportion of congruent responses across a range of awkward greeting behaviors. We also found that people used social-related and motor-related words in their descriptions for awkward interactions. Experiment 2 employed advanced computer vision techniques to present the same greeting behaviors in three different display types. All display types preserved kinematic information, but varied contextual information: (1) patch displays presented blurred scenes composed of patches; (2) body displays presented human body figures on a black background; and (3) skeleton displays presented skeletal figures of moving bodies. Participants rated the degree of awkwardness of greeting behaviors. Across display types, participants consistently discriminated awkward and natural greetings, indicating that the kinematics of body movements plays an important role in guiding awkwardness judgments. Multidimensional scaling analysis based on the similarity of awkwardness ratings revealed two primary cues: motor coordination (which accounted for most of the variability in awkwardness judgments) and social coordination. We conclude that the perception of awkwardness, while primarily inferred on the basis of kinematic information, is additionally affected by the perceived social coordination underlying human greeting behaviors.

The Impact of Autistic Traits on Self-Recognition of Body Movements

Despite the sparse visual information and paucity of self-identifying cues provided by point-light stimuli, as well as a dearth of experience in seeing our own-body movements, people can identify themselves solely based on the kinematics of body movements. The present study found converging evidence of this remarkable ability using a broad range of actions with whole-body movements. In addition, we found that individuals with a high degree of autistic traits showed worse performance in identifying own-body movements, particularly for simple actions. A Bayesian analysis showed that action complexity modulates the relationship between autistic traits and self-recognition performance. These findings reveal the impact of autistic traits on the ability to represent and recognize own-body movements.

Inferring power and dominance from dyadic nonverbal interactions in autism spectrum disorder

Research studies to date have revealed conflicting results with respect to the processing of nonverbal cues from social interactions in autism spectrum disorder (ASD). Therefore, the aim of the present study was to investigate the contribution of two important factors for the perception of dyadic social interactions, namely (a) the movement contingency and (b) the spatial context. To this end, 26 adult participants with ASD and 26 age-, sex-, and IQ-matched typically developed control participants observed animations presenting nonverbal interactions between two human virtual characters enacting power relationships. We manipulated (a) movement contingency by exchanging one of the two original agents with an agent from another dyad and (b) spatial context by changing agents' spatial orientation to a back-to-back position. Participants were asked to rate dominance and submissiveness of these agents. Results showed that the movement contingency manipulation affected accuracy and consistency of power perception and that the spatial context manipulation slowed down reaction times comparably in both groups. With regard to group differences, individuals with ASD were found to judge power relationships slower compared to control participants, potentially suggesting a more explicit processing style in ASD. Furthermore, the spatial context manipulation slowed down the reaction times more in the contingent compared to the non-contingent conditions only in the ASD group. These findings contribute to the ongoing debate whether individuals with ASD have difficulties in understanding nonverbal cues in a dyadic context by suggesting that they do so in more subtle ways than previously investigated. Autism Res 2019, 12: 505-516 © 2019 International Society for Autism Research, Wiley Periodicals, Inc. LAY SUMMARY: This study shows that the ability and speed of judging who is dominant in a social interaction depends on two factors: (a) whether their movements are matched and (b) whether they are facing each other or not. This is similarly the case for participants with and without autism. Interestingly, however, individuals with autism seem to judge generally slower, suggesting a more explicit processing style. The two factors seem to interact, suggesting that nonverbal processing difficulties are subtler than previously thought.

Intact perception of coherent motion, dynamic rigid form, and biological motion in chronic schizophrenia

BACKGROUND

Prior studies have documented biological motion perception deficits in schizophrenia, but it remains unclear whether the impairments arise from poor social cognition, perceptual organization, basic motion processing, or sustained attention/motivation. To address the issue, we had 24 chronic schizophrenia patients and 27 healthy controls perform three tasks: coherent motion, where subjects indicated whether a cloud of dots drifted leftward or rightward; dynamic rigid form, where subjects determined the tilt direction of a translating, point-light rectangle; and biological motion, where subjects judged whether a human point-light figure walked leftward or rightward. Task difficulty was staircase controlled and depended on the directional variability of the background dot motion. Catch trials were added to verify task attentiveness and engagement.

RESULTS

Patients and controls demonstrated similar performance thresholds and near-ceiling catch trial accuracy for each task (uncorrected ps > 0.1; ds < 0.35). In all but the coherent motion task, higher IQ correlated with better performance (ps < 0.001).

CONCLUSION

Schizophrenia patients have intact perception of motion coherence, dynamic rigid form, and biological motion at least for our sample and set-up. We speculate that previously documented biological motion perception deficits arose from task or stimulus differences or from group differences in IQ, attention, or motivation.

PLAViMoP: How to standardize and simplify the use of point-light displays

The study of biological point-light displays (PLDs) has fascinated researchers for more than 40 years. However, the mechanisms underlying PLD perception remain unclear, partly due to difficulties with precisely controlling and transforming PLD sequences. Furthermore, little agreement exists regarding how transformations are performed. This article introduces a new free-access program called PLAViMoP (Point-Light Display Visualization and Modification Platform) and presents the algorithms for PLD transformations actually included in the software. PLAViMoP fulfills two objectives. First, it standardizes and makes clear many classical spatial and kinematic transformations described in the PLD literature. Furthermore, given its optimized interface, PLAViMOP makes these transformations easy and fast to achieve. Overall, PLAViMoP could directly help scientists avoid technical difficulties and make possible the use of PLDs for nonacademic applications.

Causal Action: A Fundamental Constraint on Perception and Inference About Body Movements

The human body navigates the environment via locomotory movements that leverage gravity and limb biomechanics to propel the body in a particular direction. This process creates a causal link between limb movements and whole-body translation. However, it is unknown whether humans use this causal relation as a constraint in perception and inference with body movements. In the present study, participants rated actions of other individuals as more natural when limb movements (as a cause) occurred before body displacements (as an effect) than when limb movements temporally lagged behind body displacements. This causal expectation for human body movements not only affected perceptual impressions regarding the naturalness of observed actions but also guided the interpretation of motion cues within a more generalized causal context. We interpret these results within a framework of causality as evidence that the constraint of causal action plays an important role in perception and inference with body movements.

EEG frequency tagging dissociates between neural processing of motion synchrony and human quality of multiple point-light dancers

Do we perceive a group of dancers moving in synchrony differently from a group of drones flying in-sync? The brain has dedicated networks for perception of coherent motion and interacting human bodies. However, it is unclear to what extent the underlying neural mechanisms overlap. Here we delineate these mechanisms by independently manipulating the degree of motion synchrony and the humanoid quality of multiple point-light displays (PLDs). Four PLDs moving within a group were changing contrast in cycles of fixed frequencies, which permits the identification of the neural processes that are tagged by these frequencies. In the frequency spectrum of the steady-state EEG we found two emergent frequency components, which signified distinct levels of interactions between PLDs. The first component was associated with motion synchrony, the second with the human quality of the moving items. These findings indicate that visual processing of synchronously moving dancers involves two distinct neural mechanisms: one for the perception of a group of items moving in synchrony and one for the perception of a group of moving items with human quality. We propose that these mechanisms underlie high-level perception of social interactions.

Communicative interactions in point-light displays: Choosing among multiple response alternatives

Vision scientists are increasingly relying on the point-light technique as a way to investigate the perception of human motion. Unfortunately, the lack of standardized stimulus sets has so far limited the use of this technique for studying social interaction. Here, we describe a new tool to study the interaction between two agents starting from point-light displays: the Communicative Interaction Database - 5AFC format (CID-5). The CID-5 consists of 14 communicative and seven non-communicative individual actions performed by two agents. Stimuli were constructed by combining motion capture techniques and 3-D animation software to provide precise control over the computer-generated actions. For each action stimulus, we provide coordinate files and movie files depicting the action as seen from four different perspectives. Furthermore, the archive contains a text file with a list of five alternative action descriptions to construct forced-choice paradigms. In order to validate the CID-5 format, we provide normative data collected to assess action identification within a 5AFC tasks. The CID-5 archive is freely downloadable from http://bsb-lab.org/research/ and from the supplementary materials of this article.

Social Interactions Receive Priority to Conscious Perception

Humans are social animals, constantly engaged with other people. The importance of social thought and action is hard to overstate. However, is social information so important that it actually determines which stimuli are promoted to conscious experience and which stimuli are suppressed as invisible? To address this question, we used a binocular rivalry paradigm, in which the two eyes receive different action stimuli. In two experiments we measured the conscious percept of rival actions and found that actions engaged in social interactions are granted preferential access to visual awareness over non-interactive actions. Lastly, an attentional task that presumably engaged the mentalizing system enhanced the priority assigned to social interactions in reaching conscious perception. We also found a positive correlation between human identification of interactive activity and the promotion of socially-relevant information to visual awareness. The present findings suggest that the visual system amplifies socially-relevant sensory information and actively promotes it to consciousness, thereby facilitating inferences about social interactions.

Neural adaptation in pSTS correlates with perceptual aftereffects to biological motion and with autistic traits

The adaptive nature of biological motion perception has been documented in behavioral studies, with research showing that prolonged viewing of an action can bias judgments of subsequent actions towards the opposite of its attributes. However, the neural mechanisms underlying action adaptation aftereffects remain unknown. We examined adaptation-induced changes in brain responses to an ambiguous action after adapting to walking or running actions within two bilateral regions of interest: 1) human middle temporal area (hMT+), a lower-level motion-sensitive region of cortex, and 2) posterior superior temporal sulcus (pSTS), a higher-level action-selective area. We found a significant correlation between neural adaptation strength in right pSTS and perceptual aftereffects to biological motion measured behaviorally, but not in hMT+. The magnitude of neural adaptation in right pSTS was also strongly correlated with individual differences in the degree of autistic traits. Participants with more autistic traits exhibited less adaptation-induced modulations of brain responses in right pSTS and correspondingly weaker perceptual aftereffects. These results suggest a direct link between perceptual aftereffects and adaptation of neural populations in right pSTS after prolonged viewing of a biological motion stimulus, and highlight the potential importance of this brain region for understanding differences in social-cognitive processing along the autistic spectrum.

A comparison of form processing involved in the perception of biological and nonbiological movements

Although there is evidence for specialization in the human brain for processing biological motion per se, few studies have directly examined the specialization of form processing in biological motion perception. The current study was designed to systematically compare form processing in perception of biological (human walkers) to nonbiological (rotating squares) stimuli. Dynamic form-based stimuli were constructed with conflicting form cues (position and orientation), such that the objects were perceived to be moving ambiguously in two directions at once. In Experiment 1, we used the classification image technique to examine how local form cues are integrated across space and time in a bottom-up manner. By comparing with a Bayesian observer model that embodies generic principles of form analysis (e.g., template matching) and integrates form information according to cue reliability, we found that human observers employ domain-general processes to recognize both human actions and nonbiological object movements. Experiments 2 and 3 found differential top-down effects of spatial context on perception of biological and nonbiological forms. When a background does not involve social information, observers are biased to perceive foreground object movements in the direction opposite to surrounding motion. However, when a background involves social cues, such as a crowd of similar objects, perception is biased toward the same direction as the crowd for biological walking stimuli, but not for rotating nonbiological stimuli. The model provided an accurate account of top-down modulations by adjusting the prior probabilities associated with the internal templates, demonstrating the power and flexibility of the Bayesian approach for visual form perception.