A motion capture library for the study of identity, gender, and emotion perception from biological motion

EMOKINE: A software package and computational framework for scaling up the creation of highly controlled emotional full-body movement datasets

EMOKINE is a software package and dataset creation suite for emotional full-body movement research in experimental psychology, affective neuroscience, and computer vision. A computational framework, comprehensive instructions, a pilot dataset, observer ratings, and kinematic feature extraction code are provided to facilitate future dataset creations at scale. In addition, the EMOKINE framework outlines how complex sequences of movements may advance emotion research. Traditionally, often emotional-'action'-based stimuli are used in such research, like hand-waving or walking motions. Here instead, a pilot dataset is provided with short dance choreographies, repeated several times by a dancer who expressed different emotional intentions at each repetition: anger, contentment, fear, joy, neutrality, and sadness. The dataset was simultaneously filmed professionally, and recorded using XSENS® motion capture technology (17 sensors, 240 frames/second). Thirty-two statistics from 12 kinematic features were extracted offline, for the first time in one single dataset: speed, acceleration, angular speed, angular acceleration, limb contraction, distance to center of mass, quantity of motion, dimensionless jerk (integral), head angle (with regards to vertical axis and to back), and space (convex hull 2D and 3D). Average, median absolute deviation (MAD), and maximum value were computed as applicable. The EMOKINE software is appliable to other motion-capture systems and is openly available on the Zenodo Repository. Releases on GitHub include: (i) the code to extract the 32 statistics, (ii) a rigging plugin for Python for MVNX file-conversion to Blender format (MVNX=output file XSENS® system), and (iii) a Python-script-powered custom software to assist with blurring faces; latter two under GPLv3 licenses.

SmartDetector: Automatic and vision-based approach to point-light display generation for human action perception

Over the past four decades, point-light displays (PLD) have been integrated into psychology and psychophysics, providing a valuable means to probe human perceptual skills. Leveraging the inherent kinematic information and controllable display parameters, researchers have utilized this technique to examine the mechanisms involved in learning and rehabilitation. However, classical PLD generation methods (e.g., motion capture) are difficult to apply for behavior analysis in real-world situations, such as patient care or sports activities. Therefore, there is a demand for automated and affordable tools that enable efficient and real-world-compatible generation of PLDs for psychological research. In this paper, we propose SmartDetector, a new artificial intelligence (AI)-based tool for automatic PLD creation from RGB videos. To evaluate humans' perceptual skills for processing PLD building with SmartDetector, 126 participants were randomly assigned to recognition, discrimination, or detection tasks. Results demonstrated that, irrespective of the task, PLDs generated by SmartDetector exhibited commendable perceptual performance in terms of accuracy and response times compared to literature findings. Moreover, to enhance usability and broaden accessibility, we developed an intuitive web interface for our method, making it available to a wider audience. The resulting application is available at https://plavimop.prd.fr/index.php/en/automatic-creation-pld . SmartDetector offers interesting possibilities for using PLD in research and makes the use of PLD more accessible for nonacademic applications.

The SocialVidStim: a video database of positive and negative social evaluation stimuli for use in social cognitive neuroscience paradigms

This paper describes the SocialVidStim-a database of video stimuli available to the scientific community depicting positive and negative social evaluative and neutral statements. The SocialVidStim comprises 53 diverse individuals reflecting the demographic makeup of the USA, ranging from 9 to 41 years old, saying 20-60 positive and 20-60 negative social evaluative statements (e.g. 'You are a very trustworthy/annoying person'), and 20-60 neutral statements (e.g. 'The sky is blue'), totaling 5793 videos post-production. The SocialVidStim are designed for use in behavioral and functional magetic resonance imaging paradigms, across developmental stages, in diverse populations. This study describes stimuli development and reports initial validity and reliability data on a subset videos (N = 1890) depicting individuals aged 18-41 years. Raters perceive videos as expected: positive videos elicit positively valenced ratings, negative videos elicit negatively valenced ratings and neutral videos are rated as neutral. Test-retest reliability data demonstrate intraclass correlations in the good-to-excellent range for negative and positive videos and the moderate range for neutral videos. We also report small effects on valence and arousal that should be considered during stimuli selection, including match between rater and actor sex and actor believability. The SocialVidStim is a resource for researchers and we offer suggestions for using the SocialVidStim in future research.

Hysteresis reveals a happiness bias effect in dynamic emotion recognition from ambiguous biological motion

Considering the nonlinear dynamic nature of emotion recognition, it is believed to be strongly dependent on temporal context. This can be investigated by resorting to the phenomenon of hysteresis, which features a form of serial dependence, entailed by continuous temporal stimulus trajectories. Under positive hysteresis, the percept remains stable in visual memory (persistence) while in negative hysteresis, it shifts earlier (adaptation) to the opposite interpretation. Here, we asked whether positive or negative hysteresis occurs in emotion recognition of inherently ambiguous biological motion, while testing for the controversial debate of a negative versus positive emotional bias. Participants (n = 22) performed a psychophysical experiment in which they were asked to judge stimulus transitions between two emotions, happiness and sadness, from an actor database, and report perceived emotion across time, from one emotion to the opposite as physical cues were continuously changing. Our results reveal perceptual hysteresis in ambiguous emotion recognition, with positive hysteresis (visual persistence) predominating. However, negative hysteresis (adaptation/fatigue) was also observed in particular in the direction from sadness to happiness. This demonstrates a positive (happiness) bias in emotion recognition in ambiguous biological motion recognition. Finally, the interplay between positive and negative hysteresis suggests an underlying competition between visual persistence and adaptation mechanisms during ambiguous emotion recognition.

Aesthetic preferences for prototypical movements in human actions

A commonplace sight is seeing other people walk. Our visual system specializes in processing such actions. Notably, we are not only quick to recognize actions, but also quick to judge how elegantly (or not) people walk. What movements appear appealing, and why do we have such aesthetic experiences? Do aesthetic preferences for body movements arise simply from perceiving others' positive emotions? To answer these questions, we showed observers different point-light walkers who expressed neutral, happy, angry, or sad emotions through their movements and measured the observers' impressions of aesthetic appeal, emotion positivity, and naturalness of these movements. Three experiments were conducted. People showed consensus in aesthetic impressions even after controlling for emotion positivity, finding prototypical walks more aesthetically pleasing than atypical walks. This aesthetic prototype effect could be accounted for by a computational model in which walking actions are treated as a single category (as opposed to multiple emotion categories). The aesthetic impressions were affected both directly by the objective prototypicality of the movements, and indirectly through the mediation of perceived naturalness. These findings extend the boundary of category learning, and hint at possible functions for action aesthetics.

Construction and validation of the Dalian emotional movement open-source set (DEMOS)

Human body movements are important for emotion recognition and social communication and have received extensive attention from researchers. In this field, emotional biological motion stimuli, as depicted by point-light displays, are widely used. However, the number of stimuli in the existing material library is small, and there is a lack of standardized indicators, which subsequently limits experimental design and conduction. Therefore, based on our prior kinematic dataset, we constructed the Dalian Emotional Movement Open-source Set (DEMOS) using computational modeling. The DEMOS has three views (i.e., frontal 0°, left 45°, and left 90°) and in total comprises 2664 high-quality videos of emotional biological motion, each displaying happiness, sadness, anger, fear, disgust, and neutral. All stimuli were validated in terms of recognition accuracy, emotional intensity, and subjective movement. The objective movement for each expression was also calculated. The DEMOS can be downloaded for free from https://osf.io/83fst/ . To our knowledge, this is the largest multi-view emotional biological motion set based on the whole body. The DEMOS can be applied in many fields, including affective computing, social cognition, and psychiatry.

A 5-emotions stimuli set for emotion perception research with full-body dance movements

Ekman famously contended that there are different channels of emotional expression (face, voice, body), and that emotion recognition ability confers an adaptive advantage to the individual. Yet, still today, much emotion perception research is focussed on emotion recognition from the face, and few validated emotionally expressive full-body stimuli sets are available. Based on research on emotional speech perception, we created a new, highly controlled full-body stimuli set. We used the same-sequence approach, and not emotional actions (e.g., jumping of joy, recoiling in fear): One professional dancer danced 30 sequences of (dance) movements five times each, expressing joy, anger, fear, sadness or a neutral state, one at each repetition. We outline the creation of a total of 150, 6-s-long such video stimuli, that show the dancer as a white silhouette on a black background. Ratings from 90 participants (emotion recognition, aesthetic judgment) showed that intended emotion was recognized above chance (chance: 20%; joy: 45%, anger: 48%, fear: 37%, sadness: 50%, neutral state: 51%), and that aesthetic judgment was sensitive to the intended emotion (beauty ratings: joy > anger > fear > neutral state, and sad > fear > neutral state). The stimuli set, normative values and code are available for download.

PLAViMoP database: A new continuously assessed and collaborative 3D point-light display dataset

It was more than 45 years ago that Gunnar Johansson invented the point-light display technique. This showed for the first time that kinematics is crucial for action recognition, and that humans are very sensitive to their conspecifics' movements. As a result, many of today's researchers use point-light displays to better understand the mechanisms behind this recognition ability. In this paper, we propose PLAViMoP, a new database of 3D point-light displays representing everyday human actions (global and fine-motor control movements), sports movements, facial expressions, interactions, and robotic movements. Access to the database is free, at https://plavimop.prd.fr/en/motions . Moreover, it incorporates a search engine to facilitate action retrieval. In this paper, we describe the construction, functioning, and assessment of the PLAViMoP database. Each sequence was analyzed according to four parameters: type of movement, movement label, sex of the actor, and age of the actor. We provide both the mean scores for each assessment of each point-light display, and the comparisons between the different categories of sequences. Our results are discussed in the light of the literature and the suitability of our stimuli for research and applications.

The McNorm library: creating and validating a new library of emotionally expressive whole body dance movements

The ability to exchange affective cues with others plays a key role in our ability to create and maintain meaningful social relationships. We express our emotions through a variety of socially salient cues, including facial expressions, the voice, and body movement. While significant advances have been made in our understanding of verbal and facial communication, to date, understanding of the role played by human body movement in our social interactions remains incomplete. To this end, here we describe the creation and validation of a new set of emotionally expressive whole-body dance movement stimuli, named the Motion Capture Norming (McNorm) Library, which was designed to reconcile a number of limitations associated with previous movement stimuli. This library comprises a series of point-light representations of a dancer's movements, which were performed to communicate to observers neutrality, happiness, sadness, anger, and fear. Based on results from two validation experiments, participants could reliably discriminate the intended emotion expressed in the clips in this stimulus set, with accuracy rates up to 60% (chance = 20%). We further explored the impact of dance experience and trait empathy on emotion recognition and found that neither significantly impacted emotion discrimination. As all materials for presenting and analysing this movement library are openly available, we hope this resource will aid other researchers in further exploration of affective communication expressed by human bodily movement.

Ties between reading faces, bodies, eyes, and autistic traits

While reading covered with masks faces during the COVID-19 pandemic, for efficient social interaction, we need to combine information from different sources such as the eyes (without faces hidden by masks) and bodies. This may be challenging for individuals with neuropsychiatric conditions, in particular, autism spectrum disorders. Here we examined whether reading of dynamic faces, bodies, and eyes are tied in a gender-specific way, and how these capabilities are related to autistic traits expression. Females and males accomplished a task with point-light faces along with a task with point-light body locomotion portraying different emotional expressions. They had to infer emotional content of displays. In addition, participants were administered the Reading the Mind in the Eyes Test, modified and Autism Spectrum Quotient questionnaire. The findings show that only in females, inferring emotions from dynamic bodies and faces are firmly linked, whereas in males, reading in the eyes is knotted with face reading. Strikingly, in neurotypical males only, accuracy of face, body, and eyes reading was negatively tied with autistic traits. The outcome points to gender-specific modes in social cognition: females rely upon merely dynamic cues while reading faces and bodies, whereas males most likely trust configural information. The findings are of value for examination of face and body language reading in neuropsychiatric conditions, in particular, autism, most of which are gender/sex-specific. This work suggests that if male individuals with autistic traits experience difficulties in reading covered with masks faces, these deficits may be unlikely compensated by reading (even dynamic) bodies and faces. By contrast, in females, reading covered faces as well as reading language of dynamic bodies and faces are not compulsorily connected to autistic traits preventing them from paying high costs for maladaptive social interaction.

Working Memory Capacity of Biological Motion's Basic Unit: Decomposing Biological Motion From the Perspective of Systematic Anatomy

Many studies have shown that about three biological motions (BMs) can be maintained in working memory. However, no study has yet analyzed the difficulties of experiment materials used, which partially affect the ecological validity of the experiment results. We use the perspective of system anatomy to decompose BM, and thoroughly explore the influencing factors of difficulties of BMs, including presentation duration, joints to execute motions, limbs to execute motions, type of articulation interference tasks, and number of joints and planes involved in the BM. We apply the change detection paradigm supplemented by the articulation interference task to measure the BM working memory capacity (WMC) of participants. Findings show the following: the shorter the presentation duration, the less participants remembered; the more their wrist moved, the less accurate their memory was; repeating verbs provided better results than did repeating numerals to suppress verbal encoding; the more complex the BM, the less participants remembered; and whether the action was executed by the handed limbs did not affect the WMC. These results indicate that there are many factors that can be used to adjust BM memory load. These factors can help sports psychology professionals to better evaluate the difficulty of BMs, and can also partially explain the differences in estimations of BM WMC in previous studies.

Presenting KAPODI – The Searchable Database of Emotional Stimuli Sets

Emotional stimuli such as images, words, or video clips are often used in studies researching emotion. New sets are continuously being published, creating an immense number of available sets and complicating the task for researchers who are looking for suitable stimuli. This paper presents the KAPODI-database of emotional stimuli sets that are freely available or available upon request. Over 45 aspects including over 25 key set characteristics have been extracted and listed for each set. The database facilitates finding of and comparison between individual sets. It currently contains sets published between 1963 and 2020. A searchable online version ( https://airtable.com/shrnVoUZrwu6riP9b ) allows users to select specific set characteristics and to find matching sets accordingly, as well as to add new published sets.

Modeling Data-Driven Dominance Traits for Virtual Characters Using Gait Analysis

We present a data-driven algorithm for generating gaits of virtual characters with varying dominance traits. Our formulation utilizes a user study to establish a data-driven dominance mapping between gaits and dominance labels. We use our dominance mapping to generate walking gaits for virtual characters that exhibit a variety of dominance traits while interacting with the user. Furthermore, we extract gait features based on known criteria in visual perception and psychology literature that can be used to identify the dominance levels of any walking gait. We validate our mapping and the perceived dominance traits by a second user study in an immersive virtual environment. Our gait dominance classification algorithm can classify the dominance traits of gaits with ˜73 percent accuracy. We also present an application of our approach that simulates interpersonal relationships between virtual characters. To the best of our knowledge, ours is the first practical approach to classifying gait dominance and generate dominance traits in virtual characters.

Emotion Transfer for 3D Hand and Full Body Motion Using StarGAN

In this paper, we propose a new data-driven framework for 3D hand and full-body motion emotion transfer. Specifically, we formulate the motion synthesis task as an image-to-image translation problem. By presenting a motion sequence as an image representation, the emotion can be transferred by our framework using StarGAN. To evaluate our proposed method’s effectiveness, we first conducted a user study to validate the perceived emotion from the captured and synthesized hand motions. We further evaluate the synthesized hand and full body motions qualitatively and quantitatively. Experimental results show that our synthesized motions are comparable to the captured motions and those created by an existing method in terms of naturalness and visual quality.

Kinematic dataset of actors expressing emotions

Human body movements can convey a variety of emotions and even create advantages in some special life situations. However, how emotion is encoded in body movements has remained unclear. One reason is that there is a lack of public human body kinematic dataset regarding the expressing of various emotions. Therefore, we aimed to produce a comprehensive dataset to assist in recognizing cues from all parts of the body that indicate six basic emotions (happiness, sadness, anger, fear, disgust, surprise) and neutral expression. The present dataset was created using a portable wireless motion capture system. Twenty-two semi-professional actors (half male) completed performances according to the standardized guidance and preferred daily events. A total of 1402 recordings at 125 Hz were collected, consisting of the position and rotation data of 72 anatomical nodes. To our knowledge, this is now the largest emotional kinematic dataset of the human body. We hope this dataset will contribute to multiple fields of research and practice, including social neuroscience, psychiatry, computer vision, and biometric and information forensics.

Human activity recognition using magnetic induction-based motion signals and deep recurrent neural networks

Recognizing human physical activities using wireless sensor networks has attracted significant research interest due to its broad range of applications, such as healthcare, rehabilitation, athletics, and senior monitoring. There are critical challenges inherent in designing a sensor-based activity recognition system operating in and around a lossy medium such as the human body to gain a trade-off among power consumption, cost, computational complexity, and accuracy. We introduce an innovative wireless system based on magnetic induction for human activity recognition to tackle these challenges and constraints. The magnetic induction system is integrated with machine learning techniques to detect a wide range of human motions. This approach is successfully evaluated using synthesized datasets, laboratory measurements, and deep recurrent neural networks.

From Emotions to Mood Disorders: A Survey on Gait Analysis Methodology

Mood disorders affect more than 300 million people worldwide and can cause devastating consequences. Elderly people and patients with neurological conditions are particularly susceptible to depression. Gait and body movements can be affected by mood disorders, and thus they can be used as a surrogate sign, as well as an objective index for pervasive monitoring of emotion and mood disorders in daily life. Here we review evidence that demonstrates the relationship between gait, emotions and mood disorders, highlighting the potential of a multimodal approach that couples gait data with physiological signals and home-based monitoring for early detection and management of mood disorders. This could enhance self-awareness, enable the development of objective biomarkers that identify high risk subjects and promote subject-specific treatment.

The Relationship between Biological Motion-Based Visual Consciousness and Attention: An Electroencephalograph Study

Understanding and predicting the intentions of others through limb movements are vital to social interaction. The processing of biological motion is unique from the processing of motion of inanimate objects. Presently, there is controversy over whether visual consciousness of biological motion is regulated by visual attention. In addition, the neural mechanisms involved in biological motion-related visual awareness are not known. In the current study, the relationship between visual awareness (aware vs unaware), represented by a point-light walker and biological-motion-based attention, manipulated by a difference in congruence (congruent, incongruent) between the direction of a pre-cue and that of biological motion was explored. The neural mechanisms involved in processing the stimuli were explored through electroencephalography. Both early (50-150 ms, 100-200 ms, and 174-226 ms after target presentation) and late (350-550 ms after target presentation) awareness-related neural processings were observed during a biological motion-based congruency task. Early processing was localized to occipital-parietal regions, such as the left postcentral gyrus, the left middle occipital gyrus, and the right precentral gyrus. In the 174-226-ms window, the activity in the occipital region was gradually replaced by activity in the parietal and frontal regions. Late processing was localized to frontal-parietal regions, such as the right dorsal superior frontal gyrus, the left medial superior frontal gyrus, and the occipito-temporal regions. Congruency-related processing occurred in the 246-260-ms window and was localized to the right superior occipital gyrus. In summary, due to its complexity, biological motion awareness has a unique neural basis.

What Does a Hand-Over Tell?-Individuality of ShortMotion Sequences

How much information with regard to identity and further individual participant characteristics are revealed by relatively short spatio-temporal motion trajectories of a person? We study this question by selecting a set of individual participant characteristics and analysing motion captured trajectories of an exemplary class of familiar movements, namely handover of an object to another person. The experiment is performed with different participants under different, predefined conditions. A selection of participant characteristics, such as the Big Five personality traits, gender, weight, or sportiness, are assessed and we analyse the impact of the three factor groups “participant identity”, “participant characteristics”, and “experimental conditions” on the observed hand trajectories. The participants’ movements are recorded via optical marker-based hand motion capture. One participant, the giver, hands over an object to the receiver. The resulting time courses of three-dimensional positions of markers are analysed. Multidimensional scaling is used to project trajectories to points in a dimension-reduced feature space. Supervised learning is also applied. We find that “participant identity” seems to have the highest correlation with the trajectories, with factor group “experimental conditions” ranking second. On the other hand, it is not possible to find a correlation between the “participant characteristics” and the hand trajectory features.

A multi-sensory code for emotional arousal

People express emotion using their voice, face and movement, as well as through abstract forms as in art, architecture and music. The structure of these expressions often seems intuitively linked to its meaning: romantic poetry is written in flowery curlicues, while the logos of death metal bands use spiky script. Here, we show that these associations are universally understood because they are signalled using a multi-sensory code for emotional arousal. Specifically, variation in the central tendency of the frequency spectrum of a stimulus-its spectral centroid-is used by signal senders to express emotional arousal, and by signal receivers to make emotional arousal judgements. We show that this code is used across sounds, shapes, speech and human body movements, providing a strong multi-sensory signal that can be used to efficiently estimate an agent's level of emotional arousal.

Anticipatory judgements associated with vision of an opponent’s end-effector: An approach by motion perturbation and spatial occlusion

This study was aimed at determining how the visual information of an end-effector (racket) and the intermediate extremity (arm) of a tennis server contribute to the receiver’s anticipatory judgement of ball direction. In all, 15 experienced tennis players and 15 novice counterparts viewed a spatially occluded computer graphics animation of a tennis serve (no-occlusion, racket-occlusion, and body-occlusion) and made anticipatory judgements of ball direction on a visual analogue scale (VAS). The patterns of the serve motions were generated by a simulation technique that computationally perturbs the rotation speed of the selected racket-arm joint (forearm pronation and elbow extension) on a captured serve motion. The results suggested that the anticipatory judgements were monotonically attuned with the perturbation rate of the forearm pronation speed excepting under the conditions of the racket-occlusion model. Although such attunements were not observed in the elbow perturbation conditions, the results of correlation analysis indicated that the residual information in the spatially occluded models had a similar effect to the no-occlusion model within the individual experienced participants. The findings support the notion that end-effector (racket) provides deterministic cues for anticipation, as well as imply that players are able to benefit from the relative motion of an intermediate extremity (elbow extension).

Not all about sex: neural and biobehavioral functions of human dance

This paper provides an integrative review of neuroscientific and biobehavioral evidence about the effects of dance on the individual across cultural differences. Dance moves us, and many derive aesthetic pleasure from it. However, in addition-and beyond aesthetics-we propose that dance has noteworthy, deeper neurobiological effects. We first summarize evidence that illustrates the centrality of dance to human life indirectly from archaeology, comparative psychology, developmental psychology, and cross-cultural psychology. Second, we review empirical evidence for six neural and biobehavioral functions of dance: (1) attentional focus/flow, (2) basic emotional experiences, (3) imagery, (4) communication, (5) self-intimation, and (6) social cohesion. We discuss the reviewed evidence in relation to current debates in the field of empirical enquiry into the functions of human dance, questioning the positions that dance is (1) just for pleasure, (2) all about sex, (3) just for mood management and well-being, and (4) for experts only. Being a young field, evidence is still piecemeal and inconclusive. This review aims to take a step toward a systematization of an emerging avenue of research: a neuro- and biobehavioral science of dance.

Using a Kinect sensor to acquire biological motion: Toolbox and evaluation

Biological motion (BM) is the movement of animate entities, which conveys rich social information. To obtain pure BM, researchers nowadays predominantly use point-light displays (PLDs), which depict BM through a set of light points (e.g., 12 points) placed at distinct joints of a moving human body. Most prevalent BM stimuli are created by state-of-the-art motion capture systems. Although these stimuli are highly precise, the motion capture system is expensive and bulky, and its process of constructing a PLD-based BM is time-consuming and complex. These factors impede the investigation of BM mechanisms. In this study, we propose a free Kinect-based biological motion capture (KBC) toolbox based on the Kinect Sensor 2.0 in C++. The KBC toolbox aims to help researchers acquire PLD-based BM in an easy, low-cost, and user-friendly way. We conducted three experiments to examine whether KBC-generated BM can genuinely reflect the processing characteristics of BM: (1) Is BM from this source processed globally in vision? (2) Does its BM (e.g., from the feet) retain detailed local information? and (3) Does the BM convey emotional information? We obtained positive results in response to all three questions. Therefore, we think that the KBC toolbox can be useful in generating BM for future research.

Facilitating Effects of Emotion on the Perception of Biological Motion: Evidence for a Happiness Superiority Effect

It has been reported that visual perception can be influenced not only by the physical features of a stimulus but also by the emotional valence of the stimulus, even without explicit emotion recognition. Some previous studies reported an anger superiority effect while others found a happiness superiority effect during visual perception. It thus remains unclear as to which emotion is more influential. In the present study, we conducted two experiments using biological motion (BM) stimuli to examine whether emotional valence of the stimuli would affect BM perception; and if so, whether a specific type of emotion is associated with a superiority effect. Point-light walkers with three emotion types (anger, happiness, and neutral) were used, and the threshold to detect BM within noise was measured in Experiment 1. Participants showed higher performance in detecting happy walkers compared with the angry and neutral walkers. Follow-up motion velocity analysis revealed that physical difference among the stimuli was not the main factor causing the effect. The results of the emotion recognition task in Experiment 2 also showed a happiness superiority effect, as in Experiment 1. These results show that emotional valence (happiness) of the stimuli can facilitate the processing of BM.

A dyadic stimulus set of audiovisual affective displays for the study of multisensory, emotional, social interactions

We describe the creation of the first multisensory stimulus set that consists of dyadic, emotional, point-light interactions combined with voice dialogues. Our set includes 238 unique clips, which present happy, angry and neutral emotional interactions at low, medium and high levels of emotional intensity between nine different actor dyads. The set was evaluated in a between-design experiment, and was found to be suitable for a broad potential application in the cognitive and neuroscientific study of biological motion and voice, perception of social interactions and multisensory integration. We also detail in this paper a number of supplementary materials, comprising AVI movie files for each interaction, along with text files specifying the three dimensional coordinates of each point-light in each frame of the movie, as well as unprocessed AIFF audio files for each dialogue captured. The full set of stimuli is available to download from: http://motioninsocial.com/stimuli_set/ .

Communicative interactions in point-light displays: Choosing among multiple response alternatives

Vision scientists are increasingly relying on the point-light technique as a way to investigate the perception of human motion. Unfortunately, the lack of standardized stimulus sets has so far limited the use of this technique for studying social interaction. Here, we describe a new tool to study the interaction between two agents starting from point-light displays: the Communicative Interaction Database - 5AFC format (CID-5). The CID-5 consists of 14 communicative and seven non-communicative individual actions performed by two agents. Stimuli were constructed by combining motion capture techniques and 3-D animation software to provide precise control over the computer-generated actions. For each action stimulus, we provide coordinate files and movie files depicting the action as seen from four different perspectives. Furthermore, the archive contains a text file with a list of five alternative action descriptions to construct forced-choice paradigms. In order to validate the CID-5 format, we provide normative data collected to assess action identification within a 5AFC tasks. The CID-5 archive is freely downloadable from http://bsb-lab.org/research/ and from the supplementary materials of this article.

The Automatic Detection of Chronic Pain-Related Expression: Requirements, Challenges and the Multimodal EmoPain Dataset

Pain-related emotions are a major barrier to effective self rehabilitation in chronic pain. Automated coaching systems capable of detecting these emotions are a potential solution. This paper lays the foundation for the development of such systems by making three contributions. First, through literature reviews, an overview of how pain is expressed in chronic pain and the motivation for detecting it in physical rehabilitation is provided. Second, a fully labelled multimodal dataset (named 'EmoPain') containing high resolution multiple-view face videos, head mounted and room audio signals, full body 3D motion capture and electromyographic signals from back muscles is supplied. Natural unconstrained pain related facial expressions and body movement behaviours were elicited from people with chronic pain carrying out physical exercises. Both instructed and non-instructed exercises were considered to reflect traditional scenarios of physiotherapist directed therapy and home-based self-directed therapy. Two sets of labels were assigned: level of pain from facial expressions annotated by eight raters and the occurrence of six pain-related body behaviours segmented by four experts. Third, through exploratory experiments grounded in the data, the factors and challenges in the automated recognition of such expressions and behaviour are described, the paper concludes by discussing potential avenues in the context of these findings also highlighting differences for the two exercise scenarios addressed.

Adaptive Body Gesture Representation for Automatic Emotion Recognition

We present a computational model and a system for the automated recognition of emotions starting from full-body movement. Three-dimensional motion data of full-body movements are obtained either from professional optical motion-capture systems (Qualisys) or from low-cost RGB-D sensors (Kinect and Kinect2). A number of features are then automatically extracted at different levels, from kinematics of a single joint to more global expressive features inspired by psychology and humanistic theories (e.g., contraction index, fluidity, and impulsiveness). An abstraction layer based on dictionary learning further processes these movement features to increase the model generality and to deal with intraclass variability, noise, and incomplete information characterizing emotion expression in human movement. The resulting feature vector is the input for a classifier performing real-time automatic emotion recognition based on linear support vector machines. The recognition performance of the proposed model is presented and discussed, including the tradeoff between precision of the tracking measures (we compare the Kinect RGB-D sensor and the Qualisys motion-capture system) versus dimension of the training dataset. The resulting model and system have been successfully applied in the development of serious games for helping autistic children learn to recognize and express emotions by means of their full-body movement.

Locality sensitivity discriminant analysis-based feature ranking of human emotion actions recognition

[Purpose] Computational intelligence similar to pattern recognition is frequently confronted with high-dimensional data. Therefore, the reduction of the dimensionality is critical to make the manifold features amenable. Procedures that are analytically or computationally manageable in smaller amounts of data and low-dimensional space can become important to produce a better classification performance. [Methods] Thus, we proposed two stage reduction techniques. Feature selection-based ranking using information gain (IG) and Chi-square (Chisq) are used to identify the best ranking of the features selected for emotion classification in different actions including knocking, throwing, and lifting. Then, feature reduction-based locality sensitivity discriminant analysis (LSDA) and principal component analysis (PCA) are used to transform the selected feature to low-dimensional space. Two-stage feature selection-reduction methods such as IG-PCA, IG-LSDA, Chisq-PCA, and Chisq-LSDA are proposed. [Results] The result confirms that applying feature ranking combined with a dimensional-reduction method increases the performance of the classifiers. [Conclusion] The dimension reduction was performed using LSDA by denoting the features of the highest importance determined using IG and Chisq to not only improve the effectiveness but also reduce the computational time.

The influences of emotional intensity for happiness and sadness on walking

Walking is one of the most common activities that we perform every day. Even if the main goal of walking is to move from one place to another place, walking can also convey emotional clues in social context. Those clues can be used to improve interactions or any messages we want to express. However, there are not many studies on the effects of the intensity of the emotions on the walking. In this paper, the authors propose to assess the differences between the expression of emotion regarding the expressed intensity (low, middle, high and exaggerated). We observed two professional actors perform emotive walking, with different intensities and we analyzed the recorded data. For each emotion, we analyzed characteristic features which can be used in the future to model gait patterns and to recognize emotions from the gait parameters. Additionally, we found characteristics which can be used to create new emotion expression for our biped robot Kobian, improving the human-robot interaction.

Audiovisual integration of emotional signals from others' social interactions

Audiovisual perception of emotions has been typically examined using displays of a solitary character (e.g., the face-voice and/or body-sound of one actor). However, in real life humans often face more complex multisensory social situations, involving more than one person. Here we ask if the audiovisual facilitation in emotion recognition previously found in simpler social situations extends to more complex and ecological situations. Stimuli consisting of the biological motion and voice of two interacting agents were used in two experiments. In Experiment 1, participants were presented with visual, auditory, auditory filtered/noisy, and audiovisual congruent and incongruent clips. We asked participants to judge whether the two agents were interacting happily or angrily. In Experiment 2, another group of participants repeated the same task, as in Experiment 1, while trying to ignore either the visual or the auditory information. The findings from both experiments indicate that when the reliability of the auditory cue was decreased participants weighted more the visual cue in their emotional judgments. This in turn translated in increased emotion recognition accuracy for the multisensory condition. Our findings thus point to a common mechanism of multisensory integration of emotional signals irrespective of social stimulus complexity.

Biological motion perception is affected by age and cognitive style in children aged 8-15

The current paper aims to address the question of how biological motion perception in different social contexts is influenced by age or also affected by cognitive styles. We examined developmental changes of biological motion perception among 141 school children aged 8-15 using point-light displays in monadic and dyadic social contexts. Furthermore, the cognitive styles of participants were investigated using empathizing-systemizing questionnaires. Results showed that the age and empathizing ability strongly predicted improvement in action perception in both contexts. However the systemizing ability was an independent predictor of performance only in monadic contexts. Furthermore, accuracy of action perception increased significantly from 46.4% (SD = 16.1) in monadic to 62.5% (SD = 11.5) in dyadic social contexts. This study can help to identify the roles of social context in biological motion perception and shows that children with different cognitive styles may present different biological motion perception.

Tactile input and empathy modulate the perception of ambiguous biological motion

Evidence has shown that task-irrelevant auditory cues can bias perceptual decisions regarding directional information associated with biological motion, as indicated in perceptual tasks using point-light walkers (PLWs) (Brooks et al., 2007). In the current study, we extended the investigation of cross-modal influences to the tactile domain by asking how tactile input resolves perceptual ambiguity in visual apparent motion, and how empathy plays a role in this cross-modal interaction. In Experiment 1, we simulated the tactile feedback on the observers' fingertips when the (upright or inverted) PLWs (comprised of either all red or all green dots) were walking (leftwards or rightwards). The temporal periods between tactile events and critical visual events (the PLW's feet hitting the ground) were manipulated so that the tap could lead, synchronize, or lag the visual foot-hitting-ground event. We found that the temporal structures between tactile (feedback) and visual (hitting) events systematically biases the directional perception for upright PLWs, making either leftwards or rightwards more dominant. However, this effect was absent for inverted PLWs. In Experiment 2, we examined how empathy modulates cross-modal capture. Instead of giving tactile feedback on participants' fingertips, we gave taps on their ankles and presented the PLWs with motion directions of approaching (facing toward observer)/receding (facing away from observer) to resemble normal walking postures. With the same temporal structure, we found that individuals with higher empathy were more subject to perceptual bias in the presence of tactile feedback. Taken together, our findings showed that task-irrelevant tactile input can resolve the otherwise ambiguous perception of the direction of biological motion, and this cross-modal bias was mediated by higher level social-cognitive factors, including empathy.

The MPI emotional body expressions database for narrative scenarios

Emotion expression in human-human interaction takes place via various types of information, including body motion. Research on the perceptual-cognitive mechanisms underlying the processing of natural emotional body language can benefit greatly from datasets of natural emotional body expressions that facilitate stimulus manipulation and analysis. The existing databases have so far focused on few emotion categories which display predominantly prototypical, exaggerated emotion expressions. Moreover, many of these databases consist of video recordings which limit the ability to manipulate and analyse the physical properties of these stimuli. We present a new database consisting of a large set (over 1400) of natural emotional body expressions typical of monologues. To achieve close-to-natural emotional body expressions, amateur actors were narrating coherent stories while their body movements were recorded with motion capture technology. The resulting 3-dimensional motion data recorded at a high frame rate (120 frames per second) provides fine-grained information about body movements and allows the manipulation of movement on a body joint basis. For each expression it gives the positions and orientations in space of 23 body joints for every frame. We report the results of physical motion properties analysis and of an emotion categorisation study. The reactions of observers from the emotion categorisation study are included in the database. Moreover, we recorded the intended emotion expression for each motion sequence from the actor to allow for investigations regarding the link between intended and perceived emotions. The motion sequences along with the accompanying information are made available in a searchable MPI Emotional Body Expression Database. We hope that this database will enable researchers to study expression and perception of naturally occurring emotional body expressions in greater depth.

A norming study and library of 203 dance movements

Dance stimuli have been used in experimental studies of (i) how movement is processed in the brain; (ii) how affect is perceived from bodily movement; and (iii) how dance can be a source of aesthetic experience. However, stimulus materials across--and even within--these three domains of research have varied considerably. Thus, integrative conclusions remain elusive. Moreover, concerns have been raised that the movements selected for such stimuli are qualitatively too different from the actual art form dance, potentially introducing noise in the data. We propose a library of dance stimuli which responds to the stimuli requirements and design criteria of these three areas of research, while at the same time respecting a dance art-historical perspective, offering greater ecological validity as compared with previous dance stimulus sets. The stimuli are 5-6 s long video clips, selected from genuine ballet performances. Following a number of coding experiments, the resulting stimulus library comprises 203 ballet dance stimuli coded in (i) 25 qualitative and quantitative movement variables; (ii) affective valence and arousal; and (iii) the aesthetic qualities beauty, liking, and interest. An Excel spreadsheet with these data points accompanies this manuscript, and the stimuli can be obtained from the authors upon request.

A database of whole-body action videos for the study of action, emotion, and untrustworthiness

We present a database of high-definition (HD) videos for the study of traits inferred from whole-body actions. Twenty-nine actors (19 female) were filmed performing different actions-walking, picking up a box, putting down a box, jumping, sitting down, and standing and acting-while conveying different traits, including four emotions (anger, fear, happiness, sadness), untrustworthiness, and neutral, where no specific trait was conveyed. For the actions conveying the four emotions and untrustworthiness, the actions were filmed multiple times, with the actor conveying the traits with different levels of intensity. In total, we made 2,783 action videos (in both two-dimensional and three-dimensional format), each lasting 7 s with a frame rate of 50 fps. All videos were filmed in a green-screen studio in order to isolate the action information from all contextual detail and to provide a flexible stimulus set for future use. In order to validate the traits conveyed by each action, we asked participants to rate each of the actions corresponding to the trait that the actor portrayed in the two-dimensional videos. To provide a useful database of stimuli of multiple actions conveying multiple traits, each video name contains information on the gender of the actor, the action executed, the trait conveyed, and the rating of its perceived intensity. All videos can be downloaded free at the following address: http://www-users.york.ac.uk/~neb506/databases.html. We discuss potential uses for the database in the analysis of the perception of whole-body actions.

Empirical evaluation of the uncanny valley hypothesis fails to confirm the predicted effect of motion

The uncanny valley hypothesis states that the acceptability of an artificial character will not increase linearly in relation to its likeness to human form. Instead, after an initial rise in acceptability there will be a pronounced decrease when the character is similar, but not identical to human form (Mori, 1970/2012). Moreover, it has been claimed but never directly tested that movement would accentuate this dip and make moving characters less acceptable. We used a number of full-body animated computer characters along with a parametrically defined motion set to examine the effect of motion quality on the uncanny valley. We found that improving the motion quality systematically improved the acceptability of the characters. In particular, the character classified in the deepest location of the uncanny valley became more acceptable when it was animated. Our results showed that although an uncanny valley was found for static characters, the deepening of the valley with motion, originally predicted by Mori (1970/2012), was not obtained.

Emotion through locomotion: gender impact

Body language reading is of significance for daily life social cognition and successful social interaction, and constitutes a core component of social competence. Yet it is unclear whether our ability for body language reading is gender specific. In the present work, female and male observers had to visually recognize emotions through point-light human locomotion performed by female and male actors with different emotional expressions. For subtle emotional expressions only, males surpass females in recognition accuracy and readiness to respond to happy walking portrayed by female actors, whereas females exhibit a tendency to be better in recognition of hostile angry locomotion expressed by male actors. In contrast to widespread beliefs about female superiority in social cognition, the findings suggest that gender effects in recognition of emotions from human locomotion are modulated by emotional content of actions and opposite actor gender. In a nutshell, the study makes a further step in elucidation of gender impact on body language reading and on neurodevelopmental and psychiatric deficits in visual social cognition.

Communicative and noncommunicative point-light actions featuring high-resolution representation of the hands and fingers

We describe the creation of a set of point-light movies depicting 43 communicative gestures and 43 noncommunicative, pantomimed actions. These actions were recorded using a motion capture system that is worn on the body and provides accurate capture of the positions and movements of individual fingers. The movies created thus include point-lights on the fingers, allowing for representation of actions and gestures that would not be possible with a conventional, line-of-sight-based motion capture system. These videos would be suitable for use in cognitive and cognitive neuroscientific studies of biological motion and gesture perception. Each video is described, along with an H statistic indicating the consistency of the descriptive labels that 20 observers gave to the actions. We also produced a scrambled version of each movie, in which the starting position of each point was randomized but its local motion vector was preserved. These scrambled movies would be suitable for use as control stimuli in experimental studies. As supplementary materials, we provide QuickTime movie files of each action, along with text files specifying the three-dimensional coordinates of each point-light in each frame of each movie.

Aprendizagem motora baseada em demonstrações de movimento biológico

O objetivo deste estudo foi avaliar o processo de aprendizagem motora de uma habilidade complexa da Ginástica Artística a partir da observação de demonstrações de modelos de pontos de luz e vídeo. Dezesseis participantes divididas em grupos dos respectivos modelos executaram um pré-teste, seguido de 100 tentativas de uma parada de mãos, igualmente distribuídas em blocos de 10 tentativas em dois dias, alternando períodos de demonstração e prática, com um teste de retenção após um dia. Cinemática de braço, tronco e perna das participantes possibilitaram análise da semelhança entre a coordenação de cada participante e do modelo e do tempo de movimento; a performance das participantes também foi avaliada por duas especialistas em Ginástica Artística. Ambas as análises indicaram que os grupos não diferiram. Os resultados são discutidos em termos da hipótese de suficiência de informação nos modelos de movimento biológico particularmente aplicada ao processo de aprendizagem de habilidades motoras complexas.

Signature movements lead to efficient search for threatening actions

The ability to find and evade fighting persons in a crowd is potentially life-saving. To investigate how the visual system processes threatening actions, we employed a visual search paradigm with threatening boxer targets among emotionally-neutral walker distractors, and vice versa. We found that a boxer popped out for both intact and scrambled actions, whereas walkers did not. A reverse correlation analysis revealed that observers' responses clustered around the time of the “punch", a signature movement of boxing actions, but not around specific movements of the walker. These findings support the existence of a detector for signature movements in action perception. This detector helps in rapidly detecting aggressive behavior in a crowd, potentially through an expedited (sub)cortical threat-detection mechanism.

A FULL-BODY GESTURE DATABASE FOR HUMAN GESTURE ANALYSIS

This paper presents a full-body gesture database which contains 2D video data and 3D motion data of 14 normal gestures, 10 abnormal gestures and 30 command gestures for 20 subjects. We call this database the Korea University Gesture (KUG) database. Using 3D motion cameras and 3 sets of stereo cameras, we captured 3D motion data and 3 pairs of stereo-video data in 3 different directions for normal and abnormal gestures. In case of command gestures, 2 pairs of stereo-video data were obtained by 2 sets of stereo cameras with different focal lengths in order to capture views of whole body and upper body, simultaneously. The 2D silhouette data was synthesized by separating a subject and background in 2D stereo-video data. In this paper, we describe the gesture capture system, the organization of database, the potential usages of the database and the contact point for the KUG database. We expect that this database would be very useful for the study of 2D/3D human gesture and its application.

Do distinct atypical cortical networks process biological motion information in adults with Autism Spectrum Disorders?

Whether people with Autism Spectrum Disorders (ASDs) have a specific deficit when processing biological motion has been a topic of much debate. We used psychophysical methods to determine individual behavioural thresholds in a point-light direction discrimination paradigm for a small but carefully matched groups of adults (N=10 per group) with and without ASDs. These thresholds were used to derive individual stimulus levels in an identical fMRI task, with the purpose of equalising task performance across all participants whilst inside the scanner. The results of this investigation show that despite comparable behavioural performance both inside and outside the scanner, the group with ASDs shows a different pattern of BOLD activation from the TD group in response to the same stimulus levels. Furthermore, connectivity analysis suggests that the main differences between the groups are that the TD group utilise a unitary network with information passing from temporal to parietal regions, whilst the ASD group utilise two distinct networks; one utilising motion sensitive areas and another utilising form selective areas. Furthermore, a temporal-parietal link that is present in the TD group is missing in the ASD group. We tentatively propose that these differences may occur due to early dysfunctional connectivity in the brains of people with ASDs, which to some extent is compensated for by rewiring in high functioning adults.