Experimental Results of Affective Valence and Arousal to Avatar's Facial Expressions

Building an Emotionally Responsive Avatar with Dynamic Facial Expressions in Human—Computer Interactions

The role of affect has been long studied in human–computer interactions. Unlike previous studies that focused on seven basic emotions, an avatar named Diana was introduced who expresses a higher level of emotional intelligence. To adapt to the users various affects during interaction, Diana simulates emotions with dynamic facial expressions. When two people collaborated to build blocks, their affects were recognized and labeled using the Affdex SDK and a descriptive analysis was provided. When participants turned to collaborate with Diana, their subjective responses were collected and the length of completion was recorded. Three modes of Diana were involved: a flat-faced Diana, a Diana that used mimicry facial expressions, and a Diana that used emotionally responsive facial expressions. Twenty-one responses were collected through a five-point Likert scale questionnaire and the NASA TLX. Results from questionnaires were not statistically different. However, the emotionally responsive Diana obtained more positive responses, and people spent the longest time with the mimicry Diana. In post-study comments, most participants perceived facial expressions on Diana’s face as natural, four mentioned uncomfortable feelings caused by the Uncanny Valley effect.

Use and Usefulness of Dynamic Face Stimuli for Face Perception Studies-a Review of Behavioral Findings and Methodology

Faces that move contain rich information about facial form, such as facial features and their configuration, alongside the motion of those features. During social interactions, humans constantly decode and integrate these cues. To fully understand human face perception, it is important to investigate what information dynamic faces convey and how the human visual system extracts and processes information from this visual input. However, partly due to the difficulty of designing well-controlled dynamic face stimuli, many face perception studies still rely on static faces as stimuli. Here, we focus on evidence demonstrating the usefulness of dynamic faces as stimuli, and evaluate different types of dynamic face stimuli to study face perception. Studies based on dynamic face stimuli revealed a high sensitivity of the human visual system to natural facial motion and consistently reported dynamic advantages when static face information is insufficient for the task. These findings support the hypothesis that the human perceptual system integrates sensory cues for robust perception. In the present paper, we review the different types of dynamic face stimuli used in these studies, and assess their usefulness for several research questions. Natural videos of faces are ecological stimuli but provide limited control of facial form and motion. Point-light faces allow for good control of facial motion but are highly unnatural. Image-based morphing is a way to achieve control over facial motion while preserving the natural facial form. Synthetic facial animations allow separation of facial form and motion to study aspects such as identity-from-motion. While synthetic faces are less natural than videos of faces, recent advances in photo-realistic rendering may close this gap and provide naturalistic stimuli with full control over facial motion. We believe that many open questions, such as what dynamic advantages exist beyond emotion and identity recognition and which dynamic aspects drive these advantages, can be addressed adequately with different types of stimuli and will improve our understanding of face perception in more ecological settings.

A influência da intensidade emocional no reconhecimento de emoções em faces por crianças brasileiras

The ability to recognize emotions in faces is essential to human interaction and occurs since childhood. Hypothesis: research using the morphing technique assume that children require greater or lesser intensity of emotional expression to perceive it. Objective: to examine the emotional recognition of faces in childhood, using a task with emotional intensity variation. Method: it was applied a Test of Facial Emotion Recognition for Children to 28 children between 7 and 11 years, of both sexes, which presented 168 faces manipulated by the morphing technique, of the six basic emotions. Results: age as a trend growth of the likelihood of success at the task; more right answers for happiness and worst performances for fear; and the emotional intensity increasing at 42% the chance of success by every unit of intensity. Conclusion: these findings are relevant because they show the recognition of emotions at different levels as a more sensitive method.

Efficacy of a Virtual Reality Commercial Gaming Device in Upper Limb Recovery after Stroke: A Randomized, Controlled Study

OBJECTIVE

To compare the efficacy of a virtual reality commercial gaming device, Nintendo wii (NW) with conventional therapy and customary care in facilitating upper limb recovery after stroke.

DESIGN

Randomized, controlled, single-blinded study.

SETTING

Tertiary rehabilitation center.

PARTICIPANTS

105 subjects admitted to in inpatient rehabilitation program within 6 weeks of stroke onset.

INTERVENTIONS

Subjects were randomly assigned to one of three groups of upper limb exercises: (1) NW gaming; (2) conventional therapy; (3) control. NW gaming and conventional therapy were provided fourtimes a week for 3 weeks.

MAIN OUTCOME MEASURE(S)

The main outcome measure was Fugl-Meyer assessment (FMA) of upper limb function. Secondary outcome measures included Action Research Arm Test, Functional Independence Measure, and Stroke Impact Scale. These measures were assessed at baseline, completion of intervention (week 3) and at 4 weeks and 8 weeks after completion of intervention. The primary outcome measure was the change in FMA scores at completion of intervention.

RESULTS

The mean age was 57.5±9.8 years, and subjects were enrolled at a mean of 13.7±8.9 days after stroke. The mean baseline FMA score was 16.4±14.2. There was no difference in FMA scores between all 3 groups at the end of intervention, and at 4 and 8 weeks after completion of intervention. Similar findings were also noted for the secondary outcome measures.

CONCLUSION(S)

Twelve sessions of augmented upper limb exercises via NW gaming or conventional therapy over a 3-week period was not effective in enhancing upper limb motor recovery compared to control.

[Social cognition in patients with mood disorders. Part II: bipolar disorder : a selective review of the literature]

OBJECTIVE

Overview on the current knowledge regarding social cognition in patients with bipolar disorder.

METHODS

Selective literature research on deficits in social cognition intrinsic to bipolar disorder, their occurrence and effects.

RESULTS

Deficits in social cognition are considered to be core features of bipolar disorder. They are apparent during acute episodes of the disorder, endure when patients are in remission and have a significant negative impact on the patients' psychosocial outcomes.

CONCLUSIONS

It is important to consider deficits in social cognition as an integral part of a treatment approach to achieve mental stabilization in patients with bipolar disorder.

The effects of overall robot shape on the emotions invoked in users and the perceived personalities of robot

The affective interaction between human and robots could be influenced by various aspects of robots, which are appearance, countenance, gesture, voice, etc. Among these, the overall shape of robot could play a key role in invoking desired emotions to the users and bestowing preferred personalities to robots. In this regard, the present study experimentally investigates the effects of overall robot shape on the emotions invoked in users and the perceived personalities of robot with an objective of deriving guidelines for the affective design of service robots. In so doing, 27 different shapes of robot were selected, modeled and fabricated, which were combinations of three different shapes of head, trunk and limb (legs and arms) - rectangular-parallelepiped, cylindrical and human-like shapes. For the experiment, visual images and real prototypes of these robot shapes were presented to participants, and emotions invoked and personalities perceived from the presented robots were measured. The results showed that the overall shape of robot arouses any of three emotions named 'concerned', 'enjoyable' and 'favorable', among which 'concerned' emotion is negatively correlated with the 'big five personality factors' while 'enjoyable' and 'favorable' emotions are positively correlated. It was found that the 'big five personality factors', and 'enjoyable' and 'favorable' emotions are more strongly perceived through the real prototypes than through the visual images. It was also found that the robot shape consisting of cylindrical head, human-like trunk and cylindrical head is the best for 'conscientious' personality and 'favorable' emotion, the robot shape consisting of cylindrical head, human-like trunk and human-like limb for 'extroverted' personality, the robot shape consisting of cylindrical head, cylindrical trunk and cylindrical limb for 'anti-neurotic' personality, and the robot shape consisting of rectangular-parallelepiped head, human-like trunk and human-like limb for 'enjoyable' emotion.

A virtual reality application in role-plays of social skills training for schizophrenia: a randomized, controlled trial

Although social skills training (SST) is an effective approach for improving social skills for schizophrenia, the motivational deficit attenuates its efficacy. Virtual reality (VR) applications have allowed individuals with mental disabilities to enhance their motivation for rehabilitation. We compared SST using VR role-playing (SST-VR) to SST using traditional role-playing (SST-TR). This randomized, controlled trial included 91 inpatients with schizophrenia who were assigned to either SST-VR (n=46) or SST-TR (n=45). Both groups were administered over 10 semiweekly group sessions. An experienced, blinded rater assessed vocal, nonverbal and conversational skills. We also obtained data on motivation for SST and various social abilities. Throughout the 10 sessions, the SST-VR group (n=33) showed greater interest in SST and generalization of the skills than the SST-TR group (n=31). After SST, the SST-VR group improved more in conversational skills and assertiveness than the SST-TR group, but less in nonverbal skills. The VR application in role-plays of SST for schizophrenia may be particularly beneficial in terms of improving the conversational skills and assertiveness, possibly through its advantages in enhancing motivation for SST and generalization of the skills, and thus it may be a useful supplement to traditional SST.

Judgment of the humanness of an interlocutor is in the eye of the beholder

Despite tremendous advances in artificial language synthesis, no machine has so far succeeded in deceiving a human. Most research focused on analyzing the behavior of "good" machine. We here choose an opposite strategy, by analyzing the behavior of "bad" humans, i.e., humans perceived as machine. The Loebner Prize in Artificial Intelligence features humans and artificial agents trying to convince judges on their humanness via computer-mediated communication. Using this setting as a model, we investigated here whether the linguistic behavior of human subjects perceived as non-human would enable us to identify some of the core parameters involved in the judgment of an agents' humanness. We analyzed descriptive and semantic aspects of dialogues in which subjects succeeded or failed to convince judges of their humanness. Using cognitive and emotional dimensions in a global behavioral characterization, we demonstrate important differences in the patterns of behavioral expressiveness of the judges whether they perceived their interlocutor as being human or machine. Furthermore, the indicators of interest displayed by the judges were predictive of the final judgment of humanness. Thus, we show that the judgment of an interlocutor's humanness during a social interaction depends not only on his behavior, but also on the judge himself. Our results thus demonstrate that the judgment of humanness is in the eye of the beholder.

Reduced activation in the mirror neuron system during a virtual social cognition task in euthymic bipolar disorder

Social cognition entails both cognitive and affective processing, and impairments in both have accounted for residual symptoms of bipolar disorder (BD). However, there has been a lack of studies identifying neural substrates responsible for social cognitive difficulties in BD patients. Fourteen euthymic BD patients and 14 healthy normal controls underwent functional MRI while performing a virtual reality social cognition task, which incorporated both cognitive and emotional dimensions, simulating real-world social situations. During the scanning, subjects tried to guess (attribute) possible reasons for expressed emotion of virtual humans (avatars) while viewing their facial expressions, just after observing their verbal and nonverbal (facial) expressions which were emotionally valenced (happy, angry and neutral). BD patients compared to normal controls showed delayed reaction times in emotional conditions, with comparable response accuracy. Healthy normal controls activated the right anterior cingulate cortex, inferior frontal, and insular cortex in emotional conditions contrasted with neutral control conditions, that is, the regions that have been related to empathic processes during viewing others' emotional expression. Relative to normal controls, BD patients showed reduced activations in the 'mirror neuron system', including the right inferior frontal cortex, premotor cortex, and insula, mainly in angry or happy condition. These results may suggest that, even during euthymic state, BD patients have difficulties in recruiting brain regions for the utilization of emotional cues as a means for understanding others. Clinical attention should be paid to emotion-related residual symptoms to help improve social outcomes in these patients.

Increased personal space of patients with schizophrenia in a virtual social environment

Virtual reality may be a good alternative method for measuring personal space and overcoming some limitations in previous studies on the social aspects of schizophrenia. Using this technology, we aimed to investigate the characteristics of personal space in patients with schizophrenia and evaluate the relationship between their social behaviors and schizophrenic symptoms. The distance from a virtual person and the angle of head orientation while talking to a virtual person in a virtual environment were measured in 30 patients with schizophrenia and 30 normal controls. It was found that patients with schizophrenia had longer distances and larger angles than did normal controls. The severity of the negative syndrome had significant inverse correlations with the distance from the angry and neutral virtual persons and with the angle of head orientation toward the happy and angry virtual persons, suggesting that negative symptoms may have a close relationship with personal space, including distancing and eye gaze. The larger personal space of patients may reflect their discomfort in close situations or cognitive deficits. Showing these profiles to patients could help them realize the amount of personal space they need.

Nonverbal social behaviors of patients with bipolar mania during interactions with virtual humans

It has been proposed that positive emotional biases could make bipolar manic (BM) patients maintain abnormally approaching behaviors during social interactions. To test this hypothesis, we measured interpersonal distance (IPD) and gaze angle of BM patients and normal controls (NCs) during social interaction in immersive virtual environment. Overall, IPDs of BM patients (n = 20) were greater than those of normal controls (n = 20). The IPD difference was even greater between NCs and BM patients with psychotic features (n = 11) than those without psychotic features (n = 9). Regardless of the presence of psychotic features, BM patients averted their gazes more than NCs, and even more while speaking than while listening. Our results might suggest negativistic social cognition of bipolar patients, as was previously found even during a manic phase, or the role of paranoid symptoms in avoidant social behaviors, in agreement with prior studies with schizophrenic patients. Use of proper space and gaze might have psychotherapeutic implication in developing secure, two-person relationship with bipolar patients regardless of the presence of disrupting manic symptoms.

Duration matters: dissociating neural correlates of detection and evaluation of social gaze

The interpretation of interpersonal gaze behavior requires the use of complex cognitive processes and guides social interactions. Among a variety of different gaze characteristics, gaze direction and gaze duration modulate crucially the meaning of the "social gaze". Nevertheless, prior neuroimaging studies disregarded the relevance of gaze duration by focusing on gaze direction only. The present functional magnetic resonance imaging (fMRI) study focused on the differentiation of these two gaze parameters. Therefore direct gaze displayed by virtual characters was contrasted with averted gaze and, additionally, systematically varied with respect to gaze duration (i.e., 1, 2.5 or 4 s). Consistent with prior findings, behavioral data showed that likeability was higher for direct than for averted gaze and increased linearly with increasing direct gaze duration. On the neural level, distinct brain regions were associated with the processing of gaze direction and gaze duration: (i) the comparison between direct and averted gaze revealed activations in bilateral occipito-temporal regions including the posterior superior temporal sulcus (pSTS); (ii) whereas increasing duration of direct gaze evoked differential neural responses in the medial prefrontal cortex (MPFC) including orbitofrontal and paracingulate regions. The results suggest two complementary cognitive processes related to different gaze parameters. On the one hand, the recruitment of multimodal sensory regions in the pSTS indicates detection of gaze direction via complex visual analysis. On the other hand, the involvement of the MPFC associated with outcome monitoring and mentalizing indicates higher-order social cognitive processes related to evaluation of the ongoing communicational input conveyed by direct gaze duration.

Recognition profile of emotions in natural and virtual faces

Background

Computer-generated virtual faces become increasingly realistic including the simulation of emotional expressions. These faces can be used as well-controlled, realistic and dynamic stimuli in emotion research. However, the validity of virtual facial expressions in comparison to natural emotion displays still needs to be shown for the different emotions and different age groups.

Methodology/Principal Findings

Thirty-two healthy volunteers between the age of 20 and 60 rated pictures of natural human faces and faces of virtual characters (avatars) with respect to the expressed emotions: happiness, sadness, anger, fear, disgust, and neutral. Results indicate that virtual emotions were recognized comparable to natural ones. Recognition differences in virtual and natural faces depended on specific emotions: whereas disgust was difficult to convey with the current avatar technology, virtual sadness and fear achieved better recognition results than natural faces. Furthermore, emotion recognition rates decreased for virtual but not natural faces in participants over the age of 40. This specific age effect suggests that media exposure has an influence on emotion recognition.

Conclusions/Significance

Virtual and natural facial displays of emotion may be equally effective. Improved technology (e.g. better modelling of the naso-labial area) may lead to even better results as compared to trained actors. Due to the ease with which virtual human faces can be animated and manipulated, validated artificial emotional expressions will be of major relevance in future research and therapeutic applications.

Amygdala activation at 3T in response to human and avatar facial expressions of emotions

Facial expressions of emotions are important in nonverbal communication. Although numerous neural structures have been identified to be involved in emotional face processing, the amygdala is thought to be a core moderator. While previous studies have relied on facial images of humans, the present study is concerned with the effect of computer-generated (avatar) emotional faces on amygdala activation. Moreover, elicited activation patterns in response to viewing avatar faces are compared with the neuronal responses to human facial expressions of emotions. Twelve healthy subjects (five females) performed facial emotion recognition tasks with optimized 3T event-related fMRI. Robust amygdala activation was apparent in response to both human and avatar emotional faces, but the response was significantly stronger to human faces in face-sensitive structures, i.e. fusiform gyri. We suggest that avatars could be a useful tool in neuroimaging studies of facial expression processing because they elicit amygdala activation similarly to human faces, yet have the advantage of being highly manipulable and fully controllable. However, the finding of differences between human and avatar faces in face-sensitive regions indicates the presence of mechanisms by which human brains can differentiate between them. This mechanism merits further investigation.