Facial Expression Recognition for Measuring Jurors' Attention in Acoustic Jury Tests

The perception of sound greatly impacts users' emotional states, expectations, affective relationships with products, and purchase decisions. Consequently, assessing the perceived quality of sounds through jury testing is crucial in product design. However, the subjective nature of jurors' responses may limit the accuracy and reliability of jury test outcomes. This research explores the utility of facial expression analysis in jury testing to enhance response reliability and mitigate subjectivity. Some quantitative indicators allow the research hypothesis to be validated, such as the correlation between jurors' emotional responses and valence values, the accuracy of jury tests, and the disparities between jurors' questionnaire responses and the emotions measured by FER (facial expression recognition). Specifically, analysis of attention levels during different statuses reveals a discernible decrease in attention levels, with 70 percent of jurors exhibiting reduced attention levels in the 'distracted' state and 62 percent in the 'heavy-eyed' state. On the other hand, regression analysis shows that the correlation between jurors' valence and their choices in the jury test increases when considering the data where the jurors are attentive. The correlation highlights the potential of facial expression analysis as a reliable tool for assessing juror engagement. The findings suggest that integrating facial expression recognition can enhance the accuracy of jury testing in product design by providing a more dependable assessment of user responses and deeper insights into participants' reactions to auditory stimuli.

Word recognition through mapping of lip movements from speech utterance using audiovisual fusion and MLP

Speech has information more than text, but under noisy environment speech sufferance from disadvantage of not properly decoded by humans and same is true with machines. speech being bimodal along with audio features if we augment visual features specifically related to lip movements. the degree of speech recognition can be improved. The objective of this work is to use audio and visual features to aid word recognition. In this work we extracted MFCC features for audio and Geometrical features of lip movements together is used in machine learning algorithm to predict the word utterances. Videos related to word utterances are extracted from TIMID database. With the statistical information related to audio and corresponding visual features from lip movements is extracted to form input feature vector to machine learning algorithm (Multi-layer perceptron). The experimental results show that using MLP we have obtained a word recognition accuracy of 91% and using KNN Classifier the accuracy attained is 61%. The results presented here have important implications for applications in HMI communication and helps hearing impaired.

Does One Size Fit All? A Case Study to Discuss Findings of an Augmented Hands-Free Robot Teleoperation Concept for People with and without Motor Disabilities

Hands-free robot teleoperation and augmented reality have the potential to create an inclusive environment for people with motor disabilities. It may allow them to teleoperate robotic arms to manipulate objects. However, the experiences evoked by the same teleoperation concept and augmented reality can vary significantly for people with motor disabilities compared to those without disabilities. In this paper, we report the experiences of Miss L., a person with multiple sclerosis, when teleoperating a robotic arm in a hands-free multimodal manner using a virtual menu and visual hints presented through the Microsoft HoloLens 2. We discuss our findings and compare her experiences to those of people without disabilities using the same teleoperation concept. Additionally, we present three learning points from comparing these experiences: a re-evaluation of the metrics used to measure performance, being aware of the bias, and considering variability in abilities, which evokes different experiences. We consider these learning points can be extrapolated to carrying human–robot interaction evaluations with mixed groups of participants with and without disabilities.

Improving Cross-Corpus Speech Emotion Recognition with Adversarial Discriminative Domain Generalization (ADDoG)

Automatic speech emotion recognition provides computers with critical context to enable user understanding. While methods trained and tested within the same dataset have been shown successful, they often fail when applied to unseen datasets. To address this, recent work has focused on adversarial methods to find more generalized representations of emotional speech. However, many of these methods have issues converging, and only involve datasets collected in laboratory conditions. In this paper, we introduce Adversarial Discriminative Domain Generalization (ADDoG), which follows an easier to train "meet in the middle" approach. The model iteratively moves representations learned for each dataset closer to one another, improving cross-dataset generalization. We also introduce Multiclass ADDoG, or MADDoG, which is able to extend the proposed method to more than two datasets, simultaneously. Our results show consistent convergence for the introduced methods, with significantly improved results when not using labels from the target dataset. We also show how, in most cases, ADDoG and MADDoG can be used to improve upon baseline state-of-the-art methods when target dataset labels are added and in-the-wild data are considered. Even though our experiments focus on cross-corpus speech emotion, these methods could be used to remove unwanted factors of variation in other settings.

A Computational Study of Expressive Facial Dynamics in Children with Autism

Several studies have established that facial expressions of children with autism are often perceived as atypical, awkward or less engaging by typical adult observers. Despite this clear deficit in the quality of facial expression production, very little is understood about its underlying mechanisms and characteristics. This paper takes a computational approach to studying details of facial expressions of children with high functioning autism (HFA). The objective is to uncover those characteristics of facial expressions, notably distinct from those in typically developing children, and which are otherwise difficult to detect by visual inspection. We use motion capture data obtained from subjects with HFA and typically developing subjects while they produced various facial expressions. This data is analyzed to investigate how the overall and local facial dynamics of children with HFA differ from their typically developing peers. Our major observations include reduced complexity in the dynamic facial behavior of the HFA group arising primarily from the eye region.

Mona Lisa is always happy - and only sometimes sad

The worldwide fascination of da Vinci's Mona Lisa has been dedicated to the emotional ambiguity of her face expression. In the present study we manipulated Mona Lisa's mouth curvature as one potential source of ambiguity and studied how a range of happier and sadder face variants influences perception. In two experimental conditions we presented different stimulus ranges with different step sizes between stimuli along the happy-sad axis of emotional face expressions. Stimuli were presented in random order and participants indicated the perceived emotional face expression (first task) and the confidence of their response (second task). The probability of responding 'happy' to the original Mona Lisa was close to 100%. Furthermore, in both conditions the perceived happiness of Mona Lisa variants described sigmoidal functions of the mouth curvature. Participants' confidence was weakest around the sigmoidal inflection points. Remarkably, the sigmoidal functions, as well as confidence values and reaction times, differed significantly between experimental conditions. Finally, participants responded generally faster to happy than to sad faces. Overall, the original Mona Lisa seems to be less ambiguous than expected. However, perception of and reaction to the emotional face content is relative and strongly depends on the used stimulus range.

Effects of the Lee Silverman Voice Treatment (LSVT® LOUD) on hypomimia in Parkinson's disease

Given associations between facial movement and voice, the potential of the Lee Silverman Voice Treatment (LSVT) to alleviate decreased facial expressivity, termed hypomimia, in Parkinson's disease (PD) was examined. Fifty-six participants--16 PD participants who underwent LSVT, 12 PD participants who underwent articulation treatment (ARTIC), 17 untreated PD participants, and 11 controls without PD--produced monologues about happy emotional experiences at pre- and post-treatment timepoints ("T1" and "T2," respectively), 1 month apart. The groups of LSVT, ARTIC, and untreated PD participants were matched on demographic and health status variables. The frequency and variability of facial expressions (Frequency and Variability) observable on 1-min monologue videorecordings were measured using the Facial Action Coding System (FACS). At T1, the Frequency and Variability of participants with PD were significantly lower than those of controls. Frequency and Variability increases of LSVT participants from T1 to T2 were significantly greater than those of ARTIC or untreated participants. Whereas the Frequency and Variability of ARTIC participants at T2 were significantly lower than those of controls, LSVT participants did not significantly differ from controls on these variables at T2. The implications of these findings, which suggest that LSVT reduces parkinsonian hypomimia, for PD-related psychosocial problems are considered.

Behavioral Signal Processing: Deriving Human Behavioral Informatics From Speech and Language: Computational techniques are presented to analyze and model expressed and perceived human behavior-variedly characterized as typical, atypical, distressed, and disordered-from speech and language cues and their applications in health, commerce, education, and beyond

The expression and experience of human behavior are complex and multimodal and characterized by individual and contextual heterogeneity and variability. Speech and spoken language communication cues offer an important means for measuring and modeling human behavior. Observational research and practice across a variety of domains from commerce to healthcare rely on speech- and language-based informatics for crucial assessment and diagnostic information and for planning and tracking response to an intervention. In this paper, we describe some of the opportunities as well as emerging methodologies and applications of human behavioral signal processing (BSP) technology and algorithms for quantitatively understanding and modeling typical, atypical, and distressed human behavior with a specific focus on speech- and language-based communicative, affective, and social behavior. We describe the three important BSP components of acquiring behavioral data in an ecologically valid manner across laboratory to real-world settings, extracting and analyzing behavioral cues from measured data, and developing models offering predictive and decision-making support. We highlight both the foundational speech and language processing building blocks as well as the novel processing and modeling opportunities. Using examples drawn from specific real-world applications ranging from literacy assessment and autism diagnostics to psychotherapy for addiction and marital well being, we illustrate behavioral informatics applications of these signal processing techniques that contribute to quantifying higher level, often subjectively described, human behavior in a domain-sensitive fashion.

A NEW APPROACH TO EMOTION RECOGNITION FROM THE LIP CONTOUR OF A SUBJECT

This paper proposes an alternative approach to emotion recognition from the outer lip contour of the subjects. Subjects exhibit their emotions through their facial expressions, and the lip region is segmented from their facial images. A lip contour model has been developed, and the parameters of the model are adapted using differential evolution algorithm to match with the actual outer contour of the lip. An SVM classifier is then employed to classify the emotion of the subject from the parameter set of the subjects' lip contour. The experiment was performed on 50 subjects in the age group 20–30 years, and the worst accuracy in emotion classification is found to be 86%.

Emotion-Aware Assistive System for Humanistic Care Based on the Orange Computing Concept

Mental care has become crucial with the rapid growth of economy and technology. However, recent movements, such as green technologies, place more emphasis on environmental issues than on mental care. Therefore, this study presents an emerging technology called orange computing for mental care applications. Orange computing refers to health, happiness, and physiopsychological care computing, which focuses on designing algorithms and systems for enhancing body and mind balance. The representative color of orange computing originates from a harmonic fusion of passion, love, happiness, and warmth. A case study on a human-machine interactive and assistive system for emotion care was conducted in this study to demonstrate the concept of orange computing. The system can detect emotional states of users by analyzing their facial expressions, emotional speech, and laughter in a ubiquitous environment. In addition, the system can provide corresponding feedback to users according to the results. Experimental results show that the system can achieve an accurate audiovisual recognition rate of 81.8% on average, thereby demonstrating the feasibility of the system. Compared with traditional questionnaire-based approaches, the proposed system can offer real-time analysis of emotional status more efficiently.

Psychiatric disorders in a patient with persistent developmental stuttering

Persistent developmental stuttering (PDS) is a common disorder of speech with no identifiable cause. Psychiatric disorders appear to be related and influence clinical manifestation of PDS. In this case report, we present the clinical evolution of 1 PDS patient submitted to pharmacological treatment with fluoxetine and speech therapy intervention. At the end of 12 weeks of treatment, she evolved from 28 at Beck Depression Inventory; 32 in the Hamilton Scale for Anxiety; 43 and 47, respectively, in the anxiety and avoidance components of the Liebowitz Social Anxiety Scale; and severe speech impairment according Iowa Scale, to 12 at Beck Depression Inventory; 8 at Hamilton Scale for Anxiety; 25 and 21 at Liebowitz Social Anxiety Scale anxiety and avoidance components, respectively; and moderate speech impairment. Diagnosing and treating psychiatric symptoms in addition to speech therapy appears to be the best therapeutic approach.

The contribution of muscular and auditory pathologies to the symptomatology of autism

Most research concerning the pathology of autism is focused on the search for central abnormalities that account for the production of symptoms. We, however, instead of looking at muscular and auditory features as merely associated manifestations, propose that they are somatic contributors by which some of the main clinical features of autism might be explained. Evidence suggests that muscles affect emotional experience. We think certain muscular dysfunctioning can impair communication and social interaction, and create stereotypic behavior, giving rise to the diagnostic features of autism. Furthermore, because speech is synchronized with facial movements and voice is controlled mainly through auditory feedback, a distortion of auditory feedback could disrupt voice, which in turn might cause parallel abnormal facial muscle functioning.