Stress and anxiety detection using facial cues from videos

Enhancing the Horse Grimace Scale (HGS): Proposed updates and anatomical descriptors for pain assessment

The use of grimace scales enables the clinical identification of changes in the facial expressions of animals caused by pain. The Horse Grimace Scale (HGS) is one such tool, comprising a pain coding system based on facial expressions and assessing six Facial Action Units (FAUs). Each FAU is accompanied by descriptions and anatomical details to assist the evaluator. However, the morphological descriptions for certain FAUs in the HGS are not sufficiently detailed, potentially hindering accurate interpretation. This study is an analytical investigation aimed at enhancing the morphoanatomical details in the HGS and providing raters with more comprehensive materials for pain evaluation in horses using this scale. To achieve this, detailed anatomical analyses were conducted using established references in veterinary anatomy. Initially, we propose substituting the term 'ear' with 'auricle' or 'pinna' and replacing 'area above the eye' with 'supraorbital region' for anatomical accuracy. Additionally, we introduce detailed morphoanatomical descriptions that identify specific landmarks, with the goal of ensuring more consistent application of the HGS and reducing interpretation variability. Furthermore, this study provides an explanation of the muscles involved in the investigated FAUs. These adjustments on the descriptions and evaluations remain unverified, however it is anticipated that the descriptive enhancements lead us to understand that higher interobserver reliability can be achieved for each of the FAUs.

Machine Learning, Deep Learning, and Data Preprocessing Techniques for Detecting, Predicting, and Monitoring Stress and Stress-Related Mental Disorders: Scoping Review

BACKGROUND

Mental stress and its consequent mental health disorders (MDs) constitute a significant public health issue. With the advent of machine learning (ML), there is potential to harness computational techniques for better understanding and addressing mental stress and MDs. This comprehensive review seeks to elucidate the current ML methodologies used in this domain to pave the way for enhanced detection, prediction, and analysis of mental stress and its subsequent MDs.

OBJECTIVE

This review aims to investigate the scope of ML methodologies used in the detection, prediction, and analysis of mental stress and its consequent MDs.

METHODS

Using a rigorous scoping review process with PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews) guidelines, this investigation delves into the latest ML algorithms, preprocessing techniques, and data types used in the context of stress and stress-related MDs.

RESULTS

A total of 98 peer-reviewed publications were examined for this review. The findings highlight that support vector machine, neural network, and random forest models consistently exhibited superior accuracy and robustness among all ML algorithms examined. Physiological parameters such as heart rate measurements and skin response are prevalently used as stress predictors due to their rich explanatory information concerning stress and stress-related MDs, as well as the relative ease of data acquisition. The application of dimensionality reduction techniques, including mappings, feature selection, filtering, and noise reduction, is frequently observed as a crucial step preceding the training of ML algorithms.

CONCLUSIONS

The synthesis of this review identified significant research gaps and outlines future directions for the field. These encompass areas such as model interpretability, model personalization, the incorporation of naturalistic settings, and real-time processing capabilities for the detection and prediction of stress and stress-related MDs.

Spontaneous eye blinks in horses (Equus caballus) are modulated by attention

Spontaneous eye blinks are brief closures of both eyelids. The spontaneous eye blink rate (SEBR) exceeds physiological corneal needs and is modulated by emotions and cognitive states, including vigilance and attention, in humans. In several animal species, the SEBR is modulated by stress and antipredator vigilance, which may limit the loss of visual information due to spontaneous eye closing. Here, we investigated whether the SEBR is modulated by attention in the domestic horse (Equus caballus). Our data supported previous studies indicating a tonic SEBR specific to each individual. We also found that, superimposed on a tonic SEBR, phasic changes were induced by cognitive processing. Attention downmodulated the SEBR, with the magnitude of blink inhibition proportional to the degree of attentional selectivity. On the other hand, reward anticipation upregulated the SEBR. Our data also suggested that horses possess the cognitive property of object permanence: they understand that an object that is no longer in their visual field has not ceased to exist. In conclusion, our results suggested that spontaneous eye blinks in horses are modulated by attentional cognitive processing.

Driving Stress-Induced Effects on the Orofacial Region and Its Functions and Health Behaviors in Riyadh: A Cross-Sectional Survey

Driving stress is a multifaceted phenomenon, and the experience of driving invokes stress. Driving causes the activation of stress-response mechanisms, leading to short-term and long-term stress responses resulting in physiological and behavioral changes. The aim of this study was to evaluate driving stress-initiated effects on orofacial functions and health behaviors in the Riyadh population. A cross-sectional survey was conducted in Riyadh using a pre-validated set of questionnaires for habitual information, a driving stress assessment using a driving-behavior inventory, and an assessment of parafunctional habits and effects on orofacial functions. The results indicate that nearly 50% of the sample spends more than two hours commuting, and more than 50% of the sample has inadequate sleep and insufficient exercise. Oral parafunctional habits like nail biting (p = 0.039) and lip or object biting (p = 0.029) had a significant correlation with aggressive driving behaviors, whereas the grinding of teeth (p = 0.011), the clenching of jaws (p = 0.048), lip or object biting (p = 0.018), and pain in mastication (p = 0.036) had a positive correlation with driving dislikes. Driving stress can be detrimental to one's health and not only impacts health behaviors but also induces oral parafunctional habits and adversely affects orofacial regions and functions. Acute driving stress responses may be transient. However, prolonged driving stress can be maladaptive and can increase the risk of chronic diseases including chronic temporomandibular joint disorders and parafunctional habit-related changes in the oral cavity.

Enhancing Arousal Level Detection in EEG Signals through Genetic Algorithm-based Feature Selection and Fast Bit Hopping

Background

This study explores a novel approach to detecting arousal levels through the analysis of electroencephalography (EEG) signals. Leveraging the Faller database with data from 18 healthy participants, we employ a 64-channel EEG system.

Methods

The approach we employ entails the extraction of ten frequency characteristics from every channel, culminating in a feature vector of 640 dimensions for each signal instance. To enhance classification accuracy, we employ a genetic algorithm for feature selection, treating it as a multiobjective optimization task. The approach utilizes fast bit hopping for efficiency, overcoming traditional bit-string limitations. A hybrid operator expedites algorithm convergence, and a solution selection strategy identifies the most suitable feature subset.

Results

Experimental results demonstrate the method's effectiveness in detecting arousal levels across diverse states, with improvements in accuracy, sensitivity, and specificity. In scenario one, the proposed method achieves an average accuracy, sensitivity, and specificity of 93.11%, 98.37%, and 99.14%, respectively. In scenario two, the averages stand at 81.35%, 88.65%, and 84.64%.

Conclusions

The obtained results indicate that the proposed method has a high capability of detecting arousal levels in different scenarios. In addition, the advantage of employing the proposed feature reduction method has been demonstrated.

A Novel AI Approach for Assessing Stress Levels in Patients with Type 2 Diabetes Mellitus Based on the Acquisition of Physiological Parameters Acquired during Daily Life

Stress is the inherent sensation of being unable to handle demands and occurrences. If not properly managed, stress can develop into a chronic condition, leading to the onset of additional chronic health issues, such as cardiovascular illnesses and diabetes. Various stress meters have been suggested in the past, along with diverse approaches for its estimation. However, in the case of more serious health issues, such as hypertension and diabetes, the results can be significantly improved. This study presents the design and implementation of a distributed wearable-sensor computing platform with multiple channels. The platform aims to estimate the stress levels in diabetes patients by utilizing a fuzzy logic algorithm that is based on the assessment of several physiological indicators. Additionally, a mobile application was created to monitor the users' stress levels and integrate data on their blood pressure and blood glucose levels. To obtain better performance metrics, validation experiments were carried out using a medical database containing data from 128 patients with chronic diabetes, and the initial results are presented in this study.

Non-invasive detection of mental fatigue in construction equipment operators through geometric measurements of facial features

INTRODUCTION

Prolonged operation of construction equipment could lead to mental fatigue, which can increase the chances of human error-related accidents as well as operators' ill-health. The objective detection of operators' mental fatigue is crucial for reducing accident risk and ensuring operator health. Electroencephalography, photoplethysmography, electrodermal activity, and eye-tracking technology have been used to mitigate this issue. These technologies are invasive and wearable sensors that can cause irritation and discomfort. Geometric measurements of facial features can serve as a noninvasive alternative approach. Its application in detecting mental fatigue of construction equipment operators has not been reported in the literature. Although the application of facial features has been widespread in other domains, such as drivers and other occupation scenarios, their ecological validity for construction excavator operators remains a knowledge gap.

METHOD

This study proposed employing geometric measurements of facial features to detect mental fatigue in construction equipment operators' facial features. In this study, seventeen operators performed excavation operations. Mental fatigue was labeled subjectively and objectively using NASA-TLX scores and EDA values. Based on geometric measurements, facial features (eyebrow, mouth outer, mouth corners, head motion, eye area, and face area) were extracted.

RESULTS

The results showed that there was significant difference in the measured metrics for high fatigue compared to low fatigue. Specifically, the most noteworthy variation was for the eye and face area metrics, with mean differences of 45.88% and 26.9%, respectively.

CONCLUSIONS

The findings showed that geometrical measurements of facial features are a useful, noninvasive approach for detecting the mental fatigue of construction equipment operators.

A Deep Learning-Based Platform for Workers' Stress Detection Using Minimally Intrusive Multisensory Devices

The advent of Industry 4.0 necessitates substantial interaction between humans and machines, presenting new challenges when it comes to evaluating the stress levels of workers who operate in increasingly intricate work environments. Undoubtedly, work-related stress exerts a significant influence on individuals' overall stress levels, leading to enduring health issues and adverse impacts on their quality of life. Although psychological questionnaires have traditionally been employed to assess stress, they lack the capability to monitor stress levels in real-time or on an ongoing basis, thus making it arduous to identify the causes and demanding aspects of work. To surmount this limitation, an effective solution lies in the analysis of physiological signals that can be continuously measured through wearable or ambient sensors. Previous studies in this field have mainly focused on stress assessment through intrusive wearable systems susceptible to noise and artifacts that degrade performance. One of our recently published papers presented a wearable and ambient hardware-software platform that is minimally intrusive, able to detect human stress without hindering normal work activities, and slightly susceptible to artifacts due to movements. A limitation of this system is its not very high performance in terms of the accuracy of detecting multiple stress levels; therefore, in this work, the focus was on improving the software performance of the platform, using a deep learning approach. To this purpose, three neural networks were implemented, and the best performance was achieved by the 1D-convolutional neural network with an accuracy of 95.38% for the identification of two levels of stress, which is a significant improvement over those obtained previously.

Internet-based identification of anxiety in university students using text and facial emotion analysis

Background

Anxiety in university students can lead to poor academic performance and even dropout. The Adult Manifest Anxiety Scale (AMAS-C) is a validated measure designed to assess the level and nature of anxiety in college students.

Objective

The aim of this study is to provide internet-based alternatives to the AMAS-C in the automated identification and prediction of anxiety in young university students. Two anxiety prediction methods, one based on facial emotion recognition and the other on text emotion recognition, are described and validated using the AMAS-C Test Anxiety, Lie and Total Anxiety scales as ground truth data.

Methods

The first method analyses facial expressions, identifying the six basic emotions (anger, disgust, fear, happiness, sadness, surprise) and the neutral expression, while the students complete a technical skills test. The second method examines emotions in posts classified as positive, negative and neutral in the students' profile on the social network Facebook. Both approaches aim to predict the presence of anxiety.

Results

Both methods achieved a high level of precision in predicting anxiety and proved to be effective in identifying anxiety disorders in relation to the AMAS-C validation tool. Text analysis-based prediction showed a slight advantage in terms of precision (86.84 %) in predicting anxiety compared to face analysis-based prediction (84.21 %).

Conclusions

The applications developed can help educators, psychologists or relevant institutions to identify at an early stage those students who are likely to fail academically at university due to an anxiety disorder.

Unlocking the Emotional World of Visual Media: An Overview of the Science, Research, and Impact of Understanding Emotion: Drawing Insights From Psychology, Engineering, and the Arts, This Article Provides a Comprehensive Overview of the Field of Emotion Analysis in Visual Media and Discusses the Latest Research, Systems, Challenges, Ethical Implications, and Potential Impact of Artificial Emotional Intelligence on Society

The emergence of artificial emotional intelligence technology is revolutionizing the fields of computers and robotics, allowing for a new level of communication and understanding of human behavior that was once thought impossible. While recent advancements in deep learning have transformed the field of computer vision, automated understanding of evoked or expressed emotions in visual media remains in its infancy. This foundering stems from the absence of a universally accepted definition of "emotion," coupled with the inherently subjective nature of emotions and their intricate nuances. In this article, we provide a comprehensive, multidisciplinary overview of the field of emotion analysis in visual media, drawing on insights from psychology, engineering, and the arts. We begin by exploring the psychological foundations of emotion and the computational principles that underpin the understanding of emotions from images and videos. We then review the latest research and systems within the field, accentuating the most promising approaches. We also discuss the current technological challenges and limitations of emotion analysis, underscoring the necessity for continued investigation and innovation. We contend that this represents a "Holy Grail" research problem in computing and delineate pivotal directions for future inquiry. Finally, we examine the ethical ramifications of emotion-understanding technologies and contemplate their potential societal impacts. Overall, this article endeavors to equip readers with a deeper understanding of the domain of emotion analysis in visual media and to inspire further research and development in this captivating and rapidly evolving field.

Video stimuli suitable for stress estimation based on biosignals

Stress can cause mental disorders such as depression and anxiety disorders. To detect such mental disorders at an early stage, it is necessary to detect stress accurately. One of the effective methods for this purpose is observing changes in biological signals caused by sensory stimuli such as video presentation. This study aims to identify effective video stimuli for stress estimation. We hypothesize that the emotional state evoked by the video stimuli influences the accuracy of stress estimation. To test this hypothesis, we utilized an open video dataset consisting of 444 responses on an emotion scale (valence and arousal) as emotional stimuli. Ninety videos were divided into emotion subsets based on the emotion scale for each video, and biological signals were measured when each video was presented to the subjects. Machine learning models were constructed for each subset, and the prediction errors were compared. The results showed that the prediction error was lower for the high valence and high arousal subsets than for the others. These results suggest that high-valence or high-arousal videos effectively estimate stress.

Assessment of a Person's Emotional State Based on His or Her Posture Parameters

This article is devoted to the study of the correlation between the emotional state of a person and the posture of his or her body in the sitting position. In order to carry out the study, we developed the first version of the hardware-software system based on a posturometric armchair, allowing the characteristics of the posture of a sitting person to be evaluated using strain gauges. Using this system, we revealed the correlation between sensor readings and human emotional states. We showed that certain readings of a sensor group are formed for a certain emotional state of a person. We also found that the groups of triggered sensors, their composition, their number, and their location are related to the states of a particular person, which led to the need to build personalized digital pose models for each person. The intellectual component of our hardware-software complex is based on the concept of co-evolutionary hybrid intelligence. The system can be used during medical diagnostic procedures and rehabilitation processes, as well as in controlling people whose professional activity is connected with increased psycho-emotional load and can cause cognitive disorders, fatigue, and professional burnout and can lead to the development of diseases.

Using gait videos to automatically assess anxiety

Background

In recent years, the number of people with anxiety disorders has increased worldwide. Methods for identifying anxiety through objective clues are not yet mature, and the reliability and validity of existing modeling methods have not been tested. The objective of this paper is to propose an automatic anxiety assessment model with good reliability and validity.

Methods

This study collected 2D gait videos and Generalized Anxiety Disorder (GAD-7) scale data from 150 participants. We extracted static and dynamic time-domain features and frequency-domain features from the gait videos and used various machine learning approaches to build anxiety assessment models. We evaluated the reliability and validity of the models by comparing the influence of factors such as the frequency-domain feature construction method, training data size, time-frequency features, gender, and odd and even frame data on the model.

Results

The results show that the number of wavelet decomposition layers has a significant impact on the frequency-domain feature modeling, while the size of the gait training data has little impact on the modeling effect. In this study, the time-frequency features contributed to the modeling, with the dynamic features contributing more than the static features. Our model predicts anxiety significantly better in women than in men (rMale = 0.666, rFemale = 0.763, p < 0.001). The best correlation coefficient between the model prediction scores and scale scores for all participants is 0.725 (p < 0.001). The correlation coefficient between the model prediction scores for odd and even frame data is 0.801~0.883 (p < 0.001).

Conclusion

This study shows that anxiety assessment based on 2D gait video modeling is reliable and effective. Moreover, we provide a basis for the development of a real-time, convenient and non-invasive automatic anxiety assessment method.

Machine learning based approaches for clinical and non-clinical depression recognition and depression relapse prediction using audiovisual and EEG modalities: A comprehensive review

Mental disorders are rapidly increasing each year and have become a major challenge affecting the social and financial well-being of individuals. There is a need for phenotypic characterization of psychiatric disorders with biomarkers to provide a rich signature for Major Depressive Disorder, improving the understanding of the pathophysiological mechanisms underlying these mental disorders. This comprehensive review focuses on depression and relapse detection modalities such as self-questionnaires, audiovisuals, and EEG, highlighting noteworthy publications in the last ten years. The article concentrates on the literature that adopts machine learning by audiovisual and EEG signals. It also outlines preprocessing, feature extraction, and public datasets for depression detection. The review concludes with recommendations that will help improve the reliability of developed models and the determinism of computational intelligence-based systems in psychiatry. To the best of our knowledge, this survey is the first comprehensive review on depression and relapse prediction by self-questionnaires, audiovisual, and EEG-based approaches. The findings of this review will serve as a useful and structured starting point for researchers studying clinical and non-clinical depression recognition and relapse through machine learning-based approaches.

Automated detection of human mental disorder

The pressures of daily life result in a proliferation of terms such as stress, anxiety, and mood swings. These feelings may be developed to depression and more complicated mental problems. Unfortunately, the mood and emotional changes are difficult to notice and considered a disease that must be treated until late. The late diagnosis appears in suicidal intensions and harmful behaviors. In this work, main human observable facial behaviors are detected and classified by a model that has developed to assess a person’s mental health. Haar feature-based cascade is used to extract the features from the detected faces from FER+ dataset. VGG model classifies if the user is normal or abnormal. Then in the case of abnormal, the model predicts if he has depression, anxiety, or other disorder according to the detected facial expression. The required assistance and support can be provided in a timely manner with this prediction. The system has achieved a 95% of overall prediction accuracy.

Virtual reality exposure aided-diagnosis system for anxiety disorders: Long short-term memory architecture for three levels of anxiety recognition

BACKGROUND

The COVID-19 pandemic has resulted in increased psychological pressure on mental health since 2019. The resulting anxiety and stress have permeated every aspect of life during confinement.

OBJECTIVE

To provide psychologists with an unbiased measure that can aid in the preliminary diagnosis of anxiety disorders and be used as an initial treatment in cognitive-behavioral therapy, this article introduces automated recognition of three levels of anxiety.

METHODS

Anxiety was elicited by exposing participants to virtual environments inspired by social situations in reference to the Liebowitz social anxiety scale. Relevant parameters, such as heart rate variability and vasoconstriction were derived from the measurement of the blood volume pulse (BVP) signal.

RESULTS

A long short-term memory architecture achieved an accuracy of approximately 98% on the training and test set.

CONCLUSION

The generated model allowed for careful study of the state of seven phobic participants during virtual reality exposure (VRE).

An Efficient AP-ANN-Based Multimethod Fusion Model to Detect Stress through EEG Signal Analysis

Stress is a universal emotion that every human experiences daily. Psychologists say stress may lead to heart attack, depression, hypertension, strokes, or even sudden death. Many technical explorations like stress detection through facial expression, speech, text, physical behaviors, etc., were explored, but no consensus has been reached on the best method. The advancement in biomedical engineering yielded a rapid development of electroencephalogram (EEG) signal analysis that has inspired the idea of a multimethod fusion approach for the first time which employs multiple techniques such as discrete wavelet transform (DWT) for de-noising, adaptive synthetic sampling (ADASYN) for class balancing, and affinity propagation (AP) as a stratified sampling model along with the artificial neural network (ANN) as the classifier model for human emotion classification. From the EEG recordings of the DEAP dataset, the artifacts are removed, the signal is decomposed using a DWT, and features are extracted and fused to form the feature vector. As the dataset is high-dimensional, feature selection is done and ADASYN is used to address the imbalance of classes resulting in large-scale data. The innovative idea of the proposed system is to perform sampling using affinity propagation as a stratified sampling-based clustering algorithm as it determines the number of representative samples automatically which makes it superior to the K-Means, K-Medoid, that requires the K-value. Those samples are used as inputs to various classification models, the comparison of the AP-ANN, AP-SVM, and AP-RF is done, and their most important five performance metrics such as accuracy, precision, recall, F1-score, and specificity were compared. From our experiment, the AP-ANN model provides better accuracy of 86.8% and greater precision of 85.7%, a higher F1 score of 84.9%, a recall rate of 84.1%, and a specificity value of 89.2% which altogether provides better results than the other existing algorithms.

Audio-Visual Stress Classification Using Cascaded RNN-LSTM Networks

The purpose of this research is to emphasize the importance of mental health and contribute to the overall well-being of humankind by detecting stress. Stress is a state of strain, whether it be mental or physical. It can result from anything that frustrates, incenses, or unnerves you in an event or thinking. Your body's response to a demand or challenge is stress. Stress affects people on a daily basis. Stress can be regarded as a hidden pandemic. Long-term (chronic) stress results in ongoing activation of the stress response, which wears down the body over time. Symptoms manifest as behavioral, emotional, and physical effects. The most common method involves administering brief self-report questionnaires such as the Perceived Stress Scale. However, self-report questionnaires frequently lack item specificity and validity, and interview-based measures can be time- and money-consuming. In this research, a novel method used to detect human mental stress by processing audio-visual data is proposed. In this paper, the focus is on understanding the use of audio-visual stress identification. Using the cascaded RNN-LSTM strategy, we achieved 91% accuracy on the RAVDESS dataset, classifying eight emotions and eventually stressed and unstressed states.

Stressors Length and the Habituation Effect-An EEG Study

The research described in this paper aimed to determine whether people respond differently to short and long stimuli and whether stress stimuli repeated over time evoke a habituation effect. To meet this goal, we performed a cognitive experiment with eight subjects. During this experiment, the subjects were presented with two trays of stress-inducing stimuli (different in length) interlaced with the main tasks. The mean beta power calculated from the EEG signal recorded from the two prefrontal electrodes (Fp1 and Fp2) was used as a stress index. The main results are as follows: (i) we confirmed the previous finding that beta power assessed from the EEG signal recorded from prefrontal electrodes is significantly higher for the STRESS condition compared to NON-STRESS condition; (ii) we found a significant difference in beta power between STRESS conditions that differed in length-the beta power was four times higher for short, compared to long, stress-inducing stimuli; (iii) we did not find enough evidence to confirm (or reject) the hypothesis that stress stimuli repeated over time evoke the habituation effect; although the general trends aggregated over subjects and stressors were negative, their slopes were not statistically significant; moreover, there was no agreement among subjects with respect to the slope of individual trends.

Real-time mental stress detection using multimodality expressions with a deep learning framework

Mental stress is becoming increasingly widespread and gradually severe in modern society, threatening people’s physical and mental health. To avoid the adverse effects of stress on people, it is imperative to detect stress in time. Many studies have demonstrated the effectiveness of using objective indicators to detect stress. Over the past few years, a growing number of researchers have been trying to use deep learning technology to detect stress. However, these works usually use single-modality for stress detection and rarely combine stress-related information from multimodality. In this paper, a real-time deep learning framework is proposed to fuse ECG, voice, and facial expressions for acute stress detection. The framework extracts the stress-related information of the corresponding input through ResNet50 and I3D with the temporal attention module (TAM), where TAM can highlight the distinguishing temporal representation for facial expressions about stress. The matrix eigenvector-based approach is then used to fuse the multimodality information about stress. To validate the effectiveness of the framework, a well-established psychological experiment, the Montreal imaging stress task (MIST), was applied in this work. We collected multimodality data from 20 participants during MIST. The results demonstrate that the framework can combine stress-related information from multimodality to achieve 85.1% accuracy in distinguishing acute stress. It can serve as a tool for computer-aided stress detection.

Body Language Analysis in Healthcare: An Overview

Given the current COVID-19 pandemic, medical research today focuses on epidemic diseases. Innovative technology is incorporated in most medical applications, emphasizing the automatic recognition of physical and emotional states. Most research is concerned with the automatic identification of symptoms displayed by patients through analyzing their body language. The development of technologies for recognizing and interpreting arm and leg gestures, facial features, and body postures is still in its early stage. More extensive research is needed using artificial intelligence (AI) techniques in disease detection. This paper presents a comprehensive survey of the research performed on body language processing. Upon defining and explaining the different types of body language, we justify the use of automatic recognition and its application in healthcare. We briefly describe the automatic recognition framework using AI to recognize various body language elements and discuss automatic gesture recognition approaches that help better identify the external symptoms of epidemic and pandemic diseases. From this study, we found that since there are studies that have proven that the body has a language called body language, it has proven that language can be analyzed and understood by machine learning (ML). Since diseases also show clear and different symptoms in the body, the body language here will be affected and have special features related to a particular disease. From this examination, we discovered that it is possible to specialize the features and language changes of each disease in the body. Hence, ML can understand and detect diseases such as pandemic and epidemic diseases and others.

Non-Contact Negative Mood State Detection Using Reliability-Focused Multi-Modal Fusion Model

Negative mood states include tension, depression, anger, fatigue, and confusion, which represent the weak internal emotions of a human. Negative mood states exert adverse impact on individuals' ability to make rational decisions, which entails the practicable method of negative mood state detection. The most commonly used negative mood state detection methods are based on the psychological scale, which requires additional work and brings inconvenience to the subject in the application scenarios. To overcome this challenge, this paper proposes a novel non-contact negative mood state detection method according to the knowledge of affective computing. The POMS-net model is used to extract temporal-spatial features from visible and infrared thermal videos, and the negative mood state detection is realized using data reliability-focused multi-modal fusion. The proposed method is verified using the HDT-BR dataset collected in the aerospace medicine experiment "Earth-Star II" and the VIRI public dataset. The experimental results on the datasets verify that our method outperforms the comparison methods.

Potentials and Challenges of Pervasive Sensing in the Intensive Care Unit

Patients in critical care settings often require continuous and multifaceted monitoring. However, current clinical monitoring practices fail to capture important functional and behavioral indices such as mobility or agitation. Recent advances in non-invasive sensing technology, high throughput computing, and deep learning techniques are expected to transform the existing patient monitoring paradigm by enabling and streamlining granular and continuous monitoring of these crucial critical care measures. In this review, we highlight current approaches to pervasive sensing in critical care and identify limitations, future challenges, and opportunities in this emerging field.

Psychophysiological reactions of Internet users exposed to fluoride information and disinformation: Protocol for a randomized controlled trial (Preprint)

Background

False messages on the internet continually propagate possible adverse effects of fluoridated oral care products and water, despite their essential role in preventing and controlling dental caries.

Objective

This study aims to evaluate the patterns of psychophysiological reactions of adults after the consumption of internet-based fluoride-related information and disinformation.

Methods

A 2-armed, single-blinded, parallel, and randomized controlled trial will be conducted with 58 parents or caregivers of children who attend the Clinics of Pediatric Dentistry at the Bauru School of Dentistry, considering an attrition of 10% and a significance level of 5%. The participants will be randomized into test and intervention groups, being respectively exposed to fluoride-related information and disinformation presented on a computer with simultaneous monitoring of their psychophysiological reactions, including analysis of their heart rates (HRs) and 7 facial features (mouth outer, mouth corner, eye area, eyebrow activity, face area, face motion, and facial center of mass). Then, participants will respond to questions about the utility and truthfulness of content, their emotional state after the experiment, eHealth literacy, oral health knowledge, and socioeconomic characteristics. The Shapiro-Wilk and Levene tests will be used to determine the normality and homogeneity of the data, which could lead to further statistical analyses for elucidating significant differences between groups, using parametric (Student t test) or nonparametric (Mann-Whitney U test) analyses. Moreover, multiple logistic regression models will be developed to evaluate the association of distinct variables with the psychophysiological aspects. Only factors with significant Wald statistics in the simple analysis will be included in the multiple models (P<.2). Furthermore, receiver operating characteristic curve analysis will be performed to determine the accuracy of the remote HR with respect to the measured HR. For all analyses, P<.05 will be considered significant.

Results

From June 2022, parents and caregivers who frequent the Clinics of Pediatric Dentistry at the Bauru School of Dentistry will be invited to participate in the study and will be randomized into 1 of the 2 groups (control or intervention). Data collection is expected to be completed in December 2023. Subsequently, the authors will analyze the data and publish the findings of the clinical trial by June 2024.

Conclusions

This randomized controlled trial aims to elucidate differences between psychophysiological patterns of adults exposed to true or false oral health content. This evidence may support the development of further studies and digital strategies, such as neural network models to automatically detect disinformation available on the internet.

Trial Registration

Brazilian Clinical Trials Registry (RBR-7q4ymr2) U1111-1263-8227; https://tinyurl.com/2kf73t3d

International Registered Report Identifier (IRRID)

PRR1-10.2196/39133

Stress Evaluation by Hemoglobin Concentration Change Using Mobile NIRS

Previous studies have reported a relationship between stress and brain activity, and stress has been quantitatively evaluated using near-infrared spectroscopy (NIRS). In the present study, we examined whether a relationship exists between salivary amylase levels and brain activity during the trail-making test (TMT) using mobile NIRS. This study aimed to assess stress levels by using mobile NIRS. Salivary amylase was measured with a salivary amylase monitor, and hemoglobin concentration was measured using Neu’s HOT-2000. Measurements were taken four times for each subject, and the values at each measurement were evaluated. Changes in the values at the first–second, second–third, and third–fourth measurements were also analyzed. Results showed that the value of the fluctuations has a higher correlation than the comparison of point values. These results suggest that the accuracy of stress assessment by NIRS can be improved by using variability and time-series data compared with stress assessment using NIRS at a single time point.

Automated Facial Expression Recognition Framework Using Deep Learning

Facial expression is one of the most significant elements which can tell us about the mental state of any person. A human can convey approximately 55% of information nonverbally and the remaining almost 45% through verbal communication. Automatic facial expression recognition is presently one of the most difficult tasks in the computer science field. Applications of facial expression recognition (FER) are not just limited to understanding human behavior and monitoring person's mood and the mental state of humans. It is also penetrating into other fields such as criminology, holographic, smart healthcare systems, security systems, education, robotics, entertainment, and stress detection. Currently, facial expressions are playing an important role in medical sciences, particularly helping the patients with bipolar disease, whose mood changes very frequently. In this study, an algorithm, automated framework for facial detection using a convolutional neural network (FD-CNN) is proposed with four convolution layers and two hidden layers to improve accuracy. An extended Cohn-Kanade (CK+) dataset is used that includes facial images of different males and females with expressions such as anger, fear, disgust, contempt, neutral, happy, sad, and surprise. In this study, FD-CNN is performed in three major steps that include preprocessing, feature extraction, and classification. By using this proposed method, an accuracy of 94% is obtained in FER. In order to validate the proposed algorithm, K-fold cross-validation is performed. After validation, sensitivity and specificity are calculated which are 94.02% and 99.14%, respectively. Furthermore, the f1 score, recall, and precision are calculated to validate the quality of the model which is 84.07%, 78.22%, and 94.09%, respectively.

Human state anxiety classification framework using EEG signals in response to exposure therapy

Human anxiety is a grave mental health concern that needs to be addressed in the appropriate manner in order to develop a healthy society. In this study, an objective human anxiety assessment framework is developed by using physiological signals of electroencephalography (EEG) and recorded in response to exposure therapy. The EEG signals of twenty-three subjects from an existing database called "A Database for Anxious States which is based on a Psychological Stimulation (DASPS)" are used for anxiety quantification into two and four levels. The EEG signals are pre-processed using appropriate noise filtering techniques to remove unwanted ocular and muscular artifacts. Channel selection is performed to select the significantly different electrodes using statistical analysis techniques for binary and four-level classification of human anxiety, respectively. Features are extracted from the data of selected EEG channels in the frequency domain. Frequency band selection is applied to select the appropriate combination of EEG frequency bands, which in this study are theta and beta bands. Feature selection is applied to the features of the selected EEG frequency bands. Finally, the selected subset of features from the appropriate frequency bands of the statistically significant EEG channels were classified using multiple machine learning algorithms. An accuracy of 94.90% and 92.74% is attained for two and four-level anxiety classification using a random forest classifier with 9 and 10 features, respectively. The proposed state anxiety classification framework outperforms the existing anxiety detection framework in terms of accuracy with a smaller number of features which reduces the computational complexity of the algorithm.

Quantifying Occupational Stress in Intensive Care Unit Nurses: An Applied Naturalistic Study of Correlations Among Stress, Heart Rate, Electrodermal Activity, and Skin Temperature

OBJECTIVE

To identify physiological correlates to stress in intensive care unit nurses.

BACKGROUND

Most research on stress correlates are done in laboratory environments; naturalistic investigation of stress remains a general gap.

METHOD

Electrodermal activity, heart rate, and skin temperatures were recorded continuously for 12-hr nursing shifts (23 participants) using a wrist-worn wearable technology (Empatica E4).

RESULTS

Positive correlations included stress and heart rate (ρ = .35, p < .001), stress and skin temperature (ρ = .49, p < .05), and heart rate and skin temperatures (ρ = .54, p = .0008).

DISCUSSION

The presence and direction of some correlations found in this study differ from those anticipated from prior literature, illustrating the importance of complementing laboratory research with naturalistic studies. Further work is warranted to recognize nursing activities associated with a high level of stress and the underlying reasons associated with changes in physiological responses.

APPLICATION

Heart rate and skin temperature may be used for real-time detection of stress, but more work is needed to validate such surrogate measures.

Potential Applications of Social Robots in Robot-Assisted Interventions for Social Anxiety

Social anxiety disorder or social phobia is a condition characterized by debilitating fear and avoidance of different social situations. We provide an overview of social anxiety and evidence-based behavioural and cognitive treatment approaches for this condition. However, treatment avoidance and attrition are high in this clinical population, which calls for innovative approaches, including computer-based interventions, that could minimize barriers to treatment and enhance treatment effectiveness. After reviewing existing assistive technologies for mental health interventions, we provide an overview of how social robots have been used in many clinical interventions. We then propose to integrate social robots in conventional behavioural and cognitive therapies for both children and adults who struggle with social anxiety. We categorize the different therapeutic roles that social robots can potentially play in activities rooted in conventional therapies for social anxiety and oriented towards symptom reduction, social skills development, and improvement in overall quality of life. We discuss possible applications of robots in this context through four scenarios. These scenarios are meant as ‘food for thought’ for the research community which we hope will inspire future research. We discuss risks and concerns for using social robots in clinical practice. This article concludes by highlighting the potential advantages as well as limitations of integrating social robots in conventional interventions to improve accessibility and standard of care as well as outlining future steps in relation to this research direction. Clearly recognizing the need for future empirical work in this area, we propose that social robots may be an effective component in robot-assisted interventions for social anxiety, not replacing, but complementing the work of clinicians. We hope that this article will spark new research, and research collaborations in the highly interdisciplinary field of robot-assisted interventions for social anxiety.

Subdividing Stress Groups into Eustress and Distress Groups Using Laterality Index Calculated from Brain Hemodynamic Response

A stress group should be subdivided into eustress (low-stress) and distress (high-stress) groups to better evaluate personal cognitive abilities and mental/physical health. However, it is challenging because of the inconsistent pattern in brain activation. We aimed to ascertain the necessity of subdividing the stress groups. The stress group was screened by salivary alpha-amylase (sAA) and then, the brain’s hemodynamic reactions were measured by functional near-infrared spectroscopy (fNIRS) based on the near-infrared biosensor. We compared the two stress subgroups categorized by sAA using a newly designed emotional stimulus-response paradigm with an international affective picture system (IAPS) to enhance hemodynamic signals induced by the target effect. We calculated the laterality index for stress (LIS) from the measured signals to identify the dominantly activated cortex in both the subgroups. Both the stress groups exhibited brain activity in the right frontal cortex. Specifically, the eustress group exhibited the largest brain activity, whereas the distress group exhibited recessive brain activity, regardless of positive or negative stimuli. LIS values were larger in the order of the eustress, control, and distress groups; this indicates that the stress group can be divided into eustress and distress groups. We built a foundation for subdividing stress groups into eustress and distress groups using fNIRS.

Deep-Learning-Based Stress Recognition with Spatial-Temporal Facial Information

In recent times, as interest in stress control has increased, many studies on stress recognition have been conducted. Several studies have been based on physiological signals, but the disadvantage of this strategy is that it requires physiological-signal-acquisition devices. Another strategy employs facial-image-based stress-recognition methods, which do not require devices, but predominantly use handcrafted features. However, such features have low discriminating power. We propose a deep-learning-based stress-recognition method using facial images to address these challenges. Given that deep-learning methods require extensive data, we constructed a large-capacity image database for stress recognition. Furthermore, we used temporal attention, which assigns a high weight to frames that are highly related to stress, as well as spatial attention, which assigns a high weight to regions that are highly related to stress. By adding a network that inputs the facial landmark information closely related to stress, we supplemented the network that receives only facial images as the input. Experimental results on our newly constructed database indicated that the proposed method outperforms contemporary deep-learning-based recognition methods.

F2GM: novel hybrid approach to detect psychological stress levels from social media interactions

Stress has become a household word which generates emotional distress, physical diseases, dysfunction and social ills. An abundant evidence is present in the literature that makes the stress research and theory high profile and important for physiological, psychological and social health. It can be legitimately said that due to the advent of social media, it has opened up inputs for the exploration of stress. The social media has become very prominent as it has touched daily lives. It has changed the way we are looking at the things, it has changed the life style, it has changed the way we are consuming the information. It has created a bridge of trust among the people of different professional’s. Social media has become undeniably a global phenomenon in the last decade or so, since the founding of social media sites like Twitter and Facebook. It is of significant importance to detect and manage the stress from theses interactions at early stage otherwise it wreaks havoc on your emotional equilibrium and your physical health. It narrows your ability to think clearly, function effectively and enjoy life. In this work our endeavor is that to present a novel method to detect the different stress levels from the social media interactions using fuzzy and factor graph methods. A correlation analysis between stressed, non-stressed and emotion tweets is carried out for social engagement correlation and behavior correlation analysis of the social media users. The proposed method performs better when results are compared with the other state of art machine learning methods.

Ocular surface cooling rate associated with tear film characteristics and the maximum interblink period

The surface of the human eye is covered with a protective tear film that refreshes with each blink. Natural blinking occurs involuntarily, but one can also voluntarily blink or refrain from blinking. The maximum time one can refrain from blinking until the onset of discomfort is the maximum interblink period (MIBP). During the interblink period the tear film evaporates and thins from the ocular surface. Infrared thermography provides a non-invasive measure of the ocular surface temperature (OST). Due to evaporation, ocular surface cooling (OSC) generally occurs when the eyes are open and exposed to the environment. The purpose of our study was to investigate the effect of OSC rate on the MIBP, and to investigate the association of the MIBP with tear film characteristics in subjects who do and do not exhibit OSC. The MIBP was measured simultaneously with OST over time. Non-invasive tear breakup time, tear meniscus height, tear lipid layer thickness, and Schirmer I test strip wetted lengths were measured on a day prior to the thermography visit. Subjects were divided into cooling and non-cooling groups based on OSC rate, and demographic and tear film characteristics were tested for inter-group differences. A faster OSC rate was associated with an exponentially shorter duration of the MIBP overall and within the cooling group alone. Faster non-invasive tear breakup time was significantly associated with a shorter MIBP in both groups. These results suggest that tear film evaporation initiates a pathway that results in the onset of ocular discomfort and the stimulus to blinking. The presence of a subset of subjects with no or minimal OSC who nevertheless have a short MIBP indicates that evaporative cooling is not the only mechanism responsible for the onset of ocular discomfort.

Imagery of negative interpersonal experiences influence the neural mechanisms of social interaction

Negative interpersonal experiences are a key contributor to psychiatric disorders. While previous research has shown that negative interpersonal experiences influence social cognition, less is known about the effects on participation in social interactions and the underlying neurobiology. To address this, we developed a new naturalistic version of a gaze-contingent paradigm using real video sequences of gaze behaviour that respond to the participants' gaze in real-time in order to create a believable and continuous interactive social situation. Additionally, participants listened to two autobiographical audio-scripts that guided them to imagine a recent stressful and a relaxing situation and performed the gaze-based social interaction task before and after the presentation of either the stressful or the relaxing audio-script. Our results demonstrate that the social interaction task robustly recruits brain areas with known involvement in social cognition, namely the medial prefrontal cortex, bilateral temporoparietal junction, superior temporal sulcus as well as the precuneus. Imagery of negative interpersonal experiences compared to relaxing imagery led to a prolonged change in affective state and to increased brain responses during the subsequent social interaction paradigm in the temporoparietal junction, medial prefrontal cortex, anterior cingulate cortex, precuneus and inferior frontal gyrus. Taken together this study presents a new naturalistic social interaction paradigm suitable to study the neural mechanisms of social interaction and the results demonstrate that the imagery of negative interpersonal experiences affects social interaction on neural levels.

Predicting individual emotion from perception-based non-contact sensor big data

This study proposes a system for estimating individual emotions based on collected indoor environment data for human participants. At the first step, we develop wireless sensor nodes, which collect indoor environment data regarding human perception, for monitoring working environments. The developed system collects indoor environment data obtained from the developed sensor nodes and the emotions data obtained from pulse and skin temperatures as big data. Then, the proposed system estimates individual emotions from collected indoor environment data. This study also investigates whether sensory data are effective for estimating individual emotions. Indoor environmental data obtained by developed sensors and emotions data obtained from vital data were logged over a period of 60 days. Emotions were estimated from indoor environmental data by machine learning method. The experimental results show that the proposed system achieves about 80% or more estimation correspondence by using multiple types of sensors, thereby demonstrating the effectiveness of the proposed system. Our obtained result that emotions can be determined with high accuracy from environmental data is a useful finding for future research approaches.

Recognition of Blinks Activity Patterns during Stress Conditions Using CNN and Markovian Analysis

This paper investigates eye behaviour through blinks activity during stress conditions. Although eye blinking is a semi-voluntary action, it is considered to be affected by one’s emotional states such as arousal or stress. The blinking rate provides information towards this direction, however, the analysis on the entire eye aperture timeseries and the corresponding blinking patterns provide enhanced information on eye behaviour during stress conditions. Thus, two experimental protocols were established to induce affective states (neutral, relaxed and stress) systematically through a variety of external and internal stressors. The study populations included 24 and 58 participants respectively performing 12 experimental affective trials. After the preprocessing phase, the eye aperture timeseries and the corresponding features were extracted. The behaviour of inter-blink intervals (IBI) was investigated using the Markovian Analysis to quantify incidence dynamics in sequences of blinks. Moreover, Convolutional Neural Networks (CNN) and Long Short-Term Memory (LSTM) network models were employed to discriminate stressed versus neutral tasks per cognitive process using the sequence of IBI. The classification accuracy reached a percentage of 81.3% which is very promising considering the unimodal analysis and the noninvasiveness modality used.

Blink rate as a measure of stress and attention in the domestic horse (Equus caballus)

Measuring animal stress is fundamentally important for assessing animal emotional state and welfare. Conventional methods of quantifying stress (cortisol levels, heart rate/heart rate variability) require specialist equipment and are not instantly available. Spontaneous blink rate (SBR) has previously been used to measure stress responses in humans and may provide a non-invasive method for measuring stress in other animal species. Here we investigated the use of SBR as a measure of stress in the domestic horse. SBR was measured before and during a low-stress event (sham clipping) and compared with heart rate variability and salivary cortisol. For the entire sample, there was a reduction in SBR (startle response) during the first minute of clipping. For horses reactive to clipping, the initial reduction in SBR was followed by an increase above baseline whereas the SBR of the non-reactive horses quickly returned to baseline. For the entire sample, SBR correlated with heart rate variability and salivary cortisol. We have demonstrated that SBR is a valid fast alternative measure of stress in horses, but the initial 'startle' response must be considered when using this parameter as a measure of animal stress.

Deep time-delay Markov network for prediction and modeling the stress and emotions state transition

To recognize stress and emotion, most of the existing methods only observe and analyze speech patterns from present-time features. However, an emotion (especially for stress) can change because it was triggered by an event while speaking. To address this issue, we propose a novel method for predicting stress and emotions by analyzing prior emotional states. We named this method the deep time-delay Markov network (DTMN). Structurally, the proposed DTMN contains a hidden Markov model (HMM) and a time-delay neural network (TDNN). We evaluated the effectiveness of the proposed DTMN by comparing it with several state transition methods in predicting an emotional state from time-series (sequences) speech data of the SUSAS dataset. The experimental results show that the proposed DTMN can accurately predict present emotional states by outperforming the baseline systems in terms of the prediction error rate (PER). We then modeled the emotional state transition using a finite Markov chain based on the prediction result. We also conducted an ablation experiment to observe the effect of different HMM values and TDNN parameters on the prediction result and the computational training time of the proposed DTMN.

Fear expressions of dogs during New Year fireworks: a video analysis

A high proportion of pet dogs show fear-related behavioural problems, with noise fears being most prevalent. Nonetheless, few studies have objectively evaluated fear expression in this species. Using owner-provided video recordings, we coded behavioural expressions of pet dogs during a real-life firework situation at New Year’s Eve and compared them to behaviour of the same dogs on a different evening without fireworks (control condition), using Wilcoxon signed ranks tests. A backwards-directed ear position, measured at the base of the ear, was most strongly associated with the fireworks condition (effect size: Cohen’s d = 0.69). Durations of locomotion (d = 0.54) and panting (d = 0.45) were also higher during fireworks than during the control condition. Vocalisations (d = 0.40), blinking (d = 0.37), and hiding (d = 0.37) were increased during fireworks, but this was not significant after sequential Bonferroni correction. This could possibly be attributed to the high inter-individual variability in the frequency of blinking and the majority of subjects not vocalising or hiding at all. Thus, individual differences must be taken into account when aiming to assess an individual’s level of fear, as relevant measures may not be the same for all individuals. Firework exposure was not associated with an elevated rate of other so-called ‘stress signals’, lip licking and yawning.

Video-Based Stress Detection through Deep Learning

Stress has become an increasingly serious problem in the current society, threatening mankind’s well-beings. With the ubiquitous deployment of video cameras in surroundings, detecting stress based on the contact-free camera sensors becomes a cost-effective and mass-reaching way without interference of artificial traits and factors. In this study, we leverage users’ facial expressions and action motions in the video and present a two-leveled stress detection network (TSDNet). TSDNet firstly learns face- and action-level representations separately, and then fuses the results through a stream weighted integrator with local and global attention for stress identification. To evaluate the performance of TSDNet, we constructed a video dataset containing 2092 labeled video clips, and the experimental results on the built dataset show that: (1) TSDNet outperformed the hand-crafted feature engineering approaches with detection accuracy 85.42% and F1-Score 85.28%, demonstrating the feasibility and effectiveness of using deep learning to analyze one’s face and action motions; and (2) considering both facial expressions and action motions could improve detection accuracy and F1-Score of that considering only face or action method by over 7%.

Assessment of Implicit and Explicit Measures of Mental Workload in Working Situations: Implications for Industry 4.0

Nowadays, in the context of Industry 4.0, advanced working environments aim at achieving a high degree of human–machine collaboration. This phenomenon occurs, on the one hand, through the correct interpretation of operators’ data by machines that can adapt their functioning to support workers, and on the other hand, by ensuring the transparency of the actions of the system itself. This study used an ad hoc system that allowed the co-registration of a set of participants’ implicit and explicit (I/E) data in two experimental conditions that varied in the level of mental workload (MWL). Findings showed that the majority of the considered I/E measures were able to discriminate the different task-related mental demands and some implicit measures were capable of predicting task performance in both tasks. Moreover, self-reported measures showed that participants were aware of such differences in MWL. Finally, the paradigm’s ecology highlights that task and environmental features may affect the reliability of the various I/E measures. Thus, these factors have to be considered in the design and development of advanced adaptive systems within the industrial context.