Measuring emotion recognition by people with Parkinson's disease using eye-tracking with dynamic facial expressions

Primate eye tracking with carbon-nanotube-paper-composite based capacitive sensors and machine learning algorithms

BACKGROUND

Accurate real-time eye tracking is crucial in oculomotor system research. While the scleral search coil system is the gold standard, its implantation procedure and bulkiness pose challenges. Camera-based systems are affected by ambient lighting and require high computational and electric power.

NEW METHOD

This study presents a novel eye tracker using proximity capacitive sensors made of carbon-nanotube-paper-composite (CPC). These sensors detect femtofarad-level capacitance changes caused by primate corneal movement during horizontal and vertical eye rotations. Data processing and machine learning algorithms are evaluated to enhance the accuracy of gaze angle prediction.

RESULTS

The system performance is benchmarked against the scleral coil during smooth pursuits, saccades tracking, and fixations. The eye tracker demonstrates up to 0.97 correlation with the coil in eye tracking and is capable of estimating gaze angle with a median absolute error as low as 0.30°.

COMPARISON

The capacitive eye tracker demonstrates good consistency and accuracy in comparison to the gold-standard scleral search coil method.

CONCLUSIONS

This lightweight, non-invasive capacitive eye tracker offers potential as an alternative to traditional coil and camera-based systems in oculomotor research and vision science.

Webcam eye tracking close to laboratory standards: Comparing a new webcam-based system and the EyeLink 1000

This paper aims to compare a new webcam-based eye-tracking system, integrated into the Labvanced platform for online experiments, to a "gold standard" lab-based eye tracker (EyeLink 1000 - SR Research). Specifically, we simultaneously recorded data with both eye trackers in five different tasks, analyzing their real-time performance. These tasks were a subset of a standardized test battery for eye trackers, including a Large Grid task, Smooth Pursuit eye movements, viewing natural images, and two Head Movements tasks (roll, yaw). The results show that the webcam-based system achieved an overall accuracy of 1.4°, and a precision of 1.1° (standard deviation (SD) across subjects), an error of about 0.5° larger than the EyeLink system. Interestingly, both accuracy (1.3°) and precision (0.9°) were slightly better for centrally presented targets, the region of interest in many psychophysical experiments. Remarkably, the correlation of raw gaze samples between the EyeLink and webcam-based was at about 90% for the Large Grid task and about 80% for Free View and Smooth Pursuit. Overall, these results put the performance of the webcam-based system roughly on par with mobile eye-tracking devices (Ehinger et al. PeerJ, 7, e7086, 2019; Tonsen et al., 2020) and demonstrate substantial improvement compared to existing webcam eye-tracking solutions (Papoutsaki et al., 2017).

Emotion Recognition in Multiple System Atrophy: An Exploratory Eye-Tracking Study

OBJECTIVE

Emotional processing is a core feature of social interactions and has been well studied in patients with idiopathic Parkinson's disease (PD), albeit with contradictory.

RESULTS

. However, these studies excluded patients with atypical parkinsonism, such as multiple system atrophy (MSA). The objective of this exploratory study was to provide better insights into emotion processing in patients with MSA using eye tracking data.

METHODS

We included 21 MSA patients, 15 PD patients and 19 matched controls in this study. Participants performed a dynamic and a static emotion recognition task, and gaze fixations were analyzed in different areas of interest. Participants underwent neuropsychological testing and assessment of depression and alexithymia.

RESULTS

MSA patients were less accurate in recognizing anger than controls (p = 0.02) and had overall fewer fixations than controls (p = 0.001). In the static task, MSA patients had fewer fixations (p < 0.001) and a longer time to first fixation (p = 0.026) on the eye region. Furthermore, MSA patients had a longer fixation duration overall than PD patients (p = 0.004) and longer fixations on the nose than controls (p = 0.005). Alexithymia scores were higher in MSA patients compared to controls (p = 0.038).

CONCLUSION

This study demonstrated impaired recognition of anger in MSA patients compared to HCs. Fewer and later fixations on the eyes along with a center bias suggest avoidance of eye contact, which may be a characteristic gaze behavior in MSA patients.

Pediatricians' focus of sight at pain assessment during a neonatal heel puncture

OBJECTIVE

To evaluate the focus of pediatricians' gaze during the heel prick of neonates.

METHODS

Prospective study in which pediatricians wearing eye tracker glasses evaluated neonatal pain before/after a heel prtick. Pediatricians scored the pain they perceived in the neonate in a verbal analogue numerical scale (0=no pain; 10=maximum pain). The outcomes measured were number and time of visual fixations in upper face, lower face, and hands, in two 10-second periods, before (pre) and after the puncture (post). These outcomes were compared between the periods, and according to pediatricians' pain perception: absent/mild (score: 0-5) and moderate/intense (score: 6-10).

RESULTS

24 pediatricians (31 years old, 92% female) evaluated 24 neonates. The median score attributed to neonatal pain during the heel prick was 7.0 (Interquartile range: 5-8). Compared to pre-, in the post-periods, more pediatricians fixed their gaze on the lower face (63 vs. 92%; p=0.036) and the number of visual fixations was greater on the lower face (2.0 vs. 5.0; p=0.018). There was no difference in the number and time of visual fixations according to the intensity of pain.

CONCLUSIONS

At bedside, pediatricians change their focus of attention on the neonatal face after a painful procedure, focusing mainly on the lower part of the face.

Eye tracking technology in medical practice: a perspective on its diverse applications

Eye tracking technology has emerged as a valuable tool in the field of medicine, offering a wide range of applications across various disciplines. This perspective article aims to provide a comprehensive overview of the diverse applications of eye tracking technology in medical practice. By summarizing the latest research findings, this article explores the potential of eye tracking technology in enhancing diagnostic accuracy, assessing and improving medical performance, as well as improving rehabilitation outcomes. Additionally, it highlights the role of eye tracking in neurology, cardiology, pathology, surgery, as well as rehabilitation, offering objective measures for various medical conditions. Furthermore, the article discusses the utility of eye tracking in autism spectrum disorders, attention-deficit/hyperactivity disorder (ADHD), and human-computer interaction in medical simulations and training. Ultimately, this perspective article underscores the transformative impact of eye tracking technology on medical practice and suggests future directions for its continued development and integration.

Observation and imitation of object-directed hand movements in Parkinson's disease

Action observation and imitation may facilitate movement in Parkinson's disease (PD). People with PD have been found to imitate intransitive actions similarly to neurologically healthy older adults, but their imitation of object-directed hand movements has not previously been investigated using kinematic measures. The present study examined observation and imitation of object-directed hand movements in 18 participants with PD and 21 neurologically healthy age-matched control participants. Participants observed and immediately imitated sequences showing a human hand reaching for and transferring an object between horizontal positions. Both groups significantly modulated their finger movements, showing higher vertical amplitude when imitating elevated compared to direct trajectories. In addition, movements were lower in vertical amplitude and higher in velocity when imitating the reaching segment than the transfer segment. Eye-tracking revealed that controls made smaller saccades when observing predictable than unpredictable elevated movements, but no effects of predictability on eye movements were found for the PD group. This study provides quantitative evidence that people with mild to moderate PD can imitate object-directed hand movement kinematics, although their prediction of such movements may be reduced. These findings suggest that interventions targeting object-directed actions may capitalize on the ability of people with PD to imitate kinematic parameters of a demonstrated movement.

Development and validation of a natural dynamic facial expression stimulus set

Emotion research commonly uses either controlled and standardised pictures or natural video stimuli to measure participants' reactions to emotional content. Natural stimulus materials can be beneficial; however, certain measures such as neuroscientific methods, require temporally and visually controlled stimulus material. The current study aimed to create and validate video stimuli in which a model displays positive, neutral and negative expressions. These stimuli were kept as natural as possible while editing timing and visual features to make them suitable for neuroscientific research (e.g. EEG). The stimuli were successfully controlled regarding their features and the validation studies show that participants reliably classify the displayed expression correctly and perceive it as genuine. In conclusion, we present a motion stimulus set that is perceived as natural and that is suitable for neuroscientific research, as well as a pipeline describing successful editing methods for controlling natural stimuli.

Machine Learning and Eye Movements Give Insights into Neurodegenerative Disease Mechanisms

Humans are a vision-dominated species; what we perceive depends on where we look. Therefore, eye movements (EMs) are essential to our interactions with the environment, and experimental findings show EMs are affected in neurodegenerative disorders (ND). This could be a reason for some cognitive and movement disorders in ND. Therefore, we aim to establish whether changes in EM-evoked responses can tell us about the progression of ND, such as Alzheimer's (AD) and Parkinson's diseases (PD), in different stages. In the present review, we have analyzed the results of psychological, neurological, and EM (saccades, antisaccades, pursuit) tests to predict disease progression with machine learning (ML) methods. Thanks to ML algorithms, from the high-dimensional parameter space, we were able to find significant EM changes related to ND symptoms that gave us insights into ND mechanisms. The predictive algorithms described use various approaches, including granular computing, Naive Bayes, Decision Trees/Tables, logistic regression, C-/Linear SVC, KNC, and Random Forest. We demonstrated that EM is a robust biomarker for assessing symptom progression in PD and AD. There are navigation problems in 3D space in both diseases. Consequently, we investigated EM experiments in the virtual space and how they may help find neurodegeneration-related brain changes, e.g., related to place or/and orientation problems. In conclusion, EM parameters with clinical symptoms are powerful precision instruments that, in addition to their potential for predictions of ND progression with the help of ML, could be used to indicate the different preclinical stages of both diseases.

Clear Aligners and Smart Eye Tracking Technology as a New Communication Strategy between Ethical and Legal Issues

Smart eye-tracking technology (SEET) that determines visual attention using smartphones can be used to determine the aesthetic perception of different types of clear aligners. Its value as a communication and comprehension tool, in addition to the ethical and legal concerns which it entails, can be assessed. One hundred subjects (50 F, 50 M; age range 15-70) were equally distributed in non-orthodontic (A) and orthodontic (B) groups. A smartphone-based SEET app assessed their knowledge of and opinions on aligners. Subjects evaluated images of smiles not wearing aligners, with/without attachments and with straight/scalloped gingival margins, as a guided calibration step which formed the image control group. Subsequently, the subjects rated the same smiles, this time wearing aligners (experimental images group). Questionnaire data and average values for each group of patients, and images relating to fixation times and overall star scores, were analyzed using these tests: chi-square, t-test, Mann-Whitney U, Spearman's rho, and Wilcoxon (p < 0.05). One-way ANOVA and related post-hoc tests were also applied. Orthodontic patients were found to be better informed than non-orthodontic patients. Aesthetic perception could be swayed by several factors. Attachments scored lower in aesthetic evaluation. Lips distracted attention from attachments and improved evaluations. Attachment-free aligners were better rated overall. A more thorough understanding as to the opinions, expectations and aesthetic perception of aligners can improve communication with patients. Mobile SEET is remarkably promising, although it does require a careful medicolegal risk-benefit assessments for responsible and professional use.

Moving beyond emotions in Parkinson's disease

OBJECTIVE

Emotion recognition is a fundamental neurocognitive capacity that is a critical predictor of interpersonal function and, in turn, mental health. Although people with Parkinson's disease (PD) often exhibit difficulties recognizing emotions, almost all studies to date have focused on basic emotions (happiness, sadness, anger, surprise, fear, and disgust), with little consideration of how more cognitively complex self-conscious emotions such as contempt, embarrassment, and pride might also be affected. Further, the few studies that have considered self-conscious emotions have relied on high intensity, static stimuli. The aim of the present study was to therefore provide the first examination of how self-conscious emotion recognition is affected by PD using a dynamic, dual-intensity measure that more closely captures how emotion recognition judgements are made in daily life.

METHOD

People with PD (n = 42) and neurotypical controls (n = 42) completed a validated measure of self-conscious facial emotion recognition. For comparative purposes, in addition to a broader clinical test battery, both groups also completed a traditional static emotion recognition measure and a measure of self-conscious emotional experience.

RESULTS

Relative to controls, the PD group did not differ in their capacity to recognize basic emotions but were impaired in their recognition of self-conscious emotions. These difficulties were associated with elevated negative affect and poorer subjective well-being.

CONCLUSIONS

Difficulties recognizing self-conscious emotions may be more problematic for people with PD than difficulties recognizing basic ones, with implications for interventions focused on helping people with this disorder develop and maintain strong social networks.

PRACTITIONER POINTS

This is the first direct investigation into how the recognition of self-conscious emotion is affected in Parkinson's disease using dynamic, dual-intensity stimuli, thus providing an important extension to prior literature that has focused solely on basic emotion recognition and/or relied on static, high-intensity stimuli. Results revealed preserved basic facial emotional recognition coexisting with impairment in all three self-conscious emotions assessed, therefore suggesting that the latter stimuli type may function as a more sensitive indicator of Parkinson's disease-related social cognitive impairment. Problems with self-conscious emotion recognition in people with Parkinson's disease were associated with poorer broader subjective well-being and increased negative affect. This aligns with the broader literature linking interpersonal difficulties with poorer clinical outcomes in this cohort.

Feelings first? Sex differences in affective and cognitive processes in emotion recognition

The recognition of emotional expressions is important for social understanding and interaction, but findings on the relationship between emotion recognition, empathy, and theory of mind, as well as sex differences in these relationships, have been inconsistent. This may reflect the relative involvement of affective and cognitive processes at different stages of emotion recognition and in different experimental paradigms. In this study, images of faces were morphed from neutral to full expression of five basic emotions (anger, disgust, fear, happiness, and sadness), which participants were asked to identify as quickly and accurately as possible. Accuracy and response times from healthy males (n = 46) and females (n = 43) were analysed in relation to self-reported empathy (Empathy Quotient; EQ) and mentalising/theory of mind (Reading the Mind in the Eyes Test). Females were faster and more accurate than males in recognising dynamic emotions. Linear mixed-effects modelling showed that response times were inversely related to the emotional empathy subscale of the EQ, but this was accounted for by a female advantage on both measures. Accuracy was unrelated to EQ scores but was predicted independently by sex and Eyes Test scores. These findings suggest that rapid processing of dynamic emotional expressions is strongly influenced by sex, which may reflect the greater involvement of affective processes at earlier stages of emotion recognition.

Human Emotion Recognition using Polar-Based Lagged Poincare Plot Indices of Eye-Blinking Data

Emotion recognition using bio-signals is currently a hot and challenging topic in human–computer interferences, robotics, and affective computing. A broad range of literature has been published by analyzing the internal/external behaviors of the subjects in confronting emotional events/stimuli. Eye movements, as an external behavior, are frequently used in the multi-modal emotion recognition system. On the other hand, classic statistical features of the signal have generally been assessed, and the evaluation of its dynamics has been neglected so far. For the first time, the dynamics of single-modal eye-blinking data are characterized. Novel polar-based indices of the lagged Poincare plots were introduced. The optimum lag was estimated using mutual information. After reconstruction of the plot, the polar measures of all points were characterized using statistical measures. The support vector machine (SVM), decision tree, and Naïve Bayes were implemented to complete the process of classification. The highest accuracy of 100% with an average accuracy of 84.17% was achieved for fear/sad discrimination using SVM. The suggested framework provided outstanding performances in terms of recognition rates, simplicity of the methodology, and less computational cost. Our results also showed that eye-blinking data possesses the potential for emotion recognition, especially in classifying fear emotion.

Automated Computer Vision Assessment of Hypomimia in Parkinson Disease: Proof-of-Principle Pilot Study

BACKGROUND

Facial expressions require the complex coordination of 43 different facial muscles. Parkinson disease (PD) affects facial musculature leading to "hypomimia" or "masked facies."

OBJECTIVE

We aimed to determine whether modern computer vision techniques can be applied to detect masked facies and quantify drug states in PD.

METHODS

We trained a convolutional neural network on images extracted from videos of 107 self-identified people with PD, along with 1595 videos of controls, in order to detect PD hypomimia cues. This trained model was applied to clinical interviews of 35 PD patients in their on and off drug motor states, and seven journalist interviews of the actor Alan Alda obtained before and after he was diagnosed with PD.

RESULTS

The algorithm achieved a test set area under the receiver operating characteristic curve of 0.71 on 54 subjects to detect PD hypomimia, compared to a value of 0.75 for trained neurologists using the United Parkinson Disease Rating Scale-III Facial Expression score. Additionally, the model accuracy to classify the on and off drug states in the clinical samples was 63% (22/35), in contrast to an accuracy of 46% (16/35) when using clinical rater scores. Finally, each of Alan Alda's seven interviews were successfully classified as occurring before (versus after) his diagnosis, with 100% accuracy (7/7).

CONCLUSIONS

This proof-of-principle pilot study demonstrated that computer vision holds promise as a valuable tool for PD hypomimia and for monitoring a patient's motor state in an objective and noninvasive way, particularly given the increasing importance of telemedicine.

Tunable Q wavelet transform based emotion classification in Parkinson's disease using Electroencephalography

Parkinson's disease (PD) is a severe incurable neurological disorder. It is mostly characterized by non-motor symptoms like fatigue, dementia, anxiety, speech and communication problems, depression, and so on. Electroencephalography (EEG) play a key role in the detection of the true emotional state of a person. Various studies have been proposed for the detection of emotional impairment in PD using filtering, Fourier transforms, wavelet transforms, and non-linear methods. However, these methods require a selection of basis and are confined in terms of accuracy. In this paper, tunable Q wavelet transform (TQWT) is proposed for the classification of emotions in PD and normal controls (NC). EEG signals of six emotional states namely happiness, sadness, fear, anger, surprise, and disgust are studied. Power, entropy, and statistical moments based features are elicited from the highpass and lowpass sub-bands of TQWT. Six features selected by statistical analysis are classified with a k-nearest neighbor, probabilistic neural network, random forest, decision tree, and extreme learning machine. Three performance measures are obtained, maximum mean accuracy, sensitivity, and specificity of 96.16%, 97.59%, and 88.51% for NC and 93.88%, 96.33%, and 81.67% for PD are achieved with a probabilistic neural network. The proposed method proved to be very effective such that it classifies emotions in PD and could be used as a potential tool for diagnosing emotional impairment in hospitals.

How Our Gaze Reacts to Another Person's Tears? Experimental Insights Into Eye Tracking Technology

Crying is an ubiquitous human behavior through which an emotion is expressed on the face together with visible tears and constitutes a slippery riddle for researchers. To provide an answer to the question "How our gaze reacts to another person's tears?," we made use of eye tracking technology to study a series of visual stimuli. By presenting an illustrative example through an experimental setting specifically designed to study the "tearing effect," the present work aims to offer methodological insight on how to use eye-tracking technology to study non-verbal cues. A sample of 30 healthy young women with normal visual acuity performed a within-subjects task in which they evaluated images of real faces with and without tears while their eye movements were tracked. Tears were found to be a magnet for visual attention in the task of facial attribution, facilitating a greater perception of emotional intensity. Moreover, the inspection pattern changed qualitatively and quantitatively, with our participants becoming fully focused on the tears when they were visible. The mere presence of a single tear running down a cheek was associated with an increased emotional inference and greater perception of sincerity. Using normalized and validated tools (Reading the Eyes in the Mind Test and the SALAMANCA screening test for personality disorders), we measured the influence of certain characteristics of the participants on their performance of the experimental task. On the one hand, a higher level of cognitive empathy helped to classify tearful faces with higher emotional intensity and tearless faces with less emotional intensity. On the other hand, we observed that less sincerity was attributed to the tearful faces as the SALAMANCA test scores rose in clusters A (strange and extravagant) and B (immature and emotionally unstable) of our sample. The present findings highlight the advantages of using eye tracking technology to study non-verbal cues and draw attention to methodological issues that should be taken into account. Further exploration of the relationship between empathy and tear perception could be a fruitful avenue of future research using eye tracking.