Capacity of Generative AI to Interpret Human Emotions From Visual and Textual Data: Pilot Evaluation Study

BACKGROUND

Mentalization, which is integral to human cognitive processes, pertains to the interpretation of one's own and others' mental states, including emotions, beliefs, and intentions. With the advent of artificial intelligence (AI) and the prominence of large language models in mental health applications, questions persist about their aptitude in emotional comprehension. The prior iteration of the large language model from OpenAI, ChatGPT-3.5, demonstrated an advanced capacity to interpret emotions from textual data, surpassing human benchmarks. Given the introduction of ChatGPT-4, with its enhanced visual processing capabilities, and considering Google Bard's existing visual functionalities, a rigorous assessment of their proficiency in visual mentalizing is warranted.

OBJECTIVE

The aim of the research was to critically evaluate the capabilities of ChatGPT-4 and Google Bard with regard to their competence in discerning visual mentalizing indicators as contrasted with their textual-based mentalizing abilities.

METHODS

The Reading the Mind in the Eyes Test developed by Baron-Cohen and colleagues was used to assess the models' proficiency in interpreting visual emotional indicators. Simultaneously, the Levels of Emotional Awareness Scale was used to evaluate the large language models' aptitude in textual mentalizing. Collating data from both tests provided a holistic view of the mentalizing capabilities of ChatGPT-4 and Bard.

RESULTS

ChatGPT-4, displaying a pronounced ability in emotion recognition, secured scores of 26 and 27 in 2 distinct evaluations, significantly deviating from a random response paradigm (P<.001). These scores align with established benchmarks from the broader human demographic. Notably, ChatGPT-4 exhibited consistent responses, with no discernible biases pertaining to the sex of the model or the nature of the emotion. In contrast, Google Bard's performance aligned with random response patterns, securing scores of 10 and 12 and rendering further detailed analysis redundant. In the domain of textual analysis, both ChatGPT and Bard surpassed established benchmarks from the general population, with their performances being remarkably congruent.

CONCLUSIONS

ChatGPT-4 proved its efficacy in the domain of visual mentalizing, aligning closely with human performance standards. Although both models displayed commendable acumen in textual emotion interpretation, Bard's capabilities in visual emotion interpretation necessitate further scrutiny and potential refinement. This study stresses the criticality of ethical AI development for emotional recognition, highlighting the need for inclusive data, collaboration with patients and mental health experts, and stringent governmental oversight to ensure transparency and protect patient privacy.

Moderating Effects of Emotional Recognition Competency in Rejective Parenting and Adolescent Depression and Aggression

Rejective parenting is a major antecedent of emotional instability and aggressive behavioral problems. Previous studies have reported that emotional problems, such as depression, anxiety, aggression, and conduct problems in children and adolescents, improve through interventions that enhance emotional recognition competency. In this study, we explored whether the emotional recognition ability levels of individual adolescents moderated the pathway of negative parenting on aggression mediated by depression The moderated mediating effect of emotional recognition competency was investigated through examining 2265 first-year high school students using the 2021 data from the Korean Children and Youth Panel Survey 2018. There was no significant moderating effect on the direct pathway from rejective parenting to aggression. The moderating effect of emotional recognition competency on the indirect pathway leading to aggression through depression in rejective parenting was significant. These results suggest that the ability to correctly understand signals related to the emotions of others can play an important role in reducing depression and aggressive behavior by reducing conflict with people around them and experiencing more support.

Emotion recognition and baseline cortisol levels relationship in early Alzheimer disease

BACKGROUND

Emotion recognition is often impaired in early Alzheimer's disease (AD) and can be evaluated using the Reading the Mind in the Eyes Test (RMET). Similarly, cortisol levels can affect cognition and could be considered a biomarker of AD.

OBJECTIVES

The aim of this study was to analyse the relationship between the emotion recognition task and cortisol levels in participants with early Alzheimer Disease (AD).

METHODS

Complex emotion recognition was assessed with RMET, and plasma cortisol levels were determined by mass spectrometry in participants classified into mild cognitive impairment (MCI) due to AD (n = 25), mild dementia (MD) due to AD (n = 20), MCI non-AD (n = 34), MD non-AD (n = 13) and healthy controls (HC) (n = 16) groups.

RESULTS

Significantly lower positive emotion recognition was found in the MCI non-AD group (p = 0.02) and lower emotion recognition in MD (AD and non-AD) groups (p < 0.01) compared to the healthy group. In addition, significant differences were observed between cortisol and all RMET scores among the MCI and MD groups (p < 0.01). A significant correlation was also obtained between total and neutral RMET scores and cortisol levels in MD groups (p = 0.01).

CONCLUSIONS

These outcomes suggest that detection of positive emotion dysfunction could help to identify MCI non-AD patients. Furthermore, general impaired emotion recognition and high cortisol levels may be associated with cognitive impairment at mild dementia level.

Impact of Stable Environments on Maltreated Children

BACKGROUND

The development of the ability to understand others' facial expressions is thought to be dependent on the environment in which one has been reared.

METHODS

This study compared the ability to understand others' facial expressions between 15 children who were in an unstable environment, 11 children who had been maltreated before and were in a stable environment, like a foster family, and 33 children who had never been maltreated. We used the "Reading the Mind in the Eyes Test" (RMET) as measure.

RESULTS

Children who were in an unstable environment scored higher in the RMET than children who had never been maltreated.

CONCLUSIONS

The results suggested that hypersensitivity to others' facial expressions might be an adaptive response to a harmful environment, and that it might decline when in a stable environment because such sensitivity is no longer needed.

The effect of intranasal oxytocin on processing emotional stimuli during alcohol withdrawal: A randomized placebo-controlled double-blind clinical trial

Alcohol dependence is associated with difficulties in processing emotional stimuli, which can lead to interpersonal problems. The neuropeptide oxytocin has been shown to modulate the processing of emotional stimuli, however, oxytocin treatment has not yet been examined in patients with withdrawal symptoms during alcohol detoxification. The aim of the present study was to investigate the effect of oxytocin on the reading the mind in the eyes test (RMET), which indexes theory of mind ability, during a three-day period of alcohol detoxification at an addiction treatment centre in Norway. We performed a randomized, double-blind, placebo-controlled trial in 39 patients fulfilling criteria for ICD-10 diagnosis of alcohol dependence admitted for alcohol detoxification and withdrawal treatment. Participants were randomized to receive either intranasal oxytocin (24 IU) or placebo, twice daily for three days. We evaluated RMET performance on day 2 and day 3 of detoxification and differences in RMET scores between day 2 and day 3 of detoxification. Frequentist and Bayesian statistical inference suggested that oxytocin administration during alcohol withdrawal in alcohol-dependent patients did not improve RMET performance. However, exploratory analyses provided preliminary evidence that oxytocin might improve performance on the RMET negative emotion subscale (uncorrected p value = 0.038), and that oxytocin treatment might show the most promise for those with high levels of alcohol consumption (i.e., ≥20 alcohol units per day; uncorrected p value = 0.023). Moreover, alcohol consumption levels significantly predicted RMET performance on day 2, but not on day 3, of withdrawal.

Exploring social emotion processing in autism: evaluating the reading the mind in the eyes test using network analysis

BACKGROUND

Features of autism spectrum disorder (ASD) include difficulties in processing and interpreting socioemotional information. The "Reading the Mind in the Eyes" test (RMET) is a validated measurement for processing socioemotional ability. However, previous RMET studies did not explore patterns of incorrect answers and the emotional valence of the test items. This study used the Taiwanese version of the RMET and the network analysis methods to examine the differences in underlying mechanisms of socioemotional processes between 30 males with autism spectrum disorder (ASD) (mean age = 18 years) and 30 healthy control males (mean age = 17 years). For each test item, a picture of a person's eyes and partial face was shown with four words describing the emotional status on picture corners. Participants were instructed to choose one of the four words that best matched the person's thinking or feeling. We further classified the words into three valences of emotional categories to examine socioemotional processes.

RESULTS

Our results showed that ASD males performed poorer on the RMET than the controls. ASD males had higher network density and in-degree scores, especially in negative words, than control males.

CONCLUSIONS

The findings suggest that males with ASD might have deficits in mapping the best emotional concept words to the target item, especially for processing negative emotion.

Divergent effects of oxytocin on "mind-reading" in healthy males

The neuropeptide oxytocin (OT) has been associated with a broad range of human behaviors, particularly in the domain of social cognition, and is being discussed to play a role in a range of psychiatric disorders. Studies using the Reading The Mind In The Eyes Test (RMET) to investigate the role of OT in mental state recognition reported inconsistent outcomes. The present study applied a randomized, double-blind, cross-over design, and included measures of serum OT. Twenty healthy males received intranasal placebo or OT (24 IU) before performing the RMET. Frequentist and Bayesian analyses showed that contrary to previous studies (Domes et al., 2007; Radke & de Bruijn, 2015), individuals performed worse in the OT condition compared to the placebo condition (p = 0.023, Cohen’s d = 0.55, 95% confidence interval [CI] [0.08, 1.02], BF₁₀ = 6.93). OT effects did not depend on item characteristics (difficulty, valence, intensity, sex) of the RMET. Furthermore, OT serum levels did not change after intranasal OT administration. Given that similar study designs lead to heterogeneous outcomes, our results highlight the complexity of OT effects and support evidence that OT might even interfere with social cognitive abilities. However, the Bayesian analysis approach shows that there is only moderate evidence that OT influences mind-reading, highlighting the need for larger-scale studies considering the discussed aspects that might have led to divergent study results.

Cognitive Empathy as Imagination: Evidence From Reading the Mind in the Eyes in Autism and Schizotypy

How is cognitive empathy related to sociality, imagination, and other psychological constructs? How is it altered in disorders of human social cognition? We leveraged a large data set (1,168 students, 62% female) on the Reading the Mind in the Eyes test (RMET), the Autism Quotient (AQ), and the Schizotypal Personality Questionnaire (SPQ-BR) to test the hypotheses that the RMET, as a metric of cognitive empathy, reflects mainly social abilities, imagination, or both. RMET showed the expected female bias in performance, though only for eyes that expressed emotions and not for neutral expressions. RMET performance was significantly, and more strongly, associated with the AQ and SPQ subscales that reflect aspects of imagination (AQ-Imagination and SPQ-Magical Ideation) than aspects of social abilities (AQ-Social, AQ-Communication, and SPQ-Interpersonal subscales). These results were confirmed with multiple regression analysis, which also implicated increased attention (AQ-Attention Switching and, marginally non-significantly, AQ-Attention to Detail) in RMET performance. The two imagination-related correlates of RMET performance also show the strongest sex biases for the AQ and SPQ: male biased in AQ-Imagination, and female biased in SPQ-Magical Ideation, with small to medium effect sizes. Taken together, these findings suggest that cognitive empathy, as quantified by the RMET, centrally involves imagination, which is underdeveloped (with a male bias) on the autism spectrum and overdeveloped (with a female bias) on the schizotypy spectrum, with optimal emotion-recognition performance intermediate between the two. The results, in conjunction with previous studies, implicate a combination of optimal imagination and focused attention in enhanced RMET performance.

The Association Between Reading the Mind in the Eyes Test Performance and Intelligence Quotient in Children and Adolescents With Asperger Syndrome

Background: There has been an extensive debate about a potential association between intelligence and social cognition. In this study, we aimed to assess the association between social cognition as measured with the Reading the Mind in the Eyes test (RMET) and intelligence as measured with the fourth edition of the Wechsler Intelligence Scale for Children (WISC-IV) in children and adolescents diagnosed with Asperger Syndrome (AS). Methods: We conducted a cross-sectional study among 84 children diagnosed with AS aged 6-16 years (mean = 11.64; standard deviation = 2.75; 92.9% males). We analyzed the association between RMET performance and WISC-IV total score as well as the association between RMET performance and each of the four WISC-IV indexes (processing speed index, PSI; working memory index, WMI; perceptual reasoning index, PRI, and verbal comprehension index, VCI). Results: We found a positive correlation between RMET performance and full-scale intelligence quotient (r = 0.340; p < 0.01), VCI (r = 0.310; p < 0.01), PRI (r = 0.401; p < 0.01), and WMI (r = 0.292; p < 0.01). In the linear regression model, age was a significant predictor of RMET score (β = 0.409; p < 0.001) as was PRI (β = 0.309; p = 0.019). Conclusion: Our results suggest that intelligence quotient positively influences RMET performance, indicating that intelligence increases social cognition in individuals diagnosed with AS. However, weak-to-moderate size effects were found. This study contributes to understanding the mechanisms underlying the disturbance of social cognition in children and adolescents diagnosed with AS.

Genetic and epigenetic serotonergic markers predict the ability to recognize mental states

The serotonergic (5-HT) system is related to affective and cognitive processes and explains behavioral variability in the normal and psychopathological range. For this reason, the hypothesis was put forward that genetic and epigenetic markers related to 5-HT metabolism predict individual differences in social cognitive functioning. Social cognitions are complex mental processes necessary for perceiving, interpreting and reacting to the behaviors of others. In order to test this hypothesis one of the most prominent theory of mind tasks, the reading the mind in the eye test (RMET), was administered to N = 435 participants and measures of performance were related to the functional MAO-A VNTR polymorphism (relevant for 5-HT catabolism) and to epigenetic markers in the promoter of the TPH-2 gene (relevant for 5-HT synthesis). It was postulated that genetic and epigenetic markers of high 5-HT activity are positively related to RMET performance. Results show that the MAO-A high activity allele, together with the degree of methylation at a promoter CpG site on the TPH-2 gene explain significant proportions of variance in the RMET performance even after controlling for age and sex effects. Present findings yield evidence for the importance of 5-HT for social cognition. Based on additional findings, the role of a TRP-rich diet for theory of mind functions is discussed.

I Cannot Read Your Eye Expression: Suicide Attempters Have Difficulties in Interpreting Complex Social Emotions

Background: The ability to differentiate emotions in social contexts is important for dealing with challenging social situations. Suicide attempters show some difficulties in emotion recognition that may result in hypersensitivity to social stress. However, other studies on the recognition of social complex emotions found that suicide attempters have similar performances as depressed non-attempters. Objectives: To investigate differences in social emotion recognition in patients with current Major Depressive Episode (MDE) with and without history of suicide attempt. Methods: Two hundred and ten patients with MDE were recruited among whom 115 had lifetime history of suicide attempt (suicide attempters, SA) and 95 did not (affective controls, AC). Recognition of complex social emotions was assessed using the Reading the Mind in the Eyes Test (RMET). Emotions were separated in three valence categories: positive, negative, and neutral. Verbal intelligence quotient (IQ) and attention were measured with the National Adult Reading Task (NART) and the d2 test, respectively. Results: Mixed logistic regression models adjusted for sex, lifetime bipolar disorder, verbal IQ and attention showed that the RMET performance for neutral emotions was worse in the SA than AC group (OR = 0.87 [0.75, 0.99]). Furthermore, when violent/serious SA were compared to non-violent/non-serious SA and AC, the RMET neutral valence category showed a trend for group factor (p < 0.059) and RMET scores were lower in the violent/serious SA than AC group (OR = 0.79 [0.64, 0.96]). Conclusion: Recognition of neutral emotions is poor in SA and this may complicate their daily life. Interventions to improve the understanding of complex emotions may be helpful to prevent suicidal risk in patients with depression.

Mentalizing Based on External Features in Borderline Personality Disorder Compared With Healthy Controls: The Role of Attachment Dimensions and Childhood Trauma

Extant research suggests that borderline personality disorder (BPD) is associated with impairments in mentalizing, that is, comprehending behavior in terms of underlying mental states. However, the precise nature of these impairments remains unclear. The literature is mixed concerning mental-izing based on external features of others, and specifically facial emotion recognition (FER) in BPD patients. This study investigated FER differences in 79 BPD patients and 79 matched healthy controls using the Reading the Mind in the Eyes Test (RMET). The authors also investigated attachment dimensions and childhood trauma in relation to mentalizing based on external features. Results showed that BPD patients performed worse on positive and negative emotions. Furthermore, avoidant attachment was negatively related to FER for neutral emotions, particularly in the control group. Trauma was negatively related to FER at trend level, particularly in BPD patients. The implications for this understanding of mentalizing based on external features in BPD are discussed.

Theory of Mind in Mild Cognitive Impairment - Relationship with Limbic Structures and Behavioural Change

OBJECTIVES

Older adults presenting with mild cognitive impairment (MCI) have a higher risk of developing dementia and also demonstrate impairments in social cognition. This study sought to establish whether in people with MCI, poorer theory of mind (ToM) was associated with volumetric changes in the amygdala and hippocampus, as well as early changes in behaviour.

METHODS

One hundred and fourteen people with MCI and fifty-two older adult controls completed the Reading the Mind in the Eyes Test (RMET), while close informants (e.g., spouse/family member/friend/carer) described any current behavioural changes using the Revised Cambridge Behavioural Inventory (CBI-R). A subsample of participants completed structural magnetic resonance imaging (MRI).

RESULTS

The MCI group showed poorer performance on all neuropsychological tests administered, and moderate reductions on the RMET compared to the control group (d = .44), with greater reduction observed in those with amnestic compared to non-amnestic MCI (p = .03). While a robust correlation was identified between poorer RMET performance and smaller hippocampal volume in the control group (ρ = .53, p = .01), this relationship was not apparent in the MCI group (ρ = .21, p = .11). In the MCI group, poorer RMET performance was associated with poorer everyday skills (ρ = -.26, p = .01) assessed by the CBI-R.

CONCLUSIONS

Our findings cross-validate previous reports that social cognitive deficits in ToM are a feature of MCI and also suggest that disruptions to broader neural networks are likely to be implicated. Furthermore, ToM deficits in MCI are associated with a decline in everyday skills such as writing or paying bills.

The effects of intranasal oxytocin in opioid-dependent individuals and healthy control subjects: a pilot study

RATIONALE

There has been an explosion of research on the potential benefits of the social neuropeptide oxytocin for a number of mental disorders including substance use disorders. Recent evidence suggests that intranasal oxytocin has both direct anti-addiction effects and pro-social effects that may facilitate engagement in psychosocial treatment for substance use disorders.

OBJECTIVES

We aimed to assess the tolerability of intranasal oxytocin and its effects on heroin craving, implicit association with heroin and social perceptual ability in opioid-dependent patients receiving opioid replacement therapy (ORT) and healthy control participants.

METHODS

We performed a randomized, double-blind, placebo-controlled, within- and between-subjects, crossover, proof-of-concept trial to examine the effects of oxytocin (40 international units) on a cue-induced craving task (ORT patients only), an Implicit Association Task (IAT), and two social perception tasks: the Reading the Mind in the Eyes Task (RMET) and The Awareness of Social Inference Test (TASIT).

RESULTS

Oxytocin was well tolerated by patients receiving ORT but had no significant effects on craving or IAT scores. There was a significant reduction in RMET performance after oxytocin administration versus placebo in the patient group only, and a significant reduction in TASIT performance after oxytocin in both the patient and healthy control groups.

CONCLUSIONS

A single dose of intranasal oxytocin is well tolerated by patients receiving ORT, paving the way for future investigations. Despite no significant improvement in craving or IAT scores after a single dose of oxytocin and some evidence that social perception was worsened, further investigation is required to determine the role oxytocin may play in the treatment of opioid use disorder.

CLINICAL TRIAL REGISTRATION

Methadone Oxytocin Option. ClinicalTrials.gov identifier: NCT01728909.

Does oxytocin affect mind-reading? A replication study

One of the most well-known findings in human oxytocin research is its beneficial effect on "mind-reading", i.e., inferring others' mental states just from the eye region in the Reading the Mind in the Eyes Test (RMET). Previous studies have partially confirmed these improvements and have further shown that they depend both on baseline social-emotional abilities and on specific item characteristics such as difficulty. Following the original design of Domes et al. (2007), the aim of the current study was to replicate and extend previous findings by thoroughly investigating the impact of oxytocin administration on RMET performance. We tested for potential moderation effects involving item difficulty, valence, intensity, sex of poser as well as individual differences in trait empathy measured with the Empathy Quotient (EQ) for a general score and the Interpersonal Reactivity Index (IRI) for a multidimensional assessment of cognitive and emotional empathy. Oxytocin did not affect mind-reading, neither in general nor when considering specific item characteristics. An association between oxytocin-induced changes in RMET performance and emotional empathy (the empathic concern scale of the IRI) was evident, with individuals low in emotional empathy showing greater improvement after oxytocin administration compared to placebo. The reproducibility and variability of these and prior findings needs to be addressed in future experiments. As true effects may not replicate across different studies for various reasons, this should not discourage, but encourage further research.

The association between 2D:4D ratio and cognitive empathy is contingent on a common polymorphism in the oxytocin receptor gene (OXTR rs53576)

Both testosterone and oxytocin influence an individual's accuracy in inferring another's feelings and emotions. Fetal testosterone, and the second-to-forth digit ratio (2D:4D) as its proxy, plays a role in social cognitive development, often by attenuating socio-affective skill. Conversely, oxytocin generally facilitates socio-affiliative and empathic cognition and behavior. A common polymorphism in the oxytocin receptor gene, OXTR rs53576, has been repeatedly linked with psychosocial competence, including empathy, with individuals homozygous for the G allele typically characterized by enhanced socio-cognitive skills compared to A allele carriers. We examined the role of oxytocin and testosterone in collectively contributing to individual differences in cognitive empathy as measured by Baron-Cohen's "Reading the Mind in the Eyes" task (RMET). Findings are based on a large cohort of male and female students (N=1463) of Han Chinese ethnicity. In line with existing literature, women outperformed men in the RMET. Men showed significantly lower 2D:4D ratio compared to women, indicating higher exposure to testosterone during the prenatal period. Interestingly, variation in the OXTR gene was found to interact with 2D:4D to predict men's (but not women's) RMET performance. Among men with GG allelic variation, those with low fetal testosterone performed better on the RMET, compared to men with GG and high fetal testosterone, suggesting greater identification of another's emotional state. Taken together, our data lend unique support to the mutual influence of the oxytocin and testosterone systems in shaping core aspect of human social cognition early in development, further suggesting that this effect is gender-specific.

The eyes test is influenced more by artistic inclination and less by sex

The "Reading the Mind in the Eyes" test was developed by Baron-Cohen and his co-workers. This test provides them the unique opportunity to evaluate social cognition assessing the ability to recognize the mental state of others using only the expressions around the eyes. In healthy populations, however, it has produced conflicting results, particularly regarding sex differences and number of items to use. In this study we performed two studies: The first one investigated the presence of gender effects and the sensitivity of test stimuli; the second one considered other individual factors (i.e., artistic attitude, social empathy and personality traits) that could influence the ability to understand emotions from gaze. Our results demonstrated a sex effect, which can be more or less attenuated by the nature of the stimuli. This could be as aforementioned the result of the following, empathy or artistic attitude in being proficient in understanding the mental states of others.

Oxytocin effects on mind-reading are moderated by experiences of maternal love withdrawal: an fMRI study

The neuropeptide oxytocin has been shown to stimulate a range of social behaviors. However, recent studies indicate that the effects of intranasal oxytocin are more nuanced than previously thought and that contextual factors and individual characteristics moderate the beneficiary oxytocin effects. In this randomized-controlled trial we examine the influence of intranasally administered oxytocin on neural activity during mind-reading with fMRI, taking into account harsh caregiving experiences as a potential moderator. Participants were 50 women who received a nasal spray containing either 16 IU of oxytocin or a placebo and had reported how often their mother used love withdrawal as a disciplinary strategy. Participants performed an adapted version of the Reading the Mind in the Eyes Test (RMET), a task which requires individuals to infer mental states by looking at photographs of the eye region of faces. We found that oxytocin enhanced neural activation in the superior temporal gyrus (STG) and insula during the RMET. Moreover, oxytocin increased RMET performance outside the scanner. However, the oxytocin induced changes in STG activation and RMET performance were only brought about in potentially less socially proficient individuals who had low RMET performance, that is, participants reporting higher levels of maternal love withdrawal.

The Eyes Test as a Measure of Individual Differences: How much of the Variance Reflects Verbal IQ?

Developed by Baron-Cohen et al. (1997, 2001), the Reading the Mind in the Eyes Test requires individuals to determine mental states from photos of pairs of eyes. Used in over 250 studies, it has been conceptualized as an advanced theory of mind test that is relatively free of general cognitive abilities. Given the sensitivity of the instrument, many studies with healthy adult samples have used this instrument as a measure of individual differences in social-perceptual processes that contribute to theory of mind and overall phenotype. We administered the two-subtest Wechsler Abbreviated Scale of Intelligence, a face-processing task (Cambridge Face Memory Test), and the Eyes Test to 42 college students. Surprisingly, verbal IQ contributed significantly to the variance in Eyes Test performance while the face perception measure did not. These findings have both practical and theoretical ramifications for interpreting Eyes Test results in normative adult samples.