Advancing Naturalistic Affective Science with Deep Learning

How trait impressions of faces shape subsequent mental state inferences

People form impressions of one another in a split second from faces. However, people also infer others' momentary mental states on the basis of context-for example, one might infer that somebody feels encouraged from the fact that they are receiving constructive feedback. How do trait judgements of faces influence these context-based mental state inferences? In this Registered Report, we asked participants to infer the mental states of unfamiliar people, identified by their neutral faces, under specific contexts. To increase generalizability, we representatively sampled all stimuli from inclusive sets using computational methods. We tested four hypotheses: that trait impressions of faces (1) are correlated with subsequent mental state inferences in a range of contexts, (2) alter the dimensional space that underlies mental state inferences, (3) are associated with specific mental state dimensions in this space and (4) causally influence mental state inferences. We found evidence in support of all hypotheses.

Naturalistic multimodal emotion data with deep learning can advance the theoretical understanding of emotion

What are emotions? Despite being a century-old question, emotion scientists have yet to agree on what emotions exactly are. Emotions are diversely conceptualised as innate responses (evolutionary view), mental constructs (constructivist view), cognitive evaluations (appraisal view), or self-organising states (dynamical systems view). This enduring fragmentation likely stems from the limitations of traditional research methods, which often adopt narrow methodological approaches. Methods from artificial intelligence (AI), particularly those leveraging big data and deep learning, offer promising approaches for overcoming these limitations. By integrating data from multimodal markers of emotion, including subjective experiences, contextual factors, brain-bodily physiological signals and expressive behaviours, deep learning algorithms can uncover and map their complex relationships within multidimensional spaces. This multimodal emotion framework has the potential to provide novel, nuanced insights into long-standing questions, such as whether emotion categories are innate or learned and whether emotions exhibit coherence or degeneracy, thereby refining emotion theories. Significant challenges remain, particularly in obtaining comprehensive naturalistic multimodal emotion data, highlighting the need for advances in synchronous measurement of naturalistic multimodal emotion.

Feelings are Messy: The Feelings We Study in Affective Science Should Be Too

Affective science has taken up the challenge of building a bridge between basic affective science and practical applications. The articles in the Future of Affective Science issue lay out methodological and conceptual frameworks that allow us to expand affective science into real-world settings and to handle naturalistic methods. Along with these advances, accomplishing this goal will require additionally refocusing the types of experiences we study, and the measures of experience we are interested in. This paper explores the necessity for basic affective science to embrace the messy and complex nature of human emotion in order to bridge the gap between theoretical concepts and real-world applicability. Specifically, this involves studying experiences that do not fit as neatly into dominant conceptual frameworks, such as valenced scales and the most common discrete emotion categories, and that may be more difficult to measure or experimentally control. This makes the gap between affective science and real-world feelings larger. To move the field towards incorporating emotional complexity in an empirical manner, I propose measurement standards that err on the side of less fixed-choice options and using stimuli chosen for their potential to elicit highly complex responses over time within the same individual. Designing studies that can measure these experiences will push emotion theories to explain data they were not originally designed for, likely leading to refinement and collaboration. These approaches will help capture the full spectrum of human emotional experience, leading to a more nuanced and applicable understanding of affective science.

Affectivism and the Emotional Elephant: How a Componential Approach Can Reconcile Opposing Theories to Serve the Future of Affective Sciences

This article discusses how the affectivism framework and the componential approach to emotion may serve the future of affective sciences. A particular aim of the article is to show that an appraisal-based componential approach to emotion can help reconcile opposing theories. It begins by contextualizing the evolution of emotion science within the framework of affectivism, acknowledging that the significant epistemological differences between various theories have paradoxically spurred interest in studying emotion across various perspectives and disciplines. If affectivism is regarded as the pursuit of a deeper understanding of not only emotions and other affective processes but also cognitive and behavioral processes, then its success can be partly attributed to the existence of multiple approaches, allowing each discipline and perspective to advance using the most suitable theory and methodology. We contend that a componential approach reveals that the five principal theories of emotion have each focused on one of five components of emotion. Overall, based on the analysis of several articles published in the special issue on the future of affective science, we argue that affective scientists are well equipped not only to build a future in which conceptual and methodological tools will be used to test diverging hypotheses between competing theories but also to acknowledge and celebrate where such theories converge.

A Louder Call for the Integration of Multiple Nonverbal Channels in the Study of Affect

Affective science has increasingly sought to represent emotional experiences multimodally, measuring affect through a combination of self-report ratings, linguistic output, physiological measures, and/or nonverbal expressions. However, despite widespread recognition that non-facial nonverbal cues are an important facet of expressive behavior, measures of nonverbal expressions commonly focus solely on facial movements. This Commentary represents a call for affective scientists to integrate a larger range of nonverbal cues-including gestures, postures, and vocal cues-alongside facial cues in efforts to represent the experience of emotion and its communication. Using the measurement and analysis of vocal cues as an illustrative case, the Commentary considers challenges, potential solutions, and the theoretical and translational significance of working to integrate multiple nonverbal channels in the study of affect.

Understanding Naturalistic Facial Expressions with Deep Learning and Multimodal Large Language Models

This paper provides a comprehensive overview of affective computing systems for facial expression recognition (FER) research in naturalistic contexts. The first section presents an updated account of user-friendly FER toolboxes incorporating state-of-the-art deep learning models and elaborates on their neural architectures, datasets, and performances across domains. These sophisticated FER toolboxes can robustly address a variety of challenges encountered in the wild such as variations in illumination and head pose, which may otherwise impact recognition accuracy. The second section of this paper discusses multimodal large language models (MLLMs) and their potential applications in affective science. MLLMs exhibit human-level capabilities for FER and enable the quantification of various contextual variables to provide context-aware emotion inferences. These advancements have the potential to revolutionize current methodological approaches for studying the contextual influences on emotions, leading to the development of contextualized emotion models.

The Future of Affective Science: Introduction to the Special Issue

Modern affective science-the empirical study of emotional responding and affective experience-has been active for a half-century. The Future of Affective Science special issue considers the history of this field and proposes new directions for the decades ahead. Contributors represent diverse theoretical perspectives, methodological expertise, and domains of study, and the special issue includes both literature reviews and new empirical studies as illustrations. This introductory article synthesizes the contributions, articulating the broader context of the current status of our field and highlighting common themes across articles as well as gaps notable even in this special issue. Sections of the article address theoretical and conceptual issues, research methodology, the questions we ask, and translation of basic affective science to applied domains. We conclude that much has been learned from the first 50 years of affective science, and it is now time for new theories, new research questions, and innovative methods for the decades ahead.