Advancing Naturalistic Affective Science with Deep Learning

Feelings are Messy: The Feelings We Study in Affective Science Should Be Too

Affective science has taken up the challenge of building a bridge between basic affective science and practical applications. The articles in the Future of Affective Science issue lay out methodological and conceptual frameworks that allow us to expand affective science into real-world settings and to handle naturalistic methods. Along with these advances, accomplishing this goal will require additionally refocusing the types of experiences we study, and the measures of experience we are interested in. This paper explores the necessity for basic affective science to embrace the messy and complex nature of human emotion in order to bridge the gap between theoretical concepts and real-world applicability. Specifically, this involves studying experiences that do not fit as neatly into dominant conceptual frameworks, such as valenced scales and the most common discrete emotion categories, and that may be more difficult to measure or experimentally control. This makes the gap between affective science and real-world feelings larger. To move the field towards incorporating emotional complexity in an empirical manner, I propose measurement standards that err on the side of less fixed-choice options and using stimuli chosen for their potential to elicit highly complex responses over time within the same individual. Designing studies that can measure these experiences will push emotion theories to explain data they were not originally designed for, likely leading to refinement and collaboration. These approaches will help capture the full spectrum of human emotional experience, leading to a more nuanced and applicable understanding of affective science.

A Louder Call for the Integration of Multiple Nonverbal Channels in the Study of Affect

Affective science has increasingly sought to represent emotional experiences multimodally, measuring affect through a combination of self-report ratings, linguistic output, physiological measures, and/or nonverbal expressions. However, despite widespread recognition that non-facial nonverbal cues are an important facet of expressive behavior, measures of nonverbal expressions commonly focus solely on facial movements. This Commentary represents a call for affective scientists to integrate a larger range of nonverbal cues-including gestures, postures, and vocal cues-alongside facial cues in efforts to represent the experience of emotion and its communication. Using the measurement and analysis of vocal cues as an illustrative case, the Commentary considers challenges, potential solutions, and the theoretical and translational significance of working to integrate multiple nonverbal channels in the study of affect.

Affectivism and the Emotional Elephant: How a Componential Approach Can Reconcile Opposing Theories to Serve the Future of Affective Sciences

This article discusses how the affectivism framework and the componential approach to emotion may serve the future of affective sciences. A particular aim of the article is to show that an appraisal-based componential approach to emotion can help reconcile opposing theories. It begins by contextualizing the evolution of emotion science within the framework of affectivism, acknowledging that the significant epistemological differences between various theories have paradoxically spurred interest in studying emotion across various perspectives and disciplines. If affectivism is regarded as the pursuit of a deeper understanding of not only emotions and other affective processes but also cognitive and behavioral processes, then its success can be partly attributed to the existence of multiple approaches, allowing each discipline and perspective to advance using the most suitable theory and methodology. We contend that a componential approach reveals that the five principal theories of emotion have each focused on one of five components of emotion. Overall, based on the analysis of several articles published in the special issue on the future of affective science, we argue that affective scientists are well equipped not only to build a future in which conceptual and methodological tools will be used to test diverging hypotheses between competing theories but also to acknowledge and celebrate where such theories converge.

Understanding Naturalistic Facial Expressions with Deep Learning and Multimodal Large Language Models

This paper provides a comprehensive overview of affective computing systems for facial expression recognition (FER) research in naturalistic contexts. The first section presents an updated account of user-friendly FER toolboxes incorporating state-of-the-art deep learning models and elaborates on their neural architectures, datasets, and performances across domains. These sophisticated FER toolboxes can robustly address a variety of challenges encountered in the wild such as variations in illumination and head pose, which may otherwise impact recognition accuracy. The second section of this paper discusses multimodal large language models (MLLMs) and their potential applications in affective science. MLLMs exhibit human-level capabilities for FER and enable the quantification of various contextual variables to provide context-aware emotion inferences. These advancements have the potential to revolutionize current methodological approaches for studying the contextual influences on emotions, leading to the development of contextualized emotion models.

The Future of Affective Science: Introduction to the Special Issue

Modern affective science-the empirical study of emotional responding and affective experience-has been active for a half-century. The Future of Affective Science special issue considers the history of this field and proposes new directions for the decades ahead. Contributors represent diverse theoretical perspectives, methodological expertise, and domains of study, and the special issue includes both literature reviews and new empirical studies as illustrations. This introductory article synthesizes the contributions, articulating the broader context of the current status of our field and highlighting common themes across articles as well as gaps notable even in this special issue. Sections of the article address theoretical and conceptual issues, research methodology, the questions we ask, and translation of basic affective science to applied domains. We conclude that much has been learned from the first 50 years of affective science, and it is now time for new theories, new research questions, and innovative methods for the decades ahead.