EmoCodes: a Standardized Coding System for Socio-emotional Content in Complex Video Stimuli

Social information processing is vital for inferring emotional states in others, yet affective neuroscience has only begun to scratch the surface of how we represent emotional information in the brain. Most previous affective neuroscience work has used isolated stimuli such as static images of affective faces or scenes to probe affective processing. While this work has provided rich insight to the initial stages of emotion processing (encoding cues), activation to isolated stimuli provides limited insight into later phases of emotion processing such as interpretation of cues or interactions between cues and established cognitive schemas. Recent work has highlighted the potential value of using complex video stimuli to probe socio-emotional processing, highlighting the need to develop standardized video coding schemas as this exciting field expands. Toward that end, we present a standardized and open-source coding system for complex videos, two fully coded videos, and a video and code processing Python library. The EmoCodes manual coding system provides an externally validated and replicable system for coding complex cartoon stimuli, with future plans to validate the system for other video types. The emocodes Python library provides automated tools for extracting low-level features from video files as well as tools for summarizing and analyzing the manual codes for suitability of use in neuroimaging analysis. Materials can be freely accessed at https://emocodes.org/. These tools represent an important step toward replicable and standardized study of socio-emotional processing using complex video stimuli.

SUPPLEMENTARY INFORMATION

The online version contains supplementary material available at 10.1007/s42761-021-00100-7.

Functional connectivity patterns predict naturalistic viewing versus rest across development

Cognitive states, such as rest and task engagement, share an 'intrinsic' functional network organization that is subject to minimal variation over time and yields stable signatures within an individual. Importantly, there are also transient state-specific functional connectivity (FC) patterns that vary across neural states. Here, we examine functional brain organization differences that underlie distinct states in a cross-sectional developmental sample. We compare FC fMRI data acquired during naturalistic viewing (i.e., movie-watching) and resting-state paradigms in a large cohort of 157 children and young adults aged 6-20. Naturalistic paradigms are commonly implemented in pediatric research because they maintain the child's attention and contribute to reduced head motion. It remains unknown, however, to what extent the brain-wide functional network organization is comparable during movie-watching and rest across development. Here, we identify a widespread FC pattern that predicts whether individuals are watching a movie or resting. Specifically, we develop a model for prediction of multilevel neural effects (termed PrimeNet), which can with high reliability distinguish between movie-watching and rest irrespective of age and that generalizes across movies. In turn, we characterize FC patterns in the most predictive functional networks for movie-watching versus rest and show that these patterns can indeed vary as a function of development. Collectively, these effects highlight a 'core' FC pattern that is robustly associated with naturalistic viewing, which also exhibits change across age. These results, focused here on naturalistic viewing, provide a roadmap for quantifying state-specific functional neural organization across development, which may reveal key variation in neurodevelopmental trajectories associated with behavioral phenotypes.