Natural facial motion enhances cortical responses to faces

A quantitative comparison of atlas parcellations on the human superior temporal sulcus

The superior temporal sulcus (STS) has a functional topography that has been difficult to characterize through traditional approaches. Automated atlas parcellations may be one solution while also being beneficial for both dimensional reduction and standardizing regions of interest, but they yield very different boundary definitions along the STS. Here we evaluate how well machine learning classifiers can correctly identify six social cognitive tasks from STS activation patterns dimensionally reduced using four popular atlases (Glasser et al., 2016; Gordon et al., 2016; Power et al., 2011 as projected onto the surface by Arslan et al., 2018; Schaefer et al., 2018). Functional data was summarized within each STS parcel in one of four ways, then subjected to leave-one-subject-out cross-validation SVM classification. We found that the classifiers could readily label conditions when data was parcellated using any of the four atlases, evidence that dimensional reduction to parcels did not compromise functional fingerprints. Mean activation for the social conditions was the most effective metric for classification in the right STS, whereas all the metrics classified equally well in the left STS. Interestingly, even atlases constructed from random parcellation schemes (null atlases) classified the conditions with high accuracy. We therefore conclude that the complex activation maps on the STS are readily differentiated at a coarse granular level, despite a strict topography having not yet been identified. Further work is required to identify what features have greatest potential to improve the utility of atlases in replacing functional localizers.

Age-dependent changes in the anger superiority effect: Evidence from a visual search task

The perception of threatening facial expressions is a critical skill necessary for detecting the emotional states of others and responding appropriately. The anger superiority effect hypothesis suggests that individuals are better at processing and identifying angry faces compared with other nonthreatening facial expressions. In adults, the anger superiority effect is present even after controlling for the bottom-up visual saliency, and when ecologically valid stimuli are used. However, it is as yet unclear whether this effect is present in children. To fill this gap, we tested the anger superiority effect in children ages 6-14 years in a visual search task by using emotional dynamic stimuli and equating the visual salience of target and distractors. The results suggest that in childhood, the angry superiority effect consists of improved accuracy in detecting angry faces, while in adolescence, the ability to discriminate angry faces undergoes further development, enabling faster and more accurate threat detection.

Modeling short visual events through the BOLD moments video fMRI dataset and metadata

Studying the neural basis of human dynamic visual perception requires extensive experimental data to evaluate the large swathes of functionally diverse brain neural networks driven by perceiving visual events. Here, we introduce the BOLD Moments Dataset (BMD), a repository of whole-brain fMRI responses to over 1000 short (3 s) naturalistic video clips of visual events across ten human subjects. We use the videos' extensive metadata to show how the brain represents word- and sentence-level descriptions of visual events and identify correlates of video memorability scores extending into the parietal cortex. Furthermore, we reveal a match in hierarchical processing between cortical regions of interest and video-computable deep neural networks, and we showcase that BMD successfully captures temporal dynamics of visual events at second resolution. With its rich metadata, BMD offers new perspectives and accelerates research on the human brain basis of visual event perception.

Taking consciousness for real: Increasing the ecological validity of the study of conscious vs. unconscious processes

The study of consciousness has developed well-controlled, rigorous methods for manipulating and measuring consciousness. Yet, in the process, experimental paradigms grew farther away from everyday conscious and unconscious processes, which raises the concern of ecological validity. In this review, we suggest that the field can benefit from adopting a more ecological approach, akin to other fields of cognitive science. There, this approach challenged some existing hypotheses, yielded stronger effects, and enabled new research questions. We argue that such a move is critical for studying consciousness, where experimental paradigms tend to be artificial and small effect sizes are relatively prevalent. We identify three paths for doing so-changing the stimuli and experimental settings, changing the measures, and changing the research questions themselves-and review works that have already started implementing such approaches. While acknowledging the inherent challenges, we call for increasing ecological validity in consciousness studies.

The SocialVidStim: a video database of positive and negative social evaluation stimuli for use in social cognitive neuroscience paradigms

This paper describes the SocialVidStim-a database of video stimuli available to the scientific community depicting positive and negative social evaluative and neutral statements. The SocialVidStim comprises 53 diverse individuals reflecting the demographic makeup of the USA, ranging from 9 to 41 years old, saying 20-60 positive and 20-60 negative social evaluative statements (e.g. 'You are a very trustworthy/annoying person'), and 20-60 neutral statements (e.g. 'The sky is blue'), totaling 5793 videos post-production. The SocialVidStim are designed for use in behavioral and functional magetic resonance imaging paradigms, across developmental stages, in diverse populations. This study describes stimuli development and reports initial validity and reliability data on a subset videos (N = 1890) depicting individuals aged 18-41 years. Raters perceive videos as expected: positive videos elicit positively valenced ratings, negative videos elicit negatively valenced ratings and neutral videos are rated as neutral. Test-retest reliability data demonstrate intraclass correlations in the good-to-excellent range for negative and positive videos and the moderate range for neutral videos. We also report small effects on valence and arousal that should be considered during stimuli selection, including match between rater and actor sex and actor believability. The SocialVidStim is a resource for researchers and we offer suggestions for using the SocialVidStim in future research.

Recognizing facial expressions of emotion amid noise: A dynamic advantage

Humans communicate internal states through complex facial movements shaped by biological and evolutionary constraints. Although real-life social interactions are flooded with dynamic signals, current knowledge on facial expression recognition mainly arises from studies using static face images. This experimental bias might stem from previous studies consistently reporting that young adults minimally benefit from the richer dynamic over static information, whereas children, the elderly, and clinical populations very strongly do (Richoz, Jack, Garrod, Schyns, & Caldara, 2015, Richoz, Jack, Garrod, Schyns, & Caldara, 2018b). These observations point to a near-optimal facial expression decoding system in young adults, almost insensitive to the advantage of dynamic over static cues. Surprisingly, no study has yet tested the idea that such evidence might be rooted in a ceiling effect. To this aim, we asked 70 healthy young adults to perform static and dynamic facial expression recognition of the six basic expressions while parametrically and randomly varying the low-level normalized phase and contrast signal (0%-100%) of the faces. As predicted, when 100% face signals were presented, static and dynamic expressions were recognized with equal efficiency with the exception of those with the most informative dynamics (i.e., happiness and surprise). However, when less signal was available, dynamic expressions were all better recognized than their static counterpart (peaking at ∼20%). Our data show that facial movements increase our ability to efficiently identify emotional states of others under the suboptimal visual conditions that can occur in everyday life. Dynamic signals are more effective and sensitive than static ones for decoding all facial expressions of emotion for all human observers.

Inter-subject correlation during long narratives reveals widespread neural correlates of reading ability

Recent work using fMRI inter-subject correlation analysis has provided new information about the brain's response to video and audio narratives, particularly in frontal regions not typically activated by single words. This approach is very well suited to the study of reading, where narrative is central to natural experience. But since past reading paradigms have primarily presented single words or phrases, the influence of narrative on semantic processing in the brain - and how that influence might change with reading ability - remains largely unexplored. In this study, we presented coherent stories to adolescents and young adults with a wide range of reading abilities. The stories were presented in alternating visual and auditory blocks. We used a dimensional inter-subject correlation analysis to identify regions in which better and worse readers had varying levels of consistency with other readers. This analysis identified a widespread set of brain regions in which activity timecourses were more similar among better readers than among worse readers. These differences were not detected with standard block activation analyses. Worse readers had higher correlation with better readers than with other worse readers, suggesting that the worse readers had "idiosyncratic" responses rather than using a single compensatory mechanism. Close inspection confirmed that these differences were not explained by differences in IQ or motion. These results suggest an expansion of the current view of where and how reading ability is reflected in the brain, and in doing so, they establish inter-subject correlation as a sensitive tool for future studies of reading disorders.

Identifying the cortical face network with dynamic face stimuli: A large group fMRI study

Functional magnetic resonance imaging (fMRI) studies have identified a network of face-selective regions distributed across the human brain. In the present study, we analyzed data from a large group of gender-balanced participants to investigate how reliably these face-selective regions could be identified across both cerebral hemispheres. Participants ( N =52) were scanned with fMRI while viewing short videos of faces, bodies, and objects. Results revealed that five face-selective regions: the fusiform face area (FFA), posterior superior temporal sulcus (pSTS), anterior superior temporal sulcus (aSTS), inferior frontal gyrus (IFG) and the amygdala were all larger in the right than in the left hemisphere. The occipital face area (OFA) was larger in the right hemisphere as well, but the difference between the hemispheres was not significant. The neural response to moving faces was also greater in face-selective regions in the right than in the left hemisphere. An additional analysis revealed that the pSTS and IFG were significantly larger in the right hemisphere compared to other face-selective regions. This pattern of results demonstrates that moving faces are preferentially processed in the right hemisphere and that the pSTS and IFG appear to be the strongest drivers of this laterality. An analysis of gender revealed that face-selective regions were typically larger in females ( N =26) than males ( N =26), but this gender difference was not statistically significant.

The role of the upper and lower face in the recognition of facial identity in dynamic stimuli

Studies with static faces find that upper face halves are more easily recognized than lower face halves-an upper-face advantage. However, faces are usually encountered as dynamic stimuli, and there is evidence that dynamic information influences face identity recognition. This raises the question of whether dynamic faces also show an upper-face advantage. The objective of this study was to examine whether familiarity for recently learned faces was more accurate for upper or lower face halves, and whether this depended upon whether the face was presented as static or dynamic. In Experiment 1, subjects learned a total of 12 faces--6 static images and 6 dynamic video-clips of actors in silent conversation. In experiment 2, subjects learned 12 faces, all dynamic video-clips. During the testing phase of Experiments 1 (between subjects) and 2 (within subjects), subjects were asked to recognize upper and lower face halves from either static images and/or dynamic clips. The data did not provide evidence for a difference in the upper-face advantage between static and dynamic faces. However, in both experiments, we found an upper-face advantage, consistent with prior literature, for female faces, but not for male faces. In conclusion, the use of dynamic stimuli may have little effect on the presence of an upper-face advantage, especially when the static comparison contains a series of static images, rather than a single static image, and is of sufficient image quality. Future studies could investigate the influence of face gender on the presence of an upper-face advantage.

Quantifying dynamic facial expressions under naturalistic conditions

Facial affect is expressed dynamically - a giggle, grimace, or an agitated frown. However, the characterisation of human affect has relied almost exclusively on static images. This approach cannot capture the nuances of human communication or support the naturalistic assessment of affective disorders. Using the latest in machine vision and systems modelling, we studied dynamic facial expressions of people viewing emotionally salient film clips. We found that the apparent complexity of dynamic facial expressions can be captured by a small number of simple spatiotemporal states - composites of distinct facial actions, each expressed with a unique spectral fingerprint. Sequential expression of these states is common across individuals viewing the same film stimuli but varies in those with the melancholic subtype of major depressive disorder. This approach provides a platform for translational research, capturing dynamic facial expressions under naturalistic conditions and enabling new quantitative tools for the study of affective disorders and related mental illnesses.

Measuring the response to visually presented faces in the human lateral prefrontal cortex

Neuroimaging studies identify multiple face-selective areas in the human brain. In the current study, we compared the functional response of the face area in the lateral prefrontal cortex to that of other face-selective areas. In Experiment 1, participants (n = 32) were scanned viewing videos containing faces, bodies, scenes, objects, and scrambled objects. We identified a face-selective area in the right inferior frontal gyrus (rIFG). In Experiment 2, participants (n = 24) viewed the same videos or static images. Results showed that the rIFG, right posterior superior temporal sulcus (rpSTS), and right occipital face area (rOFA) exhibited a greater response to moving than static faces. In Experiment 3, participants (n = 18) viewed face videos in the contralateral and ipsilateral visual fields. Results showed that the rIFG and rpSTS showed no visual field bias, while the rOFA and right fusiform face area (rFFA) showed a contralateral bias. These experiments suggest two conclusions; firstly, in all three experiments, the face area in the IFG was not as reliably identified as face areas in the occipitotemporal cortex. Secondly, the similarity of the response profiles in the IFG and pSTS suggests the areas may perform similar cognitive functions, a conclusion consistent with prior neuroanatomical and functional connectivity evidence.

Functional connectivity dynamics as a function of the fluctuation of tension during film watching

To advance the understanding of the dynamic relationship between brain activities and emotional experiences, we examined the neural patterns of tension, a unique emotion that highly depends on how an event unfolds. Specifically, the present study explored the temporal relationship between functional connectivity patterns within and between different brain functional modules and the fluctuation in tension during film watching. Due to the highly contextualized and time-varying nature of tension, we expected that multiple neural networks would be involved in the dynamic tension experience. Using the neuroimaging data of 546 participants, we conducted a dynamic brain analysis to identify the intra- and inter-module functional connectivity patterns that are significantly correlated with the fluctuation of tension over time. The results showed that the inter-module connectivity of cingulo-opercular network, fronto-parietal network, and default mode network is involved in the dynamic experience of tension. These findings demonstrate a close relationship between brain functional connectivity patterns and emotional dynamics, which supports the importance of functional connectivity dynamics in understanding our cognitive and emotional processes.

Neural Mechanisms of Facial Emotion Recognition in Autism: Distinct Roles for Anterior Cingulate and dlPFC

OBJECTIVE

The present study sought to measure and internally validate neural markers of facial emotion recognition (FER) in adolescents and young adults with ASD to inform targeted intervention.

METHOD

We utilized fMRI to measure patterns of brain activity among individuals with ASD (N = 21) and matched controls (CON; N = 20) 2 s prior to judgments about the identity of six distinct facial emotions (happy, sad, angry, surprised, fearful, disgust).

RESULTS

Predictive modeling of fMRI data (support vector classification; SVC) identified mechanistic roles for brain regions that forecasted correct and incorrect identification of facial emotion as well as sources of errors over these decisions. BOLD signal activation in bilateral insula, anterior cingulate (ACC) and right dorsolateral prefrontal cortex (dlPFC) preceded accurate FER in both controls and ASD. Predictive modeling utilizing SVC confirmed the utility of ACC in forecasting correct decisions in controls but not ASD, and further indicated that a region within the right dlPFC was the source of a type 1 error signal in ASD (i.e. neural marker reflecting an impending correct judgment followed by an incorrect behavioral response) approximately two seconds prior to emotion judgments during fMRI.

CONCLUSIONS

ACC forecasted correct decisions only among control participants. Right dlPFC was the source of a false-positive signal immediately prior to an error about the nature of a facial emotion in adolescents and young adults with ASD, potentially consistent with prior work indicating that dlPFC may play a role in attention to and regulation of emotional experience.

Caricatured facial movements enhance perception of emotional facial expressions

Although faces “in the wild” constantly undergo complicated movements, humans adeptly perceive facial identity and expression. Previous studies, focusing mainly on identity, used photographic caricature to show that distinctive form increases perceived dissimilarity. We tested whether distinctive facial movements showed similar effects, and we focussed on both perception of expression and identity. We caricatured the movements of an animated computer head, using physical motion metrics extracted from videos. We verified that these “ground truth” metrics showed the expected effects: Caricature increased physical dissimilarity between faces differing in expression and those differing in identity. Like the ground truth dissimilarity, participants’ dissimilarity perception was increased by caricature when faces differed in expression. We found these perceived dissimilarities to reflect the “representational geometry” of the ground truth. However, neither of these findings held for faces differing in identity. These findings replicated across two paradigms: pairwise ratings and multiarrangement. In a final study, motion caricature did not improve recognition memory for identity, whether manipulated at study or test. We report several forms of converging evidence for spatiotemporal caricature effects on dissimilarity perception of different expressions. However, more work needs to be done to discover what identity-specific movements can enhance face identification.

Neural correlates of integration processes during dynamic face perception

Integrating the spatiotemporal information acquired from the highly dynamic world around us is essential to navigate, reason, and decide properly. Although this is particularly important in a face-to-face conversation, very little research to date has specifically examined the neural correlates of temporal integration in dynamic face perception. Here we present statistically robust observations regarding the brain activations measured via electroencephalography (EEG) that are specific to the temporal integration. To that end, we generate videos of neutral faces of individuals and non-face objects, modulate the contrast of the even and odd frames at two specific frequencies (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f_1$$\end{document}f1 and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f_2$$\end{document}f2) in an interlaced manner, and measure the steady-state visual evoked potential as participants view the videos. Then, we analyze the intermodulation components (IMs: (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$nf_1\pm mf_2$$\end{document}nf1±mf2), a linear combination of the fundamentals with integer multipliers) that consequently reflect the nonlinear processing and indicate temporal integration by design. We show that electrodes around the medial temporal, inferior, and medial frontal areas respond strongly and selectively when viewing dynamic faces, which manifests the essential processes underlying our ability to perceive and understand our social world. The generation of IMs is only possible if even and odd frames are processed in succession and integrated temporally, therefore, the strong IMs in our frequency spectrum analysis show that the time between frames (1/60 s) is sufficient for temporal integration.

Cortical processing of dynamic bodies in the superior occipito-temporal regions of the infants' brain: Difference from dynamic faces and inversion effect

Previous functional neuroimaging studies imply a crucial role of the superior temporal regions (e.g., superior temporal sulcus: STS) for processing of dynamic faces and bodies. However, little is known about the cortical processing of moving faces and bodies in infancy. The current study used functional near-infrared spectroscopy (fNIRS) to directly compare cortical hemodynamic responses to dynamic faces (videos of approaching people with blurred bodies) and dynamic bodies (videos of approaching people with blurred faces) in infants' brain. We also examined the body-inversion effect in 5- to 8-month-old infants using hemodynamic responses as a measure. We found significant brain activity for the dynamic faces and bodies in the superior area of bilateral temporal cortices in both 5- to 6-month-old and 7- to 8-month-old infants. The hemodynamic responses to dynamic faces occurred across a broader area of cortex in 7- to 8-month-olds than in 5- to 6-month-olds, but we did not find a developmental change for dynamic bodies. There was no significant activation when the stimuli were presented upside down, indicating that these activation patterns did not result from the low-level visual properties of dynamic faces and bodies. Additionally, we found that the superior temporal regions showed a body inversion effect in infants aged over 5 months: the upright dynamic body stimuli induced stronger activation compared to the inverted stimuli. The most important contribution of the present study is that we identified cortical areas responsive to dynamic bodies and faces in two groups of infants (5-6-months and 7-8-months of age) and we found different developmental trends for the processing of bodies and faces.

Reexamining the neural network involved in perception of facial expression: A meta-analysis

Perception of facial expression is essential for social interactions. Although a few competing models have enjoyed some success to map brain regions, they are also facing difficult challenges. The current study used an updated activation likelihood estimation (ALE) method of meta-analysis to explore the involvement of brain regions in facial expression processing. The sample contained 96 functional magnetic resonance imaging (fMRI) and positron emission tomography (PET) studies of healthy adults with the results of whole-brain analyses. The key findings revealed that the ventral pathway, especially the left fusiform face area (FFA) region, was more responsive to facial expression. The left posterior FFA showed strong involvement when participants passively viewing emotional faces without being asked to judge the type of expression or other attributes of the stimuli. Through meta-analytic connectivity modeling (MACM) of the main brain regions in the ventral pathway, we constructed a co-activating neural network as a revised model of facial expression processing that assigns prominent roles to the amygdala, FFA, the occipital gyrus, and the inferior frontal gyrus.

The Human Posterior Superior Temporal Sulcus Samples Visual Space Differently From Other Face-Selective Regions

Neuroimaging studies show that ventral face-selective regions, including the fusiform face area (FFA) and occipital face area (OFA), preferentially respond to faces presented in the contralateral visual field (VF). In the current study we measured the VF response of the face-selective posterior superior temporal sulcus (pSTS). Across 3 functional magnetic resonance imaging experiments, participants viewed face videos presented in different parts of the VF. Consistent with prior results, we observed a contralateral VF bias in bilateral FFA, right OFA (rOFA), and bilateral human motion-selective area MT+. Intriguingly, this contralateral VF bias was absent in the bilateral pSTS. We then delivered transcranial magnetic stimulation (TMS) over right pSTS (rpSTS) and rOFA, while participants matched facial expressions in both hemifields. TMS delivered over the rpSTS disrupted performance in both hemifields, but TMS delivered over the rOFA disrupted performance in the contralateral hemifield only. These converging results demonstrate that the contralateral bias for faces observed in ventral face-selective areas is absent in the pSTS. This difference in VF response is consistent with face processing models proposing 2 functionally distinct pathways. It further suggests that these models should account for differences in interhemispheric connections between the face-selective areas across these 2 pathways.

Visual mechanisms for voice-identity recognition flexibly adjust to auditory noise level

Recognising the identity of voices is a key ingredient of communication. Visual mechanisms support this ability: recognition is better for voices previously learned with their corresponding face (compared to a control condition). This so‐called ‘face‐benefit’ is supported by the fusiform face area (FFA), a region sensitive to facial form and identity. Behavioural findings indicate that the face‐benefit increases in noisy listening conditions. The neural mechanisms for this increase are unknown. Here, using functional magnetic resonance imaging, we examined responses in face‐sensitive regions while participants recognised the identity of auditory‐only speakers (previously learned by face) in high (SNR −4 dB) and low (SNR +4 dB) levels of auditory noise. We observed a face‐benefit in both noise levels, for most participants (16 of 21). In high‐noise, the recognition of face‐learned speakers engaged the right posterior superior temporal sulcus motion‐sensitive face area (pSTS‐mFA), a region implicated in the processing of dynamic facial cues. The face‐benefit in high‐noise also correlated positively with increased functional connectivity between this region and voice‐sensitive regions in the temporal lobe in the group of 16 participants with a behavioural face‐benefit. In low‐noise, the face‐benefit was robustly associated with increased responses in the FFA and to a lesser extent the right pSTS‐mFA. The findings highlight the remarkably adaptive nature of the visual network supporting voice‐identity recognition in auditory‐only listening conditions.

Cortical response to naturalistic stimuli is largely predictable with deep neural networks

Naturalistic stimuli, such as movies, activate a substantial portion of the human brain, invoking a response shared across individuals. Encoding models that predict neural responses to arbitrary stimuli can be very useful for studying brain function. However, existing models focus on limited aspects of naturalistic stimuli, ignoring the dynamic interactions of modalities in this inherently context-rich paradigm. Using movie-watching data from the Human Connectome Project, we build group-level models of neural activity that incorporate several inductive biases about neural information processing, including hierarchical processing, temporal assimilation, and auditory-visual interactions. We demonstrate how incorporating these biases leads to remarkable prediction performance across large areas of the cortex, beyond the sensory-specific cortices into multisensory sites and frontal cortex. Furthermore, we illustrate that encoding models learn high-level concepts that generalize to task-bound paradigms. Together, our findings underscore the potential of encoding models as powerful tools for studying brain function in ecologically valid conditions.

Evidence for a Third Visual Pathway Specialized for Social Perception

Existing models propose that primate visual cortex is divided into two functionally distinct pathways. The ventral pathway computes the identity of an object; the dorsal pathway computes the location of an object, and the actions related to that object. Despite remaining influential, the two visual pathways model requires revision. Both human and non-human primate studies reveal the existence of a third visual pathway on the lateral brain surface. This third pathway projects from early visual cortex, via motion-selective areas, into the superior temporal sulcus (STS). Studies demonstrating that the STS computes the actions of moving faces and bodies (e.g., expressions, eye-gaze, audio-visual integration, intention, and mood) show that the third visual pathway is specialized for the dynamic aspects of social perception.

Neural processing of facial identity and expression in adults with and without autism: A multi-method approach

The ability to recognize faces and facial expressions is a common human talent. It has, however, been suggested to be impaired in individuals with autism spectrum disorder (ASD). The goal of this study was to compare the processing of facial identity and emotion between individuals with ASD and neurotypicals (NTs). Behavioural and functional magnetic resonance imaging (fMRI) data from 46 young adults (aged 17-23 years, N_ASD = 22, N_NT = 24) was analysed. During fMRI data acquisition, participants discriminated between short clips of a face transitioning from a neutral to an emotional expression. Stimuli included four identities and six emotions. We performed behavioural, univariate, multi-voxel, adaptation and functional connectivity analyses to investigate potential group differences. The ASD-group did not differ from the NT-group on behavioural identity and expression processing tasks. At the neural level, we found no differences in average neural activation, neural activation patterns and neural adaptation to faces in face-related brain regions. In terms of functional connectivity, we found that amygdala seems to be more strongly connected to inferior occipital cortex and V1 in individuals with ASD. Overall, the findings indicate that neural representations of facial identity and expression have a similar quality in individuals with and without ASD, but some regions containing these representations are connected differently in the extended face processing network.

The Trajectory of Hemispheric Lateralization in the Core System of Face Processing: A Cross-Sectional Functional Magnetic Resonance Imaging Pilot Study

Face processing is mediated by a distributed neural network commonly divided into a “core system” and an “extended system.” The core system consists of several, typically right-lateralized brain regions in the occipito-temporal cortex, including the occipital face area (OFA), the fusiform face area (FFA) and the posterior superior temporal sulcus (pSTS). It was recently proposed that the face processing network is initially bilateral and becomes right-specialized in the course of the development of reading abilities due to the competition between language-related regions in the left occipito-temporal cortex (e.g., the visual word form area, VWFA) and the FFA for common neural resources. In the present pilot study, we assessed the neural face processing network in 12 children (aged 7–9 years) and 10 adults with functional magnetic resonance imaging (fMRI). The hemispheric lateralization of the core face regions was compared between both groups. The study had two goals: First, we aimed to establish an fMRI paradigm suitable for assessing activation in the core system of face processing in young children at the single subject level. Second, we planned to collect data for a power analysis to calculate the necessary group size for a large-scale cross-sectional imaging study assessing the ontogenetic development of the lateralization of the face processing network, with focus on the FFA. It was possible to detect brain activity in the core system of 75% of children at the single subject level. The average scan-to-scan motion of the included children was comparable to adults, ruling out that potential activation differences between groups are caused by unequal motion artifacts. Hemispheric lateralization of the FFA was 0.07 ± 0.48 in children (indicating bilateral activation) and −0.32 ± 0.52 in adults (indicating right-hemispheric dominance). These results thus showed, as expected, a trend for increased lateralization in adults. The estimated effect size for the FFA lateralization difference was d = 0.78 (indicating medium to large effects). An adequately powered follow-up study (sensitivity 0.8) testing developmental changes of FFA lateralization would therefore require the inclusion of 18 children and 26 adults.

Manipulating the structure of natural scenes using wavelets to study the functional architecture of perceptual hierarchies in the brain

Functional neuroimaging experiments that employ naturalistic stimuli (natural scenes, films, spoken narratives) provide insights into cognitive function "in the wild". Natural stimuli typically possess crowded, spectrally dense, dynamic, and multimodal properties within a rich multiscale structure. However, when using natural stimuli, various challenges exist for creating parametric manipulations with tight experimental control. Here, we revisit the typical spectral composition and statistical dependences of natural scenes, which distinguish them from abstract stimuli. We then demonstrate how to selectively degrade subtle statistical dependences within specific spatial scales using the wavelet transform. Such manipulations leave basic features of the stimuli, such as luminance and contrast, intact. Using functional neuroimaging of human participants viewing degraded natural images, we demonstrate that cortical responses at different levels of the visual hierarchy are differentially sensitive to subtle statistical dependences in natural images. This demonstration supports the notion that perceptual systems in the brain are optimally tuned to the complex statistical properties of the natural world. The code to undertake these stimulus manipulations, and their natural extension to dynamic natural scenes (films), is freely available.

A sensorimotor control framework for understanding emotional communication and regulation

Our research team was asked to consider the relationship of the neuroscience of sensorimotor control to the language of emotions and feelings. Actions are the principal means for the communication of emotions and feelings in both humans and other animals, and the allostatic mechanisms controlling action also apply to the regulation of emotional states by the self and others. We consider how motor control of hierarchically organised, feedback-based, goal-directed action has evolved in humans, within a context of consciousness, appraisal and cultural learning, to serve emotions and feelings. In our linguistic analysis, we found that many emotion and feelings words could be assigned to stages in the sensorimotor learning process, but the assignment was often arbitrary. The embodied nature of emotional communication means that action words are frequently used, but that the meanings or senses of the word depend on its contextual use, just as the relationship of an action to an emotion is also contextually dependent.

Anterior superior temporal sulcus is specialized for non-rigid facial motion in both monkeys and humans

Facial motion plays a fundamental role in the recognition of facial expressions in primates, but the neural substrates underlying this special type of biological motion are not well understood. Here, we used fMRI to investigate the extent to which the specialization for facial motion is represented in the visual system and compared the neural mechanisms for the processing of non-rigid facial motion in macaque monkeys and humans. We defined the areas specialized for facial motion as those significantly more activated when subjects perceived the motion caused by dynamic faces (dynamic faces > static faces) than when they perceived the motion caused by dynamic non-face objects (dynamic objects > static objects). We found that, in monkeys, significant activations evoked by facial motion were in the fundus of anterior superior temporal sulcus (STS), which overlapped the anterior fundus face patch. In humans, facial motion activated three separate foci in the right STS: posterior, middle, and anterior STS, with the anterior STS location showing the most selectivity for facial motion compared with other facial motion areas. In both monkeys and humans, facial motion shows a gradient preference as one progresses anteriorly along the STS. Taken together, our results indicate that monkeys and humans share similar neural substrates within the anterior temporal lobe specialized for the processing of non-rigid facial motion.

Dorsal-movement and ventral-form regions are functionally connected during visual-speech recognition

Faces convey social information such as emotion and speech. Facial emotion processing is supported via interactions between dorsal-movement and ventral-form visual cortex regions. Here, we explored, for the first time, whether similar dorsal-ventral interactions (assessed via functional connectivity), might also exist for visual-speech processing. We then examined whether altered dorsal-ventral connectivity is observed in adults with high-functioning autism spectrum disorder (ASD), a disorder associated with impaired visual-speech recognition. We acquired functional magnetic resonance imaging (fMRI) data with concurrent eye tracking in pairwise matched control and ASD participants. In both groups, dorsal-movement regions in the visual motion area 5 (V5/MT) and the temporal visual speech area (TVSA) were functionally connected to ventral-form regions (i.e., the occipital face area [OFA] and the fusiform face area [FFA]) during the recognition of visual speech, in contrast to the recognition of face identity. Notably, parts of this functional connectivity were decreased in the ASD group compared to the controls (i.e., right V5/MT-right OFA, left TVSA-left FFA). The results confirmed our hypothesis that functional connectivity between dorsal-movement and ventral-form regions exists during visual-speech processing. Its partial dysfunction in ASD might contribute to difficulties in the recognition of dynamic face information relevant for successful face-to-face communication.

Amygdala responds to direct gaze in real but not in computer-generated faces

Computer-generated (CG) faces are an important visual interface for human-computer interaction in social contexts. Here we investigated whether the human brain processes emotion and gaze similarly in real and carefully matched CG faces. Real faces evoked greater responses in the fusiform face area than CG faces, particularly for fearful expressions. Emotional (angry and fearful) facial expressions evoked similar activations in the amygdala in real and CG faces. Direct as compared with averted gaze elicited greater fMRI responses in the amygdala regardless of facial expression but only for real and not for CG faces. We observed an interaction effect between gaze and emotion (i.e., the shared signal effect) in the right posterior temporal sulcus and other regions, but not in the amygdala, and we found no evidence for different shared signal effects in real and CG faces. Taken together, the present findings highlight similarities (emotional processing in the amygdala) and differences (overall processing in the fusiform face area, gaze processing in the amygdala) in the neural processing of real and CG faces.

A human subcortical network underlying social avoidance revealed by risky economic choices

Social interactions have a major impact on well-being. While many individuals actively seek social situations, others avoid them, at great cost to their private and professional life. The neural mechanisms underlying individual differences in social approach or avoidance tendencies are poorly understood. Here we estimated people’s subjective value of engaging in a social situation. In each trial, more or less socially anxious participants chose between an interaction with a human partner providing social feedback and a monetary amount. With increasing social anxiety, the subjective value of social engagement decreased; amygdala BOLD response during decision-making and when experiencing social feedback increased; ventral striatum BOLD response to positive social feedback decreased; and connectivity between these regions during decision-making increased. Amygdala response was negatively related to the subjective value of social engagement. These findings suggest a relation between trait social anxiety/social avoidance and activity in a subcortical network during social decision-making.

Your relationships with the people around you – friends, family, colleagues – have a strong influence on your overall life happiness. Even so, many people struggle to engage with the people around them. Social interactions can be stressful and many people choose to avoid them, even at a cost.

Being able to measure these tendencies experimentally is a first useful step for assessing social avoidance without relying on people’s, often biased, recollections of their actions and behaviours. But how can a tendency to avoid social situations be quantified? And what can an experiment to measure this tendency reveal about the neural underpinnings of social avoidance?

Schultz et al. asked volunteers to play a social game. If they played, the volunteers had the chance to win three euros, but they could choose not to play and receive a fixed amount of money, which varied across trials between zero and three euros. This approach allowed Schultz et al. to quantify how much the volunteers valued playing the game. The game involved playing with other virtual human partners, who gave either positive or negative social feedback depending on the outcome of the game in the form of videos of facial expressions. In a non-social control experiment, a computer gave abstract feedback in the form of symbols.

Schultz et al. found that the value people placed on playing the social game varied with their level of social anxiety (established using a standard questionnaire). The more anxious people attributed less value to engaging in the game. Neuroimaging experiments revealed that the activity and connectivity between the amygdala and ventral striatum, two parts of the brain involved in processing emotions and reward-related stimuli, varied according to people’s levels of social anxiety.

Social interactions have a major impact on the quality of life of both healthy people and those with mental disorders. Developing new ways to measure and understand the differences in the brain linked to social traits could help to characterise certain conditions and document therapy progress. Methods to quantify social anxiety and avoidance are also in line with efforts to explore the neuroscience behind the full range of human behaviour.

Strategy Shift Toward Lower Spatial Frequencies in the Recognition of Dynamic Facial Expressions of Basic Emotions: When It Moves It Is Different

Facial expressions of emotion play a key role in social interactions. While in everyday life, their dynamic and transient nature calls for a fast processing of the visual information they contain, a majority of studies investigating the visual processes underlying their recognition have focused on their static display. The present study aimed to gain a better understanding of these processes while using more ecological dynamic facial expressions. In two experiments, we directly compared the spatial frequency (SF) tuning during the recognition of static and dynamic facial expressions. Experiment 1 revealed a shift toward lower SFs for dynamic expressions in comparison to static ones. Experiment 2 was designed to verify if changes in SF tuning curves were specific to the presence of emotional information in motion by comparing the SF tuning profiles for static, dynamic, and shuffled dynamic expressions. Results showed a similar shift toward lower SFs for shuffled expressions, suggesting that the difference found between dynamic and static expressions might not be linked to informative motion per se but to the presence of motion regardless its nature.

Interactions between Conscious and Subconscious Signals: Selective Attention under Feature-Based Competition Increases Neural Selectivity during Brain Adaptation

Efficient perception in natural environments depends on neural interactions between voluntary processes within cognitive control, such as attention, and those that are automatic and subconscious, such as brain adaptation to predictable input (also called repetition suppression). Although both attention and adaptation have been studied separately and there is considerable knowledge of the neurobiology involved in each of these processes, how attention interacts with adaptation remains equivocal. We examined how attention interacts with visual and auditory adaptation by measuring neuroimaging effects consistent with changes in either neural gain or selectivity. Male and female human participants were scanned with functional magnetic resonance imaging (fMRI) first while they discriminated repetition of morphed faces or voices and either directed their attention to stimulus identity or spatial location. Attention to face or voice identity, while ignoring stimulus location, solely increased the gain of respectively face- or voice-sensitive cortex. The results were strikingly different in an experiment when participants attended to voice identity versus stimulus loudness. In this case, attention to voice while ignoring sound loudness increased neural selectivity. The combined results show that how attention affects adaptation depends on the level of feature-based competition, reconciling prior conflicting observations. The findings are theoretically important and are discussed in relation to neurobiological interactions between attention and different types of predictive signals.SIGNIFICANCE STATEMENT Adaptation to repeated environmental events is ubiquitous in the animal brain, an automatic typically subconscious, predictive signal. Cognitive influences, such as by attention, powerfully affect sensory processing and can overcome brain adaptation. However, how neural interactions occur between adaptation and attention remains controversial. We conducted fMRI experiments regulating the focus of attention during adaptation to repeated stimuli with perceptually balanced stimulus expectancy. We observed an interaction between attention and adaptation consistent with increased neural selectivity, but only under conditions of feature-based competition, challenging the notion that attention interacts with brain adaptation by only affecting response gain. This demonstrates that attention retains its full complement of mechanistic influences on sensory cortex even as it interacts with more automatic or subconscious predictive processes.

Naturalistic Stimuli in Neuroscience: Critically Acclaimed

Cognitive neuroscience has traditionally focused on simple tasks, presented sparsely and using abstract stimuli. While this approach has yielded fundamental insights into functional specialisation in the brain, its ecological validity remains uncertain. Do these tasks capture how brains function 'in the wild', where stimuli are dynamic, multimodal, and crowded? Ecologically valid paradigms that approximate real life scenarios, using stimuli such as films, spoken narratives, music, and multiperson games emerged in response to these concerns over a decade ago. We critically appraise whether this approach has delivered on its promise to deliver new insights into brain function. We highlight the challenges, technological innovations, and clinical opportunities that are required should this field meet its full potential.

A functional dissociation of face-, body- and scene-selective brain areas based on their response to moving and static stimuli

The human brain contains areas that respond selectively to faces, bodies and scenes. Neuroimaging studies have shown that a subset of these areas preferentially respond more to moving than static stimuli, but the reasons for this functional dissociation remain unclear. In the present study, we simultaneously mapped the responses to motion in face-, body- and scene-selective areas in the right hemisphere using moving and static stimuli. Participants (N = 22) were scanned using functional magnetic resonance imaging (fMRI) while viewing videos containing bodies, faces, objects, scenes or scrambled objects, and static pictures from the beginning, middle and end of each video. Results demonstrated that lateral areas, including face-selective areas in the posterior and anterior superior temporal sulcus (STS), the extrastriate body area (EBA) and the occipital place area (OPA) responded more to moving than static stimuli. By contrast, there was no difference between the response to moving and static stimuli in ventral and medial category-selective areas, including the fusiform face area (FFA), occipital face area (OFA), amygdala, fusiform body area (FBA), retrosplenial complex (RSC) and parahippocampal place area (PPA). This functional dissociation between lateral and ventral/medial brain areas that respond selectively to different visual categories suggests that face-, body- and scene-selective networks may be functionally organized along a common dimension.

Widespread and lateralized social brain activity for processing dynamic facial expressions

Dynamic facial expressions of emotions constitute natural and powerful means of social communication in daily life. A number of previous neuroimaging studies have explored the neural mechanisms underlying the processing of dynamic facial expressions, and indicated the activation of certain social brain regions (e.g., the amygdala) during such tasks. However, the activated brain regions were inconsistent across studies, and their laterality was rarely evaluated. To investigate these issues, we measured brain activity using functional magnetic resonance imaging in a relatively large sample (n = 51) during the observation of dynamic facial expressions of anger and happiness and their corresponding dynamic mosaic images. The observation of dynamic facial expressions, compared with dynamic mosaics, elicited stronger activity in the bilateral posterior cortices, including the inferior occipital gyri, fusiform gyri, and superior temporal sulci. The dynamic facial expressions also activated bilateral limbic regions, including the amygdalae and ventromedial prefrontal cortices, more strongly versus mosaics. In the same manner, activation was found in the right inferior frontal gyrus (IFG) and left cerebellum. Laterality analyses comparing original and flipped images revealed right hemispheric dominance in the superior temporal sulcus and IFG and left hemispheric dominance in the cerebellum. These results indicated that the neural mechanisms underlying processing of dynamic facial expressions include widespread social brain regions associated with perceptual, emotional, and motor functions, and include a clearly lateralized (right cortical and left cerebellar) network like that involved in language processing.

Movies in the magnet: Naturalistic paradigms in developmental functional neuroimaging

The use of movie-watching as an acquisition state for functional connectivity (FC) MRI has recently enabled multiple groups to obtain rich data sets in younger children with both substantial sample sizes and scan durations. Using naturalistic paradigms such as movies has also provided analytic flexibility for these developmental studies that extends beyond conventional resting state approaches. This review highlights the advantages and challenges of using movies for developmental neuroimaging and explores some of the methodological issues involved in designing pediatric studies with movies. Emerging themes from movie-watching studies are discussed, including an emphasis on intersubject correlations, developmental changes in network interactions under complex naturalistic conditions, and dynamic age-related changes in both sensory and higher-order network FC even in narrow age ranges. Converging evidence suggests an enhanced ability to identify brain-behavior correlations in children when using movie-watching data relative to both resting state and conventional tasks. Future directions and cautionary notes highlight the potential and the limitations of using movies to study FC in pediatric populations.

Empathy in Facial Mimicry of Fear and Disgust: Simultaneous EMG-fMRI Recordings During Observation of Static and Dynamic Facial Expressions

Real-life faces are dynamic by nature, particularly when expressing emotion. Increasing evidence suggests that the perception of dynamic displays enhances facial mimicry and induces activation in widespread brain structures considered to be part of the mirror neuron system, a neuronal network linked to empathy. The present study is the first to investigate the relations among facial muscle responses, brain activity, and empathy traits while participants observed static and dynamic (videos) facial expressions of fear and disgust. During display presentation, blood-oxygen level-dependent (BOLD) signal as well as muscle reactions of the corrugator supercilii and levator labii were recorded simultaneously from 46 healthy individuals (21 females). It was shown that both fear and disgust faces caused activity in the corrugator supercilii muscle, while perception of disgust produced facial activity additionally in the levator labii muscle, supporting a specific pattern of facial mimicry for these emotions. Moreover, individuals with higher, compared to individuals with lower, empathy traits showed greater activity in the corrugator supercilii and levator labii muscles; however, these responses were not differentiable between static and dynamic mode. Conversely, neuroimaging data revealed motion and emotional-related brain structures in response to dynamic rather than static stimuli among high empathy individuals. In line with this, there was a correlation between electromyography (EMG) responses and brain activity suggesting that the Mirror Neuron System, the anterior insula and the amygdala might constitute the neural correlates of automatic facial mimicry for fear and disgust. These results revealed that the dynamic property of (emotional) stimuli facilitates the emotional-related processing of facial expressions, especially among whose with high trait empathy.

Selective eye fixations on diagnostic face regions of dynamic emotional expressions: KDEF-dyn database

Prior research using static facial stimuli (photographs) has identified diagnostic face regions (i.e., functional for recognition) of emotional expressions. In the current study, we aimed to determine attentional orienting, engagement, and time course of fixation on diagnostic regions. To this end, we assessed the eye movements of observers inspecting dynamic expressions that changed from a neutral to an emotional face. A new stimulus set (KDEF-dyn) was developed, which comprises 240 video-clips of 40 human models portraying six basic emotions (happy, sad, angry, fearful, disgusted, and surprised). For validation purposes, 72 observers categorized the expressions while gaze behavior was measured (probability of first fixation, entry time, gaze duration, and number of fixations). Specific visual scanpath profiles characterized each emotional expression: The eye region was looked at earlier and longer for angry and sad faces; the mouth region, for happy faces; and the nose/cheek region, for disgusted faces; the eye and the mouth regions attracted attention in a more balanced manner for surprise and fear. These profiles reflected enhanced selective attention to expression-specific diagnostic face regions. The KDEF-dyn stimuli and the validation data will be available to the scientific community as a useful tool for research on emotional facial expression processing.

Human Observers and Automated Assessment of Dynamic Emotional Facial Expressions: KDEF-dyn Database Validation

Most experimental studies of facial expression processing have used static stimuli (photographs), yet facial expressions in daily life are generally dynamic. In its original photographic format, the Karolinska Directed Emotional Faces (KDEF) has been frequently utilized. In the current study, we validate a dynamic version of this database, the KDEF-dyn. To this end, we applied animation between neutral and emotional expressions (happy, sad, angry, fearful, disgusted, and surprised; 1,033-ms unfolding) to 40 KDEF models, with morphing software. Ninety-six human observers categorized the expressions of the resulting 240 video-clip stimuli, and automated face analysis assessed the evidence for 6 expressions and 20 facial action units (AUs) at 31 intensities. Low-level image properties (luminance, signal-to-noise ratio, etc.) and other purely perceptual factors (e.g., size, unfolding speed) were controlled. Human recognition performance (accuracy, efficiency, and confusions) patterns were consistent with prior research using static and other dynamic expressions. Automated assessment of expressions and AUs was sensitive to intensity manipulations. Significant correlations emerged between human observers' categorization and automated classification. The KDEF-dyn database aims to provide a balance between experimental control and ecological validity for research on emotional facial expression processing. The stimuli and the validation data are available to the scientific community.

Tracking the recognition of static and dynamic facial expressions of emotion across the life span

The effective transmission and decoding of dynamic facial expressions of emotion is omnipresent and critical for adapted social interactions in everyday life. Thus, common intuition would suggest an advantage for dynamic facial expression recognition (FER) over the static snapshots routinely used in most experiments. However, although many studies reported an advantage in the recognition of dynamic over static expressions in clinical populations, results obtained from healthy participants are contrasted. To clarify this issue, we conducted a large cross-sectional study to investigate FER across the life span in order to determine if age is a critical factor to account for such discrepancies. More than 400 observers (age range 5-96) performed recognition tasks of the six basic expressions in static, dynamic, and shuffled (temporally randomized frames) conditions, normalized for the amount of energy sampled over time. We applied a Bayesian hierarchical step-linear model to capture the nonlinear relationship between age and FER for the different viewing conditions. Although replicating the typical accuracy profiles of FER, we determined the age at which peak efficiency was reached for each expression and found greater accuracy for most dynamic expressions across the life span. This advantage in the elderly population was driven by a significant decrease in performance for static images, which was twice as large as for the young adults. Our data posit the use of dynamic stimuli as being critical in the assessment of FER in the elderly population, inviting caution when drawing conclusions from the sole use of static face images to this aim.

Use and Usefulness of Dynamic Face Stimuli for Face Perception Studies-a Review of Behavioral Findings and Methodology

Faces that move contain rich information about facial form, such as facial features and their configuration, alongside the motion of those features. During social interactions, humans constantly decode and integrate these cues. To fully understand human face perception, it is important to investigate what information dynamic faces convey and how the human visual system extracts and processes information from this visual input. However, partly due to the difficulty of designing well-controlled dynamic face stimuli, many face perception studies still rely on static faces as stimuli. Here, we focus on evidence demonstrating the usefulness of dynamic faces as stimuli, and evaluate different types of dynamic face stimuli to study face perception. Studies based on dynamic face stimuli revealed a high sensitivity of the human visual system to natural facial motion and consistently reported dynamic advantages when static face information is insufficient for the task. These findings support the hypothesis that the human perceptual system integrates sensory cues for robust perception. In the present paper, we review the different types of dynamic face stimuli used in these studies, and assess their usefulness for several research questions. Natural videos of faces are ecological stimuli but provide limited control of facial form and motion. Point-light faces allow for good control of facial motion but are highly unnatural. Image-based morphing is a way to achieve control over facial motion while preserving the natural facial form. Synthetic facial animations allow separation of facial form and motion to study aspects such as identity-from-motion. While synthetic faces are less natural than videos of faces, recent advances in photo-realistic rendering may close this gap and provide naturalistic stimuli with full control over facial motion. We believe that many open questions, such as what dynamic advantages exist beyond emotion and identity recognition and which dynamic aspects drive these advantages, can be addressed adequately with different types of stimuli and will improve our understanding of face perception in more ecological settings.

The influence of dynamics and speech on understanding humanoid facial expressions

Human communication relies mostly on nonverbal signals expressed through body language. Facial expressions, in particular, convey emotional information that allows people involved in social interactions to mutually judge the emotional states and to adjust its behavior appropriately. First studies aimed at investigating the recognition of facial expressions were based on static stimuli. However, facial expressions are rarely static, especially in everyday social interactions. Therefore, it has been hypothesized that the dynamics inherent in a facial expression could be fundamental in understanding its meaning. In addition, it has been demonstrated that nonlinguistic and linguistic information can contribute to reinforce the meaning of a facial expression making it easier to be recognized. Nevertheless, few studies have been performed on realistic humanoid robots. This experimental work aimed at demonstrating the human-like expressive capability of a humanoid robot by examining whether the effect of motion and vocal content influenced the perception of its facial expressions. The first part of the experiment aimed at studying the recognition capability of two kinds of stimuli related to the six basic expressions (i.e. anger, disgust, fear, happiness, sadness, and surprise): static stimuli, that is, photographs, and dynamic stimuli, that is, video recordings. The second and third parts were focused on comparing the same six basic expressions performed by a virtual avatar and by a physical robot under three different conditions: (1) muted facial expressions, (2) facial expressions with nonlinguistic vocalizations, and (3) facial expressions with an emotionally neutral verbal sentence. The results show that static stimuli performed by a human being and by the robot were more ambiguous than the corresponding dynamic stimuli on which motion and vocalization were associated. This hypothesis has been also investigated with a 3-dimensional replica of the physical robot demonstrating that even in case of a virtual avatar, dynamic and vocalization improve the emotional conveying capability.

Neural correlates of the natural observation of an emotionally loaded video

Studies based on a paradigm of free or natural viewing have revealed characteristics that allow us to know how the brain processes stimuli within a natural environment. This method has been little used to study brain function. With a connectivity approach, we examine the processing of emotions using an exploratory method to analyze functional magnetic resonance imaging (fMRI) data. This research describes our approach to modeling stress paradigms suitable for neuroimaging environments. We showed a short film (4.54 minutes) with high negative emotional valence and high arousal content to 24 healthy male subjects (36.42 years old; SD = 12.14) during fMRI. Independent component analysis (ICA) was used to identify networks based on spatial statistical independence. Through this analysis we identified the sensorimotor system and its influence on the dorsal attention and default-mode networks, which in turn have reciprocal activity and modulate networks described as emotional.

Brain Responses to Dynamic Facial Expressions: A Normative Meta-Analysis

Identifying facial expressions is crucial for social interactions. Functional neuroimaging studies show that a set of brain areas, such as the fusiform gyrus and amygdala, become active when viewing emotional facial expressions. The majority of functional magnetic resonance imaging (fMRI) studies investigating face perception typically employ static images of faces. However, studies that use dynamic facial expressions (e.g., videos) are accumulating and suggest that a dynamic presentation may be more sensitive and ecologically valid for investigating faces. By using quantitative fMRI meta-analysis the present study examined concordance of brain regions associated with viewing dynamic facial expressions. We analyzed data from 216 participants that participated in 14 studies, which reported coordinates for 28 experiments. Our analysis revealed bilateral fusiform and middle temporal gyri, left amygdala, left declive of the cerebellum and the right inferior frontal gyrus. These regions are discussed in terms of their relation to models of face processing.

Variation of temporal order reveals deficits in categorisation of facial expressions in patients afflicted with depression

INTRODUCTION

It is well established that depressive disorders are associated with abnormalities in the processing of affective information. However, type of stimuli, perceptual complexity and cognitive demand are important factors in evaluating these findings. In particular, processing mechanisms of perceptual boundaries in ecologically valid face stimuli are largely unknown in depression.

METHODS

In the current study, intensity-ordered frame sequences provided a dynamic visualisation of happy or sad facial expressions fading from or to neutral expressions. Patients (n = 20) with major depression (MD) disorder and controls (n = 20) indicated their perceptual boundaries between neutral and emotional face depending on direction and emotion. The averaged time of the perceptual boundary entered a group × condition ANOVA and regression analysis.

RESULTS

MD group did not systematically shift perceptual boundaries in the dynamic emotional faces but yielded altered statistics in information processing. The Gaussian distribution of boundary judgements was disturbed in depression, increasing goodness-of-fit errors in disappearing emotions. Goodness-of-fit correlated with depression symptom score (Beck Depression Inventory-II (BDI-II)) in the MD group during the disappearing sad (r(18) = 46, p = 0.04) and happy (r(18) = 51, p = 0.02) conditions.

CONCLUSION

We evaluated detection of appearing and disappearing emotions in dynamic faces. A deviant distribution of categorisation responses emerged in the MD group, which was not emotion-specific. Such a perceptional uncertainty can impede individuals' functioning in interpersonal interaction.

Visual attention mechanisms in happiness versus trustworthiness processing of facial expressions

A happy facial expression makes a person look (more) trustworthy. Do perceptions of happiness and trustworthiness rely on the same face regions and visual attention processes? In an eye-tracking study, eye movements and fixations were recorded while participants judged the un/happiness or the un/trustworthiness of dynamic facial expressions in which the eyes and/or the mouth unfolded from neutral to happy or vice versa. A smiling mouth and happy eyes enhanced perceived happiness and trustworthiness similarly, with a greater contribution of the smile relative to the eyes. This comparable judgement output for happiness and trustworthiness was reached through shared as well as distinct attentional mechanisms: (a) entry times and (b) initial fixation thresholds for each face region were equivalent for both judgements, thereby revealing the same attentional orienting in happiness and trustworthiness processing. However, (c) greater and (d) longer fixation density for the mouth region in the happiness task, and for the eye region in the trustworthiness task, demonstrated different selective attentional engagement. Relatedly, (e) mean fixation duration across face regions was longer in the trustworthiness task, thus showing increased attentional intensity or processing effort.

Multivariate Pattern Classification of Facial Expressions Based on Large-Scale Functional Connectivity

It is an important question how human beings achieve efficient recognition of others’ facial expressions in cognitive neuroscience, and it has been identified that specific cortical regions show preferential activation to facial expressions in previous studies. However, the potential contributions of the connectivity patterns in the processing of facial expressions remained unclear. The present functional magnetic resonance imaging (fMRI) study explored whether facial expressions could be decoded from the functional connectivity (FC) patterns using multivariate pattern analysis combined with machine learning algorithms (fcMVPA). We employed a block design experiment and collected neural activities while participants viewed facial expressions of six basic emotions (anger, disgust, fear, joy, sadness, and surprise). Both static and dynamic expression stimuli were included in our study. A behavioral experiment after scanning confirmed the validity of the facial stimuli presented during the fMRI experiment with classification accuracies and emotional intensities. We obtained whole-brain FC patterns for each facial expression and found that both static and dynamic facial expressions could be successfully decoded from the FC patterns. Moreover, we identified the expression-discriminative networks for the static and dynamic facial expressions, which span beyond the conventional face-selective areas. Overall, these results reveal that large-scale FC patterns may also contain rich expression information to accurately decode facial expressions, suggesting a novel mechanism, which includes general interactions between distributed brain regions, and that contributes to the human facial expression recognition.

Task-dependent enhancement of facial expression and identity representations in human cortex

What cortical mechanisms allow humans to easily discern the expression or identity of a face? Subjects detected changes in expression or identity of a stream of dynamic faces while we measured BOLD responses from topographically and functionally defined areas throughout the visual hierarchy. Responses in dorsal areas increased during the expression task, whereas responses in ventral areas increased during the identity task, consistent with previous studies. Similar to ventral areas, early visual areas showed increased activity during the identity task. If visual responses are weighted by perceptual mechanisms according to their magnitude, these increased responses would lead to improved attentional selection of the task-appropriate facial aspect. Alternatively, increased responses could be a signature of a sensitivity enhancement mechanism that improves representations of the attended facial aspect. Consistent with the latter sensitivity enhancement mechanism, attending to expression led to enhanced decoding of exemplars of expression both in early visual and dorsal areas relative to attending identity. Similarly, decoding identity exemplars when attending to identity was improved in dorsal and ventral areas. We conclude that attending to expression or identity of dynamic faces is associated with increased selectivity in representations consistent with sensitivity enhancement.