Data-driven approaches in the investigation of social perception

Neural alignment during outgroup intervention predicts future change of affect towards outgroup

While social psychology studies have shown that paradoxical thinking intervention has a moderating effect on negative attitudes toward members from rival social groups (i.e. outgroup), the neural underpinnings of the intervention have not been studied. Here, we investigate this by examining neural alignment across individuals at different phases during the intervention regarding Covid-19 vaccine-supporters' attitudes against vaccine-opposers. We raise two questions: Whether neural alignment varies during the intervention, and whether it predicts a change in outgroup attitudes measured via a survey 2 days after the intervention and compared to baseline. We test the neural alignment using magnetoencephalography-recorded neural oscillations and multiset canonical correlation analysis. We find a build-up of neural alignment which emerges at the final phase of the paradoxical thinking intervention in the precuneus-a hub of mentalizing; there was no such effect in the control conditions. In parallel, we find a behavioral build-up of dissent to the interventional stimuli. These neural and behavioral patterns predict a prosocial future change in affect and actions toward the outgroup. Together, these findings reveal a new operational pattern of mentalizing on the outgroup, which can change the way individuals may feel and behave toward members of that outgroup.

A computational approach to investigating facial attractiveness factors using geometric morphometric analysis and deep learning

Numerous studies discuss the features that constitute facial attractiveness. In recent years, computational research has received attention because it can examine facial features without relying on prior research hypotheses. This approach uses many face stimuli and models the relationship between physical facial features and attractiveness using methods such as geometric morphometrics and deep learning. However, studies using each method have been conducted independently and have technical and data-related limitations. It is also difficult to identify the factors of actual attractiveness perception using only computational methods. In this study, we examined morphometric features important for attractiveness perception through geometric morphometrics and impression evaluation. Furthermore, we used deep learning to analyze important facial features comprehensively. The results showed that eye-related areas are essential in determining attractiveness and that different racial groups contribute differently to the impact of shape and skin information on attractiveness. The approach used in this study will contribute toward understanding facial attractiveness features that are universal and diverse, extending psychological findings and engineering applications.

Threat and Reward Imminence Processing in the Human Brain

In the human brain, aversive and appetitive processing have been studied with controlled stimuli in rather static settings. In addition, the extent to which aversive-related and appetitive-related processing engage distinct or overlapping circuits remains poorly understood. Here, we sought to investigate the dynamics of aversive and appetitive processing while male and female participants engaged in comparable trials involving threat avoidance or reward seeking. A central goal was to characterize the temporal evolution of responses during periods of threat or reward imminence. For example, in the aversive domain, we predicted that the bed nucleus of the stria terminalis (BST), but not the amygdala, would exhibit anticipatory responses given the role of the former in anxious apprehension. We also predicted that the periaqueductal gray (PAG) would exhibit threat-proximity responses based on its involvement in proximal-threat processes, and that the ventral striatum would exhibit threat-imminence responses given its role in threat escape in rodents. Overall, we uncovered imminence-related temporally increasing ("ramping") responses in multiple brain regions, including the BST, PAG, and ventral striatum, subcortically, and dorsal anterior insula and anterior midcingulate, cortically. Whereas the ventral striatum generated anticipatory responses in the proximity of reward as expected, it also exhibited threat-related imminence responses. In fact, across multiple brain regions, we observed a main effect of arousal. In other words, we uncovered extensive temporally evolving, imminence-related processing in both the aversive and appetitive domain, suggesting that distributed brain circuits are dynamically engaged during the processing of biologically relevant information regardless of valence, findings further supported by network analysis.SIGNIFICANCE STATEMENT In the human brain, aversive and appetitive processing have been studied with controlled stimuli in rather static settings. Here, we sought to investigate the dynamics of aversive/appetitive processing while participants engaged in trials involving threat avoidance or reward seeking. A central goal was to characterize the temporal evolution of responses during periods of threat or reward imminence. We uncovered imminence-related temporally increasing ("ramping") responses in multiple brain regions, including the bed nucleus of the stria terminalis, periaqueductal gray, and ventral striatum, subcortically, and dorsal anterior insula and anterior midcingulate, cortically. Overall, we uncovered extensive temporally evolving, imminence-related processing in both the aversive and appetitive domain, suggesting that distributed brain circuits are dynamically engaged during the processing of biologically relevant information regardless of valence.

Neural responses to biological motion distinguish autistic and schizotypal traits

Difficulties in social interactions characterize both autism and schizophrenia and are correlated in the neurotypical population. It is unknown whether this represents a shared etiology or superficial phenotypic overlap. Both conditions exhibit atypical neural activity in response to the perception of social stimuli and decreased neural synchronization between individuals. This study investigated if neural activity and neural synchronization associated with biological motion perception are differentially associated with autistic and schizotypal traits in the neurotypical population. Participants viewed naturalistic social interactions while hemodynamic brain activity was measured with fMRI, which was modeled against a continuous measure of the extent of biological motion. General linear model analysis revealed that biological motion perception was associated with neural activity across the action observation network. However, intersubject phase synchronization analysis revealed neural activity to be synchronized between individuals in occipital and parietal areas but desynchronized in temporal and frontal regions. Autistic traits were associated with decreased neural activity (precuneus and middle cingulate gyrus), and schizotypal traits were associated with decreased neural synchronization (middle and inferior frontal gyri). Biological motion perception elicits divergent patterns of neural activity and synchronization, which dissociate autistic and schizotypal traits in the general population, suggesting that they originate from different neural mechanisms.

Functional organization of social perception in the human brain

Humans rapidly extract diverse and complex information from ongoing social interactions, but the perceptual and neural organization of the different aspects of social perception remains unresolved. We showed short movie clips with rich social content to 97 healthy participants while their haemodynamic brain activity was measured with fMRI. The clips were annotated moment-to-moment for a large set of social features and 45 of the features were evaluated reliably between annotators. Cluster analysis of the social features revealed that 13 dimensions were sufficient for describing the social perceptual space. Three different analysis methods were used to map the social perceptual processes in the human brain. Regression analysis mapped regional neural response profiles for different social dimensions. Multivariate pattern analysis then established the spatial specificity of the responses and intersubject correlation analysis connected social perceptual processing with neural synchronization. The results revealed a gradient in the processing of social information in the brain. Posterior temporal and occipital regions were broadly tuned to most social dimensions and the classifier revealed that these responses showed spatial specificity for social dimensions; in contrast Heschl gyri and parietal areas were also broadly associated with different social signals, yet the spatial patterns of responses did not differentiate social dimensions. Frontal and subcortical regions responded only to a limited number of social dimensions and the spatial response patterns did not differentiate social dimension. Altogether these results highlight the distributed nature of social processing in the brain.

Testing, explaining, and exploring models of facial expressions of emotions

Models are the hallmark of mature scientific inquiry. In psychology, this maturity has been reached in a pervasive question-what models best represent facial expressions of emotion? Several hypotheses propose different combinations of facial movements [action units (AUs)] as best representing the six basic emotions and four conversational signals across cultures. We developed a new framework to formalize such hypotheses as predictive models, compare their ability to predict human emotion categorizations in Western and East Asian cultures, explain the causal role of individual AUs, and explore updated, culture-accented models that improve performance by reducing a prevalent Western bias. Our predictive models also provide a noise ceiling to inform the explanatory power and limitations of different factors (e.g., AUs and individual differences). Thus, our framework provides a new approach to test models of social signals, explain their predictive power, and explore their optimization, with direct implications for theory development.

Threat and reward imminence processing in the human brain

In the human brain, aversive and appetitive processing have been studied with controlled stimuli in rather static settings. In addition, the extent to which aversive- and appetitive-related processing engage distinct or overlapping circuits remains poorly understood. Here, we sought to investigate the dynamics of aversive and appetitive processing while male and female participants engaged in comparable trials involving threat-avoidance or reward-seeking. A central goal was to characterize the temporal evolution of responses during periods of threat or reward imminence . For example, in the aversive domain, we predicted that the bed nucleus of the stria terminalis (BST), but not the amygdala, would exhibit anticipatory responses given the role of the former in anxious apprehension. We also predicted that the periaqueductal gray (PAG) would exhibit threat-proximity responses based on its involvement in proximal-threat processes, and that the ventral striatum would exhibit threat-imminence responses given its role in threat escape in rodents. Overall, we uncovered imminence-related temporally increasing ("ramping") responses in multiple brain regions, including the BST, PAG, and ventral striatum, subcortically, and dorsal anterior insula and anterior midcingulate, cortically. Whereas the ventral striatum generated anticipatory responses in the proximity of reward as expected, it also exhibited threat-related imminence responses. In fact, across multiple brain regions, we observed a main effect of arousal. In other words, we uncovered extensive temporally-evolving, imminence-related processing in both the aversive and appetitive domain, suggesting that distributed brain circuits are dynamically engaged during the processing of biologically relevant information irrespective of valence, findings further supported by network analysis.

Significance Statement

In the human brain, aversive and appetitive processing have been studied with controlled stimuli in rather static settings. Here, we sought to investigate the dynamics of aversive/appetitive processing while participants engaged in trials involving threat-avoidance or reward-seeking. A central goal was to characterize the temporal evolution of responses during periods of threat or reward imminence . We uncovered imminence-related temporally increasing ("ramping") responses in multiple brain regions, including the bed nucleus of the stria terminalis, periaqueductal gray, and ventral striatum, subcortically, and dorsal anterior insula and anterior midcingulate, cortically. Overall, we uncovered extensive temporally-evolving, imminence-related processing in both the aversive and appetitive domain, suggesting that distributed brain circuits are dynamically engaged during the processing of biologically relevant information irrespective of valence.

The (un)learning of social functions and its significance for mental health

Social interactions are dynamic, context-dependent, and reciprocal events that influence prospective strategies and require constant practice and adaptation. This complexity of social interactions creates several research challenges. We propose a new framework encouraging future research to investigate not only individual differences in capacities relevant for social functioning and their underlying mechanisms, but also the flexibility to adapt or update one's social abilities. We suggest three key capacities relevant for social functioning: (1) social perception, (2) sharing emotions or empathizing, and (3) mentalizing. We elaborate on how adaptations in these capacities may be investigated on behavioral and neural levels. Research on these flexible adaptations of one's social behavior is needed to specify how humans actually "learn to be social". Learning to adapt implies plasticity of the relevant brain networks involved in the underlying social processes, indicating that social abilities are malleable for different contexts. To quantify such measures, researchers need to find ways to investigate learning through dynamic changes in adaptable social paradigms and examine several factors influencing social functioning within the three aformentioned social key capacities. This framework furthers insight concerning individual differences, provides a holistic approach to social functioning, and may improve interventions for ameliorating social abilities in patients.

Understanding trait impressions from faces

Impressions from faces are made remarkably quickly and they can underpin behaviour in a wide variety of social contexts. Over the last decade many studies have sought to trace the links between facial cues and social perception and behaviour. One such body of work has shown clear overlap between the fields of face perception and social stereotyping by demonstrating a role for conceptual stereotypes in impression formation from faces. We integrate these results involving conceptual influences on impressions with another substantial body of research in visual cognition which demonstrates that much of the variance in impressions can be predicted from perceptual, data-driven models using physical cues in face images. We relate this discussion to the phylogenetic, cultural, individual and developmental origins of facial impressions and define priority research questions for the field including investigating non-WEIRD cultures, tracking the developmental trajectory of impressions and determining the malleability of impression formation.

Functional near-infrared spectroscopy imaging of the prefrontal cortex during a naturalistic comedy movie

Naturalistic stimulation (i.e., movies and auditory narratives of some minutes' length) has been a powerful approach to bringing more real-life experiences into laboratory experiments. Data-driven, intersubject correlation (ISC) analysis permits examining to what extent activity in a specific brain region correlates across participants during exposure to a naturalistic stimulus, as well as testing whether neural activity correlates with behavioral measures. Notably, most of the previous research with naturalistic stimuli was conducted using functional fMRI (fMRI). Here, we tested whether a naturalistic approach and the ISC are feasible using functional near-infrared spectroscopy (fNIRS) - the imaging method particularly suited for populations of patients and children. Fifty-three healthy adult participants watched twice a 3-min segment of a Charlie Chaplin movie while we recorded the brain activity on the surface of their prefrontal cortex using fNIRS. In addition, an independent group of 18 participants used a continuous scoring procedure to rate the extent to which they felt that different parts of the movie fragment were funny. Our two findings were as follows. First, we found higher-than-zero ISC in fNIRS signals in the prefrontal cortex lobes, a result that was particularly high in the oxygenated channels during the first repetition of the movie. Second, we found a significant negative correlation between oxygenated brain signals and ratings of the movie's humorousness. In a series of control analyses we demonstrated that this latter correlation could not be explained by various non-humor-related movie sensory properties (e.g., auditory volume and image brightness). The key overall outcome of the present study is that fNIRS in combination with the naturalistic paradigms and the ISC might be a sensitive and powerful research method to explore cognitive processing. Our results also suggest a potential role of the prefrontal cortex in humor appreciation.

Functional connectivity dynamics as a function of the fluctuation of tension during film watching

To advance the understanding of the dynamic relationship between brain activities and emotional experiences, we examined the neural patterns of tension, a unique emotion that highly depends on how an event unfolds. Specifically, the present study explored the temporal relationship between functional connectivity patterns within and between different brain functional modules and the fluctuation in tension during film watching. Due to the highly contextualized and time-varying nature of tension, we expected that multiple neural networks would be involved in the dynamic tension experience. Using the neuroimaging data of 546 participants, we conducted a dynamic brain analysis to identify the intra- and inter-module functional connectivity patterns that are significantly correlated with the fluctuation of tension over time. The results showed that the inter-module connectivity of cingulo-opercular network, fronto-parietal network, and default mode network is involved in the dynamic experience of tension. These findings demonstrate a close relationship between brain functional connectivity patterns and emotional dynamics, which supports the importance of functional connectivity dynamics in understanding our cognitive and emotional processes.

Quantitative analysis of facial proportions and facial attractiveness among Asians and Caucasians

It has been proposed that the proportions of the human face are crucial for facial aesthetics. If this is the case, we should describe the relationship among proportions of face components quantitatively. This study aims to develop a mathematical model of facial proportions to provide a quantitative description of facial attractiveness. Furthermore, we expect that plastic surgeons can use models in clinical work to enhance communication efficiency between doctors and patients. Face alignment technique was used to analyse 5500 frontal faces with diverse properties (male/female, Asian/Caucasian, ages) to obtain the ratios among the nose length ($ {N}_{L} $), the nasal base width ($ N $), and the inner canthus width ($ {E}_{I} $). A mathematical model ($ {N}_{L}^{2} = a{E}_{I}\mathrm{*}{N}_{L}+b{E}_{I}\mathrm{*}N+cN\mathrm{*}{N}_{L} $) was developed to describe the relationship among these proportions. To validate the effectiveness of this approach, we simulated the post-operative photos using Adobe Photoshop. Our findings show that the ratio of nose length to nose width, the ratio of inner canthus width to nose length and the ratio of inner canthus to nose width play a significant role in determining facial attractiveness. These results provide a possible strategy to quantitatively describe the relationship among human face proportions.

Brain Basis of Psychopathy in Criminal Offenders and General Population

Psychopathy is characterized by persistent antisocial behavior, impaired empathy, and egotistical traits. These traits vary also in normally functioning individuals. Here, we tested whether such antisocial personalities are associated with similar structural and neural alterations as those observed in criminal psychopathy. Subjects were 100 non-convicted well-functioning individuals, 19 violent male offenders, and 19 matched controls. Subjects underwent T1-weighted magnetic resonance imaging and viewed movie clips with varying violent content during functional magnetic resonance imaging. Psychopathic traits were evaluated with Levenson Self-Report Psychopathy Scale (controls) and Psychopathy Checklist-Revised (offenders). Psychopathic offenders had lower gray matter density (GMD) in orbitofrontal cortex and anterior insula. In the community sample, affective psychopathy traits were associated with lower GMD in the same areas. Viewing violence increased brain activity in periaqueductal grey matter, thalamus, somatosensory, premotor, and temporal cortices. Psychopathic offenders had increased responses to violence in thalamus and orbitofrontal, insular, and cingulate cortices. In the community sample, impulsivity-related psychopathy traits were positively associated with violence-elicited responses in similar areas. We conclude that brain characteristics underlying psychopathic spectrum in violent psychopathy are related to those observed in well-functioning individuals with asocial personality features.

Naturalistic Stimuli in Affective Neuroimaging: A Review

Naturalistic stimuli such as movies, music, and spoken and written stories elicit strong emotions and allow brain imaging of emotions in close-to-real-life conditions. Emotions are multi-component phenomena: relevant stimuli lead to automatic changes in multiple functional components including perception, physiology, behavior, and conscious experiences. Brain activity during naturalistic stimuli reflects all these changes, suggesting that parsing emotion-related processing during such complex stimulation is not a straightforward task. Here, I review affective neuroimaging studies that have employed naturalistic stimuli to study emotional processing, focusing especially on experienced emotions. I argue that to investigate emotions with naturalistic stimuli, we need to define and extract emotion features from both the stimulus and the observer.

Listeners' perceptions of the certainty and honesty of a speaker are associated with a common prosodic signature

The success of human cooperation crucially depends on mechanisms enabling individuals to detect unreliability in their conspecifics. Yet, how such epistemic vigilance is achieved from naturalistic sensory inputs remains unclear. Here we show that listeners' perceptions of the certainty and honesty of other speakers from their speech are based on a common prosodic signature. Using a data-driven method, we separately decode the prosodic features driving listeners' perceptions of a speaker's certainty and honesty across pitch, duration and loudness. We find that these two kinds of judgments rely on a common prosodic signature that is perceived independently from individuals' conceptual knowledge and native language. Finally, we show that listeners extract this prosodic signature automatically, and that this impacts the way they memorize spoken words. These findings shed light on a unique auditory adaptation that enables human listeners to quickly detect and react to unreliability during linguistic interactions.

Unsupervised inference approach to facial attractiveness

The perception of facial attractiveness is a complex phenomenon which depends on how the observer perceives not only individual facial features, but also their mutual influence and interplay. In the machine learning community, this problem is typically tackled as a problem of regression of the subject-averaged rating assigned to natural faces. However, it has been conjectured that this approach does not capture the complexity of the phenomenon. It has recently been shown that different human subjects can navigate the face-space and "sculpt" their preferred modification of a reference facial portrait. Here we present an unsupervised inference study of the set of sculpted facial vectors in such experiments. We first infer minimal, interpretable and accurate probabilistic models (through Maximum Entropy and artificial neural networks) of the preferred facial variations, that encode the inter-subject variance. The application of such generative models to the supervised classification of the gender of the subject that sculpted the face reveals that it may be predicted with astonishingly high accuracy. We observe that the classification accuracy improves by increasing the order of the non-linear effective interaction. This suggests that the cognitive mechanisms related to facial discrimination in the brain do not involve the positions of single facial landmarks only, but mainly the mutual influence of couples, and even triplets and quadruplets of landmarks. Furthermore, the high prediction accuracy of the subjects' gender suggests that much relevant information regarding the subjects may influence (and be elicited from) their facial preference criteria, in agreement with the multiple motive theory of attractiveness proposed in previous works.

Decoding dynamic affective responses to naturalistic videos with shared neural patterns

This study explored the feasibility of using shared neural patterns from brief affective episodes (viewing affective pictures) to decode extended, dynamic affective sequences in a naturalistic experience (watching movie-trailers). Twenty-eight participants viewed pictures from the International Affective Picture System (IAPS) and, in a separate session, watched various movie-trailers. We first located voxels at bilateral occipital cortex (LOC) responsive to affective picture categories by GLM analysis, then performed between-subject hyperalignment on the LOC voxels based on their responses during movie-trailer watching. After hyperalignment, we trained between-subject machine learning classifiers on the affective pictures, and used the classifiers to decode affective states of an out-of-sample participant both during picture viewing and during movie-trailer watching. Within participants, neural classifiers identified valence and arousal categories of pictures, and tracked self-reported valence and arousal during video watching. In aggregate, neural classifiers produced valence and arousal time series that tracked the dynamic ratings of the movie-trailers obtained from a separate sample. Our findings provide further support for the possibility of using pre-trained neural representations to decode dynamic affective responses during a naturalistic experience.

Imaging Real-Time Tactile Interaction With Two-Person Dual-Coil fMRI

Studies of brain mechanisms supporting social interaction are demanding because real interaction only occurs when persons are in contact. Instead, most brain imaging studies scan subjects individually. Here we present a proof-of-concept demonstration of two-person blood oxygenation dependent (BOLD) imaging of brain activity from two individuals interacting inside the bore of a single MRI scanner. We developed a custom 16-channel (8 + 8 channels) two-helmet coil with two separate receiver-coil pairs providing whole-brain coverage, while bringing participants into a shared physical space and realistic face-to-face contact. Ten subject pairs were scanned with the setup. During the experiment, subjects took turns in tapping each other's lip versus observing and feeling the taps timed by auditory instructions. Networks of sensorimotor brain areas were engaged alternatingly in the subjects during executing motor actions as well as observing and feeling them; these responses were clearly distinguishable from the auditory responses occurring similarly in both participants. Even though the signal-to-noise ratio of our coil system was compromised compared with standard 32-channel head coils, our results show that the two-person fMRI scanning is feasible for studying the brain basis of social interaction.

Dynamic interactive theory as a domain-general account of social perception

The perception of social categories, emotions, and personality traits from others' faces each have been studied extensively but in relative isolation. We synthesize emerging findings suggesting that, in each of these domains of social perception, both a variety of bottom-up facial features and top-down social cognitive processes play a part in driving initial perceptions. Among such top-down processes, social-conceptual knowledge in particular can have a fundamental structuring role in how we perceive others' faces. Extending the Dynamic Interactive framework (Freeman & Ambady, 2011), we outline a perspective whereby the perception of social categories, emotions, and traits from faces can all be conceived as emerging from an integrated system relying on domain-general cognitive properties. Such an account of social perception would envision perceptions to be a rapid, but gradual, process of negotiation between the variety of visual cues inherent to a person and the social cognitive knowledge an individual perceiver brings to the perceptual process. We describe growing evidence in support of this perspective as well as its theoretical implications for social psychology.

Measuring shared responses across subjects using intersubject correlation

Our capacity to jointly represent information about the world underpins our social experience. By leveraging one individual's brain activity to model another's, we can measure shared information across brains-even in dynamic, naturalistic scenarios where an explicit response model may be unobtainable. Introducing experimental manipulations allows us to measure, for example, shared responses between speakers and listeners or between perception and recall. In this tutorial, we develop the logic of intersubject correlation (ISC) analysis and discuss the family of neuroscientific questions that stem from this approach. We also extend this logic to spatially distributed response patterns and functional network estimation. We provide a thorough and accessible treatment of methodological considerations specific to ISC analysis and outline best practices.

Subjectivity and complexity of facial attractiveness

The origin and meaning of facial beauty represent a longstanding puzzle. Despite the profuse literature devoted to facial attractiveness, its very nature, its determinants and the nature of inter-person differences remain controversial issues. Here we tackle such questions proposing a novel experimental approach in which human subjects, instead of rating natural faces, are allowed to efficiently explore the face-space and "sculpt" their favorite variation of a reference facial image. The results reveal that different subjects prefer distinguishable regions of the face-space, highlighting the essential subjectivity of the phenomenon. The different sculpted facial vectors exhibit strong correlations among pairs of facial distances, characterising the underlying universality and complexity of the cognitive processes, and the relative relevance and robustness of the different facial distances.

Data-driven mathematical model of East-Asian facial attractiveness: the relative contributions of shape and reflectance to attractiveness judgements

Facial attractiveness is judged through a combination of multiple cues including morphology (facial shape) and skin properties (facial reflectance). While several studies have examined the way in which people in Western cultures judge facial attractiveness, there have been fewer investigations into non-Western attitudes. This is because stimuli that quantitatively vary the attractiveness of non-Western faces are rare. In the present study, we built a model of the attractiveness of East-Asian faces, judged by East-Asian observers. Therefore, 400 computer-generated East-Asian faces were created and attractiveness rating scores were collected from Japanese observers. Data-driven mathematical calculations were used to identify quantitative links between facial attractiveness and shape and reflectance properties, with no prior hypothesis. Results indicate that faces with larger eyes, smaller noses and brighter skin are judged as more attractive, regardless of the sex of the faces, possibly reflecting a general preference for femininity. Shape is shown to be a strong determinant of attractiveness for both male and female faces, while reflectance properties are less important in judging male facial attractiveness. Our model provides a tool to effectively produce East-Asian face stimuli that quantitatively varies attractiveness and can be used to elucidate visual processes related to attractiveness judgements.

CLEESE: An open-source audio-transformation toolbox for data-driven experiments in speech and music cognition

Over the past few years, the field of visual social cognition and face processing has been dramatically impacted by a series of data-driven studies employing computer-graphics tools to synthesize arbitrary meaningful facial expressions. In the auditory modality, reverse correlation is traditionally used to characterize sensory processing at the level of spectral or spectro-temporal stimulus properties, but not higher-level cognitive processing of e.g. words, sentences or music, by lack of tools able to manipulate the stimulus dimensions that are relevant for these processes. Here, we present an open-source audio-transformation toolbox, called CLEESE, able to systematically randomize the prosody/melody of existing speech and music recordings. CLEESE works by cutting recordings in small successive time segments (e.g. every successive 100 milliseconds in a spoken utterance), and applying a random parametric transformation of each segment’s pitch, duration or amplitude, using a new Python-language implementation of the phase-vocoder digital audio technique. We present here two applications of the tool to generate stimuli for studying intonation processing of interrogative vs declarative speech, and rhythm processing of sung melodies.

Moral and Affective Film Set (MAAFS): A normed moral video database

Moral psychology has relied nearly exclusively on text stimuli in the development and testing of theories. However, text stimuli lack the rich variety of morally-relevant social and contextual cues available in everyday interactions. A consequence of this pervasive ecological invalidity may be that moral psychological theories are mischaracterized by an overreliance on cue-impoverished moral stimuli. We address this limitation by developing a cue-rich Moral and Affective Film Set (MAAFS). We crowd-sourced videos of moral behaviours, using previously validated text stimuli and definitions of moral foundations as a guide for content. Crowd-sourced clips were rated by 322 American and 253 Australian participants on a range of moral and affective dimensions, including wrongness, moral foundation relevance, punishment, arousal, discrete emotion-relevance, clarity, previous exposure, and how weird/uncommon the moral acts were. The final stimulus set contained sixty nine moral videos. Ratings confirmed that the videos are reliably rated as morally wrong and feature a variety of moral concerns. The validation process revealed features that make the MAAFS useful for future research: (1) the MAAFS includes a range of videos that depict everyday transgressions, (2) certain videos evoke negative emotions at an intensity comparable to mood induction films, (3) the videos are largely novel: participants had never seen more than 90% of the videos. We anticipate the MAAFS will be a particularly valuable tool for researchers in moral psychology who seek to study morality in scenarios that approximate real-life. However, the MAAFS may be valuable for other fields of psychology, for example, affective scientists may use these videos as a mood induction procedure. The complete stimulus set, links to videos, and normative statistics can be accessed at osf.io/8w3en.

Aberrant Cortical Integration in First-Episode Psychosis During Natural Audiovisual Processing

BACKGROUND

Functional magnetic resonance imaging studies of psychotic disorders have reported both hypoactivity and hyperactivity in numerous brain regions. In line with the dysconnection hypothesis, these regions include cortical integrative hub regions. However, most earlier studies focused on a single cognitive function at a time, assessed by delivering artificial stimuli to patients with chronic psychosis. Thus, it remains unresolved whether these findings are present already in early psychosis and whether they translate to real-life-like conditions that require multisensory processing and integration.

METHODS

Scenes from the movie Alice in Wonderland (2010) were shown to 51 patients with first-episode psychosis (16 women) and 32 community-based control subjects (17 women) during 3T functional magnetic resonance imaging. We compared intersubject correlation, a measure of similarity of brain signal time courses in each voxel, between the groups. We also quantified the hubness as the number of connections each region has.

RESULTS

Intersubject correlation was significantly lower in patients with first-episode psychosis than in control subjects in the medial and lateral prefrontal, cingulate, precuneal, and parietotemporal regions, including the default mode network. Regional magnitude of between-group difference in intersubject correlation was associated with the hubness.

CONCLUSIONS

Our findings provide novel evidence for the dysconnection hypothesis by showing that during complex real-life-like stimulation, the most prominent functional alterations in psychotic disorders relate to integrative brain functions. Presence of such abnormalities in first-episode psychosis rules out long-term effects of illness or medication. These methods can be used in further studies to map widespread hub alterations in a single functional magnetic resonance imaging session and link them to potential downstream and upstream pathways.

Impact of face proportions on face attractiveness

BACKGROUND

Proportions of face components appear to play a role in facial attractiveness.

AIMS

The aim of the study was to establish the best proportions of face components in relation to whole face shape for facial attractiveness.

METHODS

Only one face component (eye, nose, or lips) of a model in a series of photographs was altered using a computer program. Alterations consisted of size reduction or augmentation by 5% or 10%. Each photograph depicted a particular face component altered to either 90%, 95%, 100%, 105%, or 110% of its original size. Collages of photographs were shown to 167 individuals (male and female) for a fixed period of 7 seconds. Their task was to indicate the most attractive photograph of a model in a presented collage.

RESULTS

In total, 48.1% of individuals preferred enhanced eyes both in males and females. We found that the preferred mean eye size in women was statistically significantly higher than that in men. In total, 64.8% of respondents preferred reduced nose proportions in women (27.5% found a reduction to 90% of the original size more attractive while 37.3% preferred a reduction to 95%). It was demonstrated that the preferred mean nose size was statistically significantly lower in females in comparison with males. Respondents expressed a greater preference for nose reduction in women in comparison with men. 38.4% of respondents (in regard to both male and female mouth) preferred reduced mouth. 40.7% of respondents preferred reduced mouth in the female model.

CONCLUSIONS

Our work delivers statistically significant evidence that facial attractiveness increases together with the enlargement of the uncovered eye surface as well as the reduction in nose and lip size. Data were obtained using modern collective intelligence methods of validation. Written consent was obtained from all study participants.

Decoding the neural representation of self and person knowledge with multivariate pattern analysis and data-driven approaches

Multivariate pattern analysis and data-driven approaches to understand how the human brain encodes sensory information and higher level conceptual knowledge have become increasingly dominant in visual and cognitive neuroscience; however, it is only in recent years that these methods have been applied to the domain of social information processing. This review examines recent research in the field of social cognitive neuroscience focusing on how multivariate pattern analysis (e.g., pattern classification, representational similarity analysis) and data-driven methods (e.g., reverse correlation, intersubject correlation) have been used to decode and characterize high-level information about the self, other persons, and social groups. We begin with a review of what is known about how self-referential processing and person perception are represented in the medial prefrontal cortex based on conventional activation-based neuroimaging approaches. This is followed by a nontechnical overview of current multivariate pattern-based and data-driven neuroimaging methods designed to characterize and/or decode neural representations. The remainder of the review focuses on examining how these methods have been applied to the topic of self, person perception, and the perception of social groups. In this review, we highlight recent trends (e.g., analysis of social networks, decoding race and social groups, and the use of naturalistic stimuli) and discuss several theoretical challenges that arise from the application of these new methods to the question of how the brain represents knowledge about the self and others. This article is categorized under: Neuroscience > Cognition.

A dual-fMRI investigation of the iterated Ultimatum Game reveals that reciprocal behaviour is associated with neural alignment

Dyadic interactions often involve a dynamic process of mutual reciprocity; to steer a series of exchanges towards a desired outcome, both interactants must adapt their own behaviour according to that of their interaction partner. Understanding the brain processes behind such bidirectional reciprocity is therefore central to social neuroscience, but this requires measurement of both individuals' brains during real-world exchanges. We achieved this by performing functional magnetic resonance imaging (fMRI) on pairs of male individuals simultaneously while they interacted in a modified iterated Ultimatum Game (iUG). In this modification, both players could express their intent and maximise their own monetary gain by reciprocating their partner's behaviour - they could promote generosity through cooperation and/or discourage unfair play with retaliation. By developing a novel model of reciprocity adapted from behavioural economics, we then show that each player's choices can be predicted accurately by estimating expected utility (EU) not only in terms of immediate payoff, but also as a reaction to their opponent's prior behaviour. Finally, for the first time we reveal that brain signals implicated in social decision making are modulated by these estimates of EU, and become correlated more strongly between interacting players who reciprocate one another.

Opioid system and human emotions

Emotions are states of vigilant readiness that guide human and animal behaviour during survival-salient situations. Categorical models of emotions posit neurally and physiologically distinct basic human emotions (anger, fear, disgust, happiness, sadness and surprise) that govern different survival functions. Opioid receptors are expressed abundantly in the mammalian emotion circuit, and the opioid system modulates a variety of functions related to arousal and motivation. Yet, its specific contribution to different basic emotions has remained poorly understood. Here, we review how the endogenous opioid system and particularly the μ receptor contribute to emotional processing in humans. Activation of the endogenous opioid system is consistently associated with both pleasant and unpleasant emotions. In general, exogenous opioid agonists facilitate approach-oriented emotions (anger, pleasure) and inhibit avoidance-oriented emotions (fear, sadness). Opioids also modulate social bonding and affiliative behaviour, and prolonged opioid abuse may render both social bonding and emotion recognition circuits dysfunctional. However, there is no clear evidence that the opioid system is able to affect the emotions associated with surprise and disgust. Taken together, the opioid systems contribute to a wide array of positive and negative emotions through their general ability to modulate the approach versus avoidance motivation associated with specific emotions. Because of the protective effects of opioid system-mediated prosociality and positive mood, the opioid system may constitute an important factor contributing to psychological and psychosomatic resilience.

LINKED ARTICLES

This article is part of a themed section on Emerging Areas of Opioid Pharmacology. To view the other articles in this section visit http://onlinelibrary.wiley.com/doi/10.1111/bph.v175.14/issuetoc.

Anger Modulates Influence Hierarchies Within and Between Emotional Reactivity and Regulation Networks

Emotion regulation is hypothesized to be mediated by the interactions between emotional reactivity and regulation networks during the dynamic unfolding of the emotional episode. Yet, it remains unclear how to delineate the effective relationships between these networks. In this study, we examined the aforementioned networks’ information flow hierarchy during viewing of an anger provoking movie excerpt. Anger regulation is particularly essential for averting individuals from aggression and violence, thus improving prosocial behavior. Using subjective ratings of anger intensity we differentiated between low and high anger periods of the film. We then applied the Dependency Network Analysis (D_EPNA), a newly developed graph theory method to quantify networks’ node importance during the two anger periods. The D_EPNA analysis revealed that the impact of the ventromedial prefrontal cortex (vmPFC) was higher in the high anger condition, particularly within the regulation network and on the connections between the reactivity and regulation networks. We further showed that higher levels of vmPFC impact on the regulation network were associated with lower subjective anger intensity during the high-anger cinematic period, and lower trait anger levels. Supporting and replicating previous findings, these results emphasize the previously acknowledged central role of vmPFC in modulating negative affect. We further show that the impact of the vmPFC relies on its correlational influence on the connectivity between reactivity and regulation networks. More importantly, the hierarchy network analysis revealed a link between connectivity patterns of the vmPFC and individual differences in anger reactivity and trait, suggesting its potential therapeutic role.

Cracking the social code of speech prosody using reverse correlation

In speech, social evaluations of a speaker’s dominance or trustworthiness are conveyed by distinguishing, but little-understood, pitch variations. This work describes how to combine state-of-the-art vocal pitch transformations with the psychophysical technique of reverse correlation and uses this methodology to uncover the prosodic prototypes that govern such social judgments in speech. This finding is of great significance, because the exact shape of these prototypes, and how they vary with sex, age, and culture, is virtually unknown, and because prototypes derived with the method can then be reapplied to arbitrary spoken utterances, thus providing a principled way to modulate personality impressions in speech.

Human listeners excel at forming high-level social representations about each other, even from the briefest of utterances. In particular, pitch is widely recognized as the auditory dimension that conveys most of the information about a speaker’s traits, emotional states, and attitudes. While past research has primarily looked at the influence of mean pitch, almost nothing is known about how intonation patterns, i.e., finely tuned pitch trajectories around the mean, may determine social judgments in speech. Here, we introduce an experimental paradigm that combines state-of-the-art voice transformation algorithms with psychophysical reverse correlation and show that two of the most important dimensions of social judgments, a speaker’s perceived dominance and trustworthiness, are driven by robust and distinguishing pitch trajectories in short utterances like the word “Hello,” which remained remarkably stable whether male or female listeners judged male or female speakers. These findings reveal a unique communicative adaptation that enables listeners to infer social traits regardless of speakers’ physical characteristics, such as sex and mean pitch. By characterizing how any given individual’s mental representations may differ from this generic code, the method introduced here opens avenues to explore dysprosody and social-cognitive deficits in disorders like autism spectrum and schizophrenia. In addition, once derived experimentally, these prototypes can be applied to novel utterances, thus providing a principled way to modulate personality impressions in arbitrary speech signals.

Development of the social brain from age three to twelve years

Human adults recruit distinct networks of brain regions to think about the bodies and minds of others. This study characterizes the development of these networks, and tests for relationships between neural development and behavioral changes in reasoning about others’ minds (‘theory of mind’, ToM). A large sample of children (n = 122, 3–12 years), and adults (n = 33), watched a short movie while undergoing fMRI. The movie highlights the characters’ bodily sensations (often pain) and mental states (beliefs, desires, emotions), and is a feasible experiment for young children. Here we report three main findings: (1) ToM and pain networks are functionally distinct by age 3 years, (2) functional specialization increases throughout childhood, and (3) functional maturity of each network is related to increasingly anti-correlated responses between the networks. Furthermore, the most studied milestone in ToM development, passing explicit false-belief tasks, does not correspond to discontinuities in the development of the social brain.

Though adults’ brains process the internal states of others’ bodies versus others’ minds in distinct brain regions, it is not clear when this functional dissociation emerges. Here, authors study 3–12 year olds and show that these networks are distinct by age 3 and become even more distinct with age.

Social neuroscience: undoing the schism between neurology and psychiatry

Multiple disorders once jointly conceived as "nervous diseases" became segregated by the distinct institutional traditions forged in neurology and psychiatry. As a result, each field specialized in the study and treatment of a subset of such conditions. Here we propose new avenues for interdisciplinary interaction through a triangulation of both fields with social neuroscience. To this end, we review evidence from five relevant domains (facial emotion recognition, empathy, theory of mind, moral cognition, and social context assessment), highlighting their common disturbances across neurological and psychiatric conditions and discussing their multiple pathophysiological mechanisms. Our proposal is anchored in multidimensional evidence, including behavioral, neurocognitive, and genetic findings. From a clinical perspective, this work paves the way for dimensional and transdiagnostic approaches, new pharmacological treatments, and educational innovations rooted in a combined neuropsychiatric training. Research-wise, it fosters new models of the social brain and a novel platform to explore the interplay of cognitive and social functions. Finally, we identify new challenges for this synergistic framework.

The Socio-Moral Image Database (SMID): A novel stimulus set for the study of social, moral and affective processes

A major obstacle for the design of rigorous, reproducible studies in moral psychology is the lack of suitable stimulus sets. Here, we present the Socio-Moral Image Database (SMID), the largest standardized moral stimulus set assembled to date, containing 2,941 freely available photographic images, representing a wide range of morally (and affectively) positive, negative and neutral content. The SMID was validated with over 820,525 individual judgments from 2,716 participants, with normative ratings currently available for all images on affective valence and arousal, moral wrongness, and relevance to each of the five moral values posited by Moral Foundations Theory. We present a thorough analysis of the SMID regarding (1) inter-rater consensus, (2) rating precision, and (3) breadth and variability of moral content. Additionally, we provide recommendations for use aimed at efficient study design and reproducibility, and outline planned extensions to the database. We anticipate that the SMID will serve as a useful resource for psychological, neuroscientific and computational (e.g., natural language processing or computer vision) investigations of social, moral and affective processes. The SMID images, along with associated normative data and additional resources are available at https://osf.io/2rqad/.

Facial First Impressions Across Culture: Data-Driven Modeling of Chinese and British Perceivers' Unconstrained Facial Impressions

People form first impressions from facial appearance rapidly, and these impressions can have considerable social and economic consequences. Three dimensions can explain Western perceivers' impressions of Caucasian faces: approachability, youthful-attractiveness, and dominance. Impressions along these dimensions are theorized to be based on adaptive cues to threat detection or sexual selection, making it likely that they are universal. We tested whether the same dimensions of facial impressions emerge across culture by building data-driven models of first impressions of Asian and Caucasian faces derived from Chinese and British perceivers' unconstrained judgments. We then cross-validated the dimensions with computer-generated average images. We found strong evidence for common approachability and youthful-attractiveness dimensions across perceiver and face race, with some evidence of a third dimension akin to capability. The models explained ~75% of the variance in facial impressions. In general, the findings demonstrate substantial cross-cultural agreement in facial impressions, especially on the most salient dimensions.

Interpreting the dimensions of neural feature representations revealed by dimensionality reduction

Recent progress in understanding the structure of neural representations in the cerebral cortex has centred around the application of multivariate classification analyses to measurements of brain activity. These analyses have proved a sensitive test of whether given brain regions provide information about specific perceptual or cognitive processes. An exciting extension of this approach is to infer the structure of this information, thereby drawing conclusions about the underlying neural representational space. These approaches rely on exploratory data-driven dimensionality reduction to extract the natural dimensions of neural spaces, including natural visual object and scene representations, semantic and conceptual knowledge, and working memory. However, the efficacy of these exploratory methods is unknown, because they have only been applied to representations in brain areas for which we have little or no secondary knowledge. One of the best-understood areas of the cerebral cortex is area MT of primate visual cortex, which is known to be important in motion analysis. To assess the effectiveness of dimensionality reduction for recovering neural representational space we applied several dimensionality reduction methods to multielectrode measurements of spiking activity obtained from area MT of marmoset monkeys, made while systematically varying the motion direction and speed of moving stimuli. Despite robust tuning at individual electrodes, and high classifier performance, dimensionality reduction rarely revealed dimensions for direction and speed. We use this example to illustrate important limitations of these analyses, and suggest a framework for how to best apply such methods to data where the structure of the neural representation is unknown.

From faces to prosocial behavior: cues, tools, and mechanisms

In this review we ask how looking at people's faces can influence prosocial behaviors towards them. Components of this process have often been studied by disparate literatures: one focused on perception and judgment of faces, using both psychological and neuroscience approaches; and a second focused on actual social behaviors, as studied in behavioral economics and decision science. Bridging these disciplines requires a more mechanistic account of how processing of particular face attributes or features influences social judgments and behaviors. Here we review these two lines of research, and suggest that combining some of their methodological tools can provide the bridging mechanistic explanations.

Attending to and neglecting people: bridging neuroscience, psychology and sociology

Human behaviour is context-dependent-based on predictions and influenced by the environment and other people. We live in a dynamic world where both the social stimuli and their context are constantly changing. Similar dynamic, natural stimuli should, in the future, be increasingly used to study social brain functions, with parallel development of appropriate signal-analysis methods. Understanding dynamic neural processes also requires accurate time-sensitive characterization of the behaviour. To go beyond the traditional stimulus-response approaches, brain activity should be recorded simultaneously from two interacting subjects to reveal why human social interaction is critically different from just reacting to each other. This theme issue on Attending to and neglecting people contains original work and review papers on person perception and social interaction. The articles cover research from neuroscience, psychology, robotics, animal interaction research and microsociology. Some of the papers are co-authored by scientists who presented their own, independent views in the recent Attention and Performance XXVI conference but were brave enough to join forces with a colleague having a different background and views. In the future, information needs to converge across disciplines to provide us a more holistic view of human behaviour, its interactive nature, as well as the temporal dynamics of our social world.

Brain responses to facial attractiveness induced by facial proportions: evidence from an fMRI study

Brain responses to facial attractiveness induced by facial proportions are investigated by using functional magnetic resonance imaging (fMRI), in 41 young adults (22 males and 19 females). The subjects underwent fMRI while they were presented with computer-generated, yet realistic face images, which had varying facial proportions, but the same neutral facial expression, baldhead and skin tone, as stimuli. Statistical parametric mapping with parametric modulation was used to explore the brain regions with the response modulated by facial attractiveness ratings (ARs). The results showed significant linear effects of the ARs in the caudate nucleus and the orbitofrontal cortex for all of the subjects, and a non-linear response profile in the right amygdala for only the male subjects. Furthermore, canonical correlation analysis was used to learn the most relevant facial ratios that were best correlated with facial attractiveness. A regression model on the fMRI-derived facial ratio components demonstrated a strong linear relationship between the visually assessed mean ARs and the predictive ARs. Overall, this study provided, for the first time, direct neurophysiologic evidence of the effects of facial ratios on facial attractiveness and suggested that there are notable gender differences in perceiving facial attractiveness as induced by facial proportions.