Categorical perception of vocal emotion expressions

Effect of Age and Gender on Categorical Perception of Vocal Emotion Under Tonal Language Background

PURPOSE

Categorical perception (CP) manifests in various aspects of human cognition. While there is mounting evidence for CP in facial emotions, CP in vocal emotions remains understudied. The current study attempted to test whether individuals with a tonal language background perceive vocal emotions categorically and to examine how factors such as gender and age influence the plasticity of these perceptual categories.

METHOD

This study examined the identification and discrimination performance of 24 Mandarin-speaking children (14 boys and 10 girls) and 32 adults (16 males and 16 females) when they were presented with three vocal emotion continua. Speech stimuli in each continuum consisted of 11 resynthesized Mandarin disyllabic words.

RESULTS

CP phenomena were detected when Mandarin participants perceived vocal emotions. We further found the modulating effect of age and gender in vocal emotion categorization.

CONCLUSIONS

Our results demonstrate for the first time that a categorical strategy is used by Mandarin speakers when perceiving vocal emotions. Furthermore, our findings reveal that the categorization ability of vocal emotions follows a prolonged course of development and the maturation patterns differ across genders. This study opens a promising line of research for investigating how sensory features are mapped to higher order perception and provides implications for our understanding of clinical populations characterized by altered emotional processing.

SUPPLEMENTAL MATERIAL

https://doi.org/10.23641/asha.27204057.

Effects of age and hearing loss on speech emotion discrimination

Better communication with older people requires not only improving speech intelligibility but also understanding how well emotions can be conveyed and the effect of age and hearing loss (HL) on emotion perception. In this paper, emotion discrimination experiments were conducted using a vocal morphing method and an HL simulator in young normal hearing (YNH) and older participants. Speech sounds were morphed to represent intermediate emotions between all combinations of happiness, sadness, and anger. Discrimination performance was compared when the YNH listened to normal sounds, when the same YNH listened to HL simulated sounds, and when older people listened to the same normal sounds. The results showed that there was no significant difference between discrimination with and without HL simulation, suggesting that peripheral HL may not affect emotion perception. The discrimination performance of the older participants was significantly worse only for the anger-happiness pair than for the other emotion pairs and for the YNH. It was also found that the difficulty increases with age, not just with hearing level.

The Sound of Emotional Prosody: Nearly 3 Decades of Research and Future Directions

Emotional voices attract considerable attention. A search on any browser using "emotional prosody" as a key phrase leads to more than a million entries. Such interest is evident in the scientific literature as well; readers are reminded in the introductory paragraphs of countless articles of the great importance of prosody and that listeners easily infer the emotional state of speakers through acoustic information. However, despite decades of research on this topic and important achievements, the mapping between acoustics and emotional states is still unclear. In this article, we chart the rich literature on emotional prosody for both newcomers to the field and researchers seeking updates. We also summarize problems revealed by a sample of the literature of the last decades and propose concrete research directions for addressing them, ultimately to satisfy the need for more mechanistic knowledge of emotional prosody.

Different stages of emotional prosody processing in healthy ageing–evidence from behavioural responses, ERPs, tDCS, and tRNS

Past research suggests that the ability to recognise the emotional intent of a speaker decreases as a function of age. Yet, few studies have looked at the underlying cause for this effect in a systematic way. This paper builds on the view that emotional prosody perception is a multi-stage process and explores which step of the recognition processing line is impaired in healthy ageing using time-sensitive event-related brain potentials (ERPs). Results suggest that early processes linked to salience detection as reflected in the P200 component and initial build-up of emotional representation as linked to a subsequent negative ERP component are largely unaffected in healthy ageing. The two groups show, however, emotional prosody recognition differences: older participants recognise emotional intentions of speakers less well than younger participants do. These findings were followed up by two neuro-stimulation studies specifically targeting the inferior frontal cortex to test if recognition improves during active stimulation relative to sham. Overall, results suggests that neither tDCS nor high-frequency tRNS stimulation at 2mA for 30 minutes facilitates emotional prosody recognition rates in healthy older adults.

Wearing N95, Surgical, and Cloth Face Masks Compromises the Perception of Emotion

According to the familiar axiom, the eyes are the window to the soul. However, wearing masks to prevent the spread of viruses such as COVID-19 involves obscuring a large portion of the face. Do the eyes carry sufficient information to allow for the accurate perception of emotions in dynamic expressions obscured by masks? What about the perception of the meanings of specific smiles? We addressed these questions in two studies. In the first, participants saw dynamic expressions of happiness, disgust, anger, and surprise that were covered by N95, surgical, or cloth masks or were uncovered and rated the extent to which the expressions conveyed each of the same four emotions. Across conditions, participants perceived significantly lower levels of the expressed (target) emotion in masked faces, and this was particularly true for expressions composed of more facial action in the lower part of the face. Higher levels of other (non-target) emotions were also perceived in masked expressions. In the second study, participants rated the extent to which three categories of smiles (reward, affiliation, and dominance) conveyed positive feelings, reassurance, and superiority, respectively. Masked smiles communicated less of the target signal than unmasked smiles, but not more of other possible signals. The present work extends recent studies of the effects of masked faces on the perception of emotion in its novel use of dynamic facial expressions (as opposed to still images) and the investigation of different types of smiles.

Emotion and Word Recognition for Unprocessed and Vocoded Speech Stimuli

OBJECTIVES

Individuals with cochlear implants (CIs) show reduced word and auditory emotion recognition abilities relative to their peers with normal hearing. Modern CI processing strategies are designed to preserve acoustic cues requisite for word recognition rather than those cues required for accessing other signal information (e.g., talker gender or emotional state). While word recognition is undoubtedly important for communication, the inaccessibility of this additional signal information in speech may lead to negative social experiences and outcomes for individuals with hearing loss. This study aimed to evaluate whether the emphasis on word recognition preservation in CI processing has unintended consequences on the perception of other talker information, such as emotional state.

DESIGN

Twenty-four young adult listeners with normal hearing listened to sentences and either reported a target word in each sentence (word recognition task) or selected the emotion of the talker (emotion recognition task) from a list of options (Angry, Calm, Happy, and Sad). Sentences were blocked by task type (emotion recognition versus word recognition) and processing condition (unprocessed versus 8-channel noise vocoder) and presented randomly within the block at three signal-to-noise ratios (SNRs) in a background of speech-shaped noise. Confusion matrices showed the number of errors in emotion recognition by listeners.

RESULTS

Listeners demonstrated better emotion recognition performance than word recognition performance at the same SNR. Unprocessed speech resulted in higher recognition rates than vocoded stimuli. Recognition performance (for both words and emotions) decreased with worsening SNR. Vocoding speech resulted in a greater negative impact on emotion recognition than it did for word recognition.

CONCLUSIONS

These data confirm prior work that suggests that in background noise, emotional prosodic information in speech is easier to recognize than word information, even after simulated CI processing. However, emotion recognition may be more negatively impacted by background noise and CI processing than word recognition. Future work could explore CI processing strategies that better encode prosodic information and investigate this effect in individuals with CIs as opposed to vocoded simulation. This study emphasized the need for clinicians to consider not only word recognition but also other aspects of speech that are critical to successful social communication.

Do People Agree on How Positive Emotions Are Expressed? A Survey of Four Emotions and Five Modalities Across 11 Cultures

While much is known about how negative emotions are expressed in different modalities, our understanding of the nonverbal expressions of positive emotions remains limited. In the present research, we draw upon disparate lines of theoretical and empirical work on positive emotions, and systematically examine which channels are thought to be used for expressing four positive emotions: feeling moved, gratitude, interest, and triumph. Employing the intersubjective approach, an established method in cross-cultural psychology, we first explored how the four positive emotions were reported to be expressed in two North American community samples (Studies 1a and 1b: n = 1466). We next confirmed the cross-cultural generalizability of our findings by surveying respondents from ten countries that diverged on cultural values (Study 2: n = 1826). Feeling moved was thought to be signaled with facial expressions, gratitude with the use of words, interest with words, face and voice, and triumph with body posture, vocal cues, facial expressions, and words. These findings provide cross-culturally consistent findings of differential expressions across positive emotions. Notably, positive emotions were thought to be expressed via modalities that go beyond the face.

Beyond Correlation: Acoustic Transformation Methods for the Experimental Study of Emotional Voice and Speech

While acoustic analysis methods have become a commodity in voice emotion research, experiments that attempt not only to describe but to computationally manipulate expressive cues in emotional voice and speech have remained relatively rare. We give here a nontechnical overview of voice-transformation techniques from the audio signal-processing community that we believe are ripe for adoption in this context. We provide sound examples of what they can achieve, examples of experimental questions for which they can be used, and links to open-source implementations. We point at a number of methodological properties of these algorithms, such as being specific, parametric, exhaustive, and real-time, and describe the new possibilities that these open for the experimental study of the emotional voice.

The perception of caricatured emotion in voice

Affective vocalisations such as screams and laughs can convey strong emotional content without verbal information. Previous research using morphed vocalisations (e.g. 25% fear/75% anger) has revealed categorical perception of emotion in voices, showing sudden shifts at emotion category boundaries. However, it is currently unknown how further modulation of vocalisations beyond the veridical emotion (e.g. 125% fear) affects perception. Caricatured facial expressions produce emotions that are perceived as more intense and distinctive, with faster recognition relative to the original and anti-caricatured (e.g. 75% fear) emotions, but a similar effect using vocal caricatures has not been previously examined. Furthermore, caricatures can play a key role in assessing how distinctiveness is identified, in particular by evaluating accounts of emotion perception with reference to prototypes (distance from the central stimulus) and exemplars (density of the stimulus space). Stimuli consisted of four emotions (anger, disgust, fear, and pleasure) morphed at 25% intervals between a neutral expression and each emotion from 25% to 125%, and between each pair of emotions. Emotion perception was assessed using emotion intensity ratings, valence and arousal ratings, speeded categorisation and paired similarity ratings. We report two key findings: 1) across tasks, there was a strongly linear effect of caricaturing, with caricatured emotions (125%) perceived as higher in emotion intensity and arousal, and recognised faster compared to the original emotion (100%) and anti-caricatures (25%-75%); 2) our results reveal evidence for a unique contribution of a prototype-based account in emotion recognition. We show for the first time that vocal caricature effects are comparable to those found previously with facial caricatures. The set of caricatured vocalisations provided open a promising line of research for investigating vocal affect perception and emotion processing deficits in clinical populations.

A neurocognitive model of perceptual decision-making on emotional signals

Humans make various kinds of decisions about which emotions they perceive from others. Although it might seem like a split-second phenomenon, deliberating over which emotions we perceive unfolds across several stages of decisional processing. Neurocognitive models of general perception postulate that our brain first extracts sensory information about the world then integrates these data into a percept and lastly interprets it. The aim of the present study was to build an evidence-based neurocognitive model of perceptual decision-making on others' emotions. We conducted a series of meta-analyses of neuroimaging data spanning 30 years on the explicit evaluations of others' emotional expressions. We find that emotion perception is rather an umbrella term for various perception paradigms, each with distinct neural structures that underline task-related cognitive demands. Furthermore, the left amygdala was responsive across all classes of decisional paradigms, regardless of task-related demands. Based on these observations, we propose a neurocognitive model that outlines the information flow in the brain needed for a successful evaluation of and decisions on other individuals' emotions. HIGHLIGHTS: Emotion classification involves heterogeneous perception and decision-making tasks Decision-making processes on emotions rarely covered by existing emotions theories We propose an evidence-based neuro-cognitive model of decision-making on emotions Bilateral brain processes for nonverbal decisions, left brain processes for verbal decisions Left amygdala involved in any kind of decision on emotions.

The representation and plasticity of body emotion expression

Emotions are expressed by the face, the voice and the whole body. Research on the face and the voice has not only demonstrated that emotions are perceived categorically, but that this perception can be manipulated. The purpose of this study was to investigate, via two separate experiments using adaptation and multisensory techniques, whether the perception of body emotion expressions also shows categorical effects and plasticity. We used an approach developed for studies investigating both face and voice emotion perception and created novel morphed affective body stimuli, which varied in small incremental steps between emotions. Participants were instructed to perform an emotion categorisation of these morphed bodies after adaptation to bodies conveying different expressions (Experiment 1), or while simultaneously hearing affective voices (Experiment 2). We show that not only is body expression perceived categorically, but that both adaptation to affective body expressions and concurrent presentation of vocal affective information can shift the categorical boundary between body expressions, specifically for the angry body expressions. Overall, our findings provide significant new insights into emotional body categorisation, which may prove important in gaining a deeper understanding of body expression perception in everyday social situations.

Pre-verbal infants perceive emotional facial expressions categorically

Adults perceive emotional expressions categorically, with discrimination being faster and more accurate between expressions from different emotion categories (i.e. blends with two different predominant emotions) than between two stimuli from the same category (i.e. blends with the same predominant emotion). The current study sought to test whether facial expressions of happiness and fear are perceived categorically by pre-verbal infants, using a new stimulus set that was shown to yield categorical perception in adult observers (Experiments 1 and 2). These stimuli were then used with 7-month-old infants (N  =  34) using a habituation and visual preference paradigm (Experiment 3). Infants were first habituated to an expression of one emotion, then presented with the same expression paired with a novel expression either from the same emotion category or from a different emotion category. After habituation to fear, infants displayed a novelty preference for pairs of between-category expressions, but not within-category ones, showing categorical perception. However, infants showed no novelty preference when they were habituated to happiness. Our findings provide evidence for categorical perception of emotional expressions in pre-verbal infants, while the asymmetrical effect challenges the notion of a bias towards negative information in this age group.

Music Communicates Affects, Not Basic Emotions - A Constructionist Account of Attribution of Emotional Meanings to Music

Basic Emotion theory has had a tremendous influence on the affective sciences, including music psychology, where most researchers have assumed that music expressivity is constrained to a limited set of basic emotions. Several scholars suggested that these constrains to musical expressivity are explained by the existence of a shared acoustic code to the expression of emotions in music and speech prosody. In this article we advocate for a shift from this focus on basic emotions to a constructionist account. This approach proposes that the phenomenon of perception of emotions in music arises from the interaction of music's ability to express core affects and the influence of top-down and contextual information in the listener's mind. We start by reviewing the problems with the concept of Basic Emotions, and the inconsistent evidence that supports it. We also demonstrate how decades of developmental and cross-cultural research on music and emotional speech have failed to produce convincing findings to conclude that music expressivity is built upon a set of biologically pre-determined basic emotions. We then examine the cue-emotion consistencies between music and speech, and show how they support a parsimonious explanation, where musical expressivity is grounded on two dimensions of core affect (arousal and valence). Next, we explain how the fact that listeners reliably identify basic emotions in music does not arise from the existence of categorical boundaries in the stimuli, but from processes that facilitate categorical perception, such as using stereotyped stimuli and close-ended response formats, psychological processes of construction of mental prototypes, and contextual information. Finally, we outline our proposal of a constructionist account of perception of emotions in music, and spell out the ways in which this approach is able to make solve past conflicting findings. We conclude by providing explicit pointers about the methodological choices that will be vital to move beyond the popular Basic Emotion paradigm and start untangling the emergence of emotional experiences with music in the actual contexts in which they occur.

Is There a Role for Language in Emotion Perception?

What is the relationship between language, emotion concepts, and perceptual categories? Here I compare the strong Whorfian view of linguistic relativity, which argues that language plays a necessary role in the perception of emotions, to the alternative view that different levels of processing (e.g., linguistic, conceptual, perceptual) are relatively independent and thus, that language does not play a foundational role in emotion perception. I examine neuropsychological studies that have tested strong claims of linguistic relativity, and discuss research on categorical perception of emotional expressions, where the two accounts have been directly tested against each other. As in other perceptual domains, there is little evidence that language plays a foundational role in the perception of emotion.

Children's and adults' perception of questions and statements from terminal fundamental frequency contours

The present study compared children's and adults' identification and discrimination of declarative questions and statements on the basis of terminal cues alone. Children (8-11 years, n = 41) and adults (n = 21) judged utterances as statements or questions from sentences with natural statement and question endings and with manipulated endings that featured intermediate fundamental frequency (F₀) values. The same adults and a different sample of children (n = 22) were also tested on their discrimination of the utterances. Children's judgments shifted more gradually across categories than those of adults, but their category boundaries were comparable. In the discrimination task, adults found cross-boundary comparisons more salient than within-boundary comparisons. Adults' performance on the identification and discrimination tasks is consistent with but not definitive regarding categorical perception of statements and questions. Children, by contrast, discriminated the cross-boundary comparisons no better than other comparisons. The findings indicate age-related sharpening in the perception of statements and questions based on terminal F₀ cues and the gradual emergence of distinct perceptual categories.

The contribution of phonation type to the perception of vocal emotions in German: an articulatory synthesis study

Vocal emotions are signaled by specific patterns of prosodic parameters, most notably pitch, phone duration, intensity, and phonation type. Phonation type was so far the least accessible parameter in emotion research, because it was difficult to extract from speech signals and difficult to manipulate in natural or synthetic speech. The present study built on recent advances in articulatory speech synthesis to exclusively control phonation type in re-synthesized German sentences spoken with seven different emotions. The goal was to find out to what extent the sole change of phonation type affects the perception of these emotions. Therefore, portrayed emotional utterances were re-synthesized with their original phonation type, as well as with each purely breathy, modal, and pressed phonation, and then rated by listeners with respect to the perceived emotions. Highly significant effects of phonation type on the recognition rates of the original emotions were found, except for disgust. While fear, anger, and the neutral emotion require specific phonation types for correct perception, sadness, happiness, boredom, and disgust primarily rely on other prosodic parameters. These results can help to improve the expression of emotions in synthesized speech and facilitate the robust automatic recognition of vocal emotions.

What does music express? Basic emotions and beyond

Numerous studies have investigated whether music can reliably convey emotions to listeners, and—if so—what musical parameters might carry this information. Far less attention has been devoted to the actual contents of the communicative process. The goal of this article is thus to consider what types of emotional content are possible to convey in music. I will argue that the content is mainly constrained by the type of coding involved, and that distinct types of content are related to different types of coding. Based on these premises, I suggest a conceptualization in terms of “multiple layers” of musical expression of emotions. The “core” layer is constituted by iconically-coded basic emotions. I attempt to clarify the meaning of this concept, dispel the myths that surround it, and provide examples of how it can be heuristic in explaining findings in this domain. However, I also propose that this “core” layer may be extended, qualified, and even modified by additional layers of expression that involve intrinsic and associative coding. These layers enable listeners to perceive more complex emotions—though the expressions are less cross-culturally invariant and more dependent on the social context and/or the individual listener. This multiple-layer conceptualization of expression in music can help to explain both similarities and differences between vocal and musical expression of emotions.

Context-dependent categorical perception of surprise

Evidence regarding the categorical perception of surprise facial expressions has been equivocal. Surprise is inherently ambiguous with respect to valence: it could be positive or negative. If this ambiguity interferes with categorical perception, disambiguating the valence might facilitate categorical perception. Participants identified and discriminated images that were selected from expression continua: happy-fear, surprise-fear, happy-surprise. Half were presented with a context for the surprise expressions indicating positive or negative valence. Both groups had a typical identification curve, but discrimination performance was better predicted by identification in the context condition for happy-fear and surprise-fear continua, suggesting that categorical perception was facilitated by the disambiguating context.

Feeling backwards? How temporal order in speech affects the time course of vocal emotion recognition

Recent studies suggest that the time course for recognizing vocal expressions of basic emotion in speech varies significantly by emotion type, implying that listeners uncover acoustic evidence about emotions at different rates in speech (e.g., fear is recognized most quickly whereas happiness and disgust are recognized relatively slowly; Pell and Kotz, 2011). To investigate whether vocal emotion recognition is largely dictated by the amount of time listeners are exposed to speech or the position of critical emotional cues in the utterance, 40 English participants judged the meaning of emotionally-inflected pseudo-utterances presented in a gating paradigm, where utterances were gated as a function of their syllable structure in segments of increasing duration from the end of the utterance (i.e., gated syllable-by-syllable from the offset rather than the onset of the stimulus). Accuracy for detecting six target emotions in each gate condition and the mean identification point for each emotion in milliseconds were analyzed and compared to results from Pell and Kotz (2011). We again found significant emotion-specific differences in the time needed to accurately recognize emotions from speech prosody, and new evidence that utterance-final syllables tended to facilitate listeners' accuracy in many conditions when compared to utterance-initial syllables. The time needed to recognize fear, anger, sadness, and neutral from speech cues was not influenced by how utterances were gated, although happiness and disgust were recognized significantly faster when listeners heard the end of utterances first. Our data provide new clues about the relative time course for recognizing vocally-expressed emotions within the 400–1200 ms time window, while highlighting that emotion recognition from prosody can be shaped by the temporal properties of speech.

Perceptual adaptation to gender and expressive properties in speech: the role of fundamental frequency

Recent studies have demonstrated perceptual adaptation to nonlinguistic properties of speech involving voice gender and emotional expression. The present study extends this work by examining the contribution of fundamental frequency (F0) to these effects. Voice recordings of vowel-consonant-vowel (VCV) syllables from six talkers were processed using the STRAIGHT vocoder and an auditory morphing technique to synthesize gender (experiment 1) and expressive (experiment 2) speech sound continua ranging from one category endpoint to the other (female to male; angry to happy). Continuum endpoints served as adaptors for F0 present and F0 removed conditions. F0 removed stimuli were created by replacing the periodic excitation source with broadband noise. Confirming previous findings, aftereffects were found in the F0 present condition, resulting in a decreased likelihood to identify test stimuli as belonging to the adaptor category. No aftereffects appeared when F0 was removed, highlighting the importance of F0 in adaptation. However, in an identification test listeners were still able to categorize F0 removed stimuli at better-than-chance levels, indicating that residual cues for gender and emotion were available even when F0 was not present.

Sensory-based and higher-order operations contribute to abnormal emotional prosody processing in schizophrenia: an electrophysiological investigation

BACKGROUND

Schizophrenia is characterized by deficits in emotional prosody (EP) perception. However, it is not clear which stages of processing prosody are abnormal and whether the presence of semantic content contributes to the abnormality. This study aimed to examine event-related potential (ERP) correlates of EP processing in 15 chronic schizophrenia individuals and 15 healthy controls.

METHOD

A total of 114 sentences with neutral semantic content [sentences with semantic content (SSC) condition] were generated by a female speaker (38 with happy, 38 with angry, and 38 with neutral intonation). The same sentences were synthesized and presented in the 'pure prosody' sentences (PPS) condition where semantic content was unintelligible.

RESULTS

Group differences were observed for N100 and P200 amplitude: patients were characterized by more negative N100 for SSC, and more positive P200 for angry and happy SSC and happy PPS. Correlations were found between delusions and P200 amplitude for happy SSC and PPS. Higher error rates in the recognition of EP were also observed in schizophrenia: higher error rates in neutral SSC were associated with reduced N100, and higher error rates in angry SSC were associated with reduced P200.

CONCLUSIONS

These results indicate that abnormalities in prosody processing occur at the three stages of EP processing, and are enhanced in SSC. Correlations between P200 amplitude for happy prosody and delusions suggest a role that abnormalities in the processing of emotionally salient acoustic cues may play in schizophrenia symptomatology. Correlations between ERP and behavioral data point to a relationship between early sensory abnormalities and prosody recognition in schizophrenia.

Voice identity discrimination in schizophrenia

Voices provide a wealth of socially-relevant information, including cues to a speaker's identity and emotion. Deficits recognising emotion from voice have been extensively described in schizophrenia, and linked specifically to auditory hallucinations (AH), but relatively little attention has been given to examining the ability to analyse speaker identity. Hence, the current study assessed the ability to discriminate between different speakers in people with schizophrenia (including 33 with and 32 without AH) compared to 32 healthy controls. Participants rated the degree of perceived identity similarity of pairs of unfamiliar voices pronouncing three-syllable words. Multidimensional scaling of the dissimilarity matrices was performed and the resulting dimensions were interpreted, a posteriori, via correlations with acoustic measures relevant to voice identity. A two-dimensional perceptual space was found to be appropriate for both schizophrenia patients and controls, with axes corresponding to the average fundamental frequency (F0) and formant dispersion (D(f)). Patients with schizophrenia did not differ from healthy controls in their reliance on F0 in differentiating voices, suggesting that the ability to use pitch-based cues for discriminating voice identity may be relatively preserved in schizophrenia. On the other hand, patients (both with and without AH) made less use of D(f) in discriminating voices compared to healthy controls. This distorted pattern of responses suggests some potentially important differences in voice identity processing in schizophrenia. Formant dispersion has been linked to perceptions of dominance, masculinity, size and age in healthy individuals. These findings open some interesting new directions for future research.

Authenticity affects the recognition of emotions in speech: behavioral and fMRI evidence

The aim of the present study was to determine how authenticity of emotion expression in speech modulates activity in the neuronal substrates involved in emotion recognition. Within an fMRI paradigm, participants judged either the authenticity (authentic or play acted) or emotional content (anger, fear, joy, or sadness) of recordings of spontaneous emotions and reenactments by professional actors. When contrasting between task types, active judgment of authenticity, more than active judgment of emotion, indicated potential involvement of the theory of mind (ToM) network (medial prefrontal cortex, temporoparietal cortex, retrosplenium) as well as areas involved in working memory and decision making (BA 47). Subsequently, trials with authentic recordings were contrasted with those of reenactments to determine the modulatory effects of authenticity. Authentic recordings were found to enhance activity in part of the ToM network (medial prefrontal cortex). This effect of authenticity suggests that individuals integrate recollections of their own experiences more for judgments involving authentic stimuli than for those involving play-acted stimuli. The behavioral and functional results show that authenticity of emotional prosody is an important property influencing human responses to such stimuli, with implications for studies using play-acted emotions.

Reference repulsion in the categorical perception of biological motion

Perceiving biological motion is important for understanding the intentions and future actions of others. Perceiving an approaching person's behavior may be particularly important, because such behavior often precedes social interaction. To this end, the visual system may devote extra resources for perceiving an oncoming person's heading. If this were true, humans should show increased sensitivity for perceiving approaching headings, and as a result, a repulsive perceptual effect around the categorical boundary of leftward/rightward motion. We tested these predictions and found evidence for both. First, observers were especially sensitive to the heading of an approaching person; variability in estimates of a person's heading decreased near the category boundary of leftward/rightward motion. Second, we found a repulsion effect around the category boundary; a person walking approximately toward the observer was perceived as being repelled away from straight ahead. This repulsive effect was greatly exaggerated for perception of a very briefly presented person or perception of a chaotic crowd, suggesting that repulsion may protect against categorical errors when sensory noise is high. The repulsion effect with a crowd required integration of local motion and human form, suggesting an origin in high-level stages of visual processing. Similar repulsive effects may underlie categorical perception with other social features. Overall, our results show that a person's direction of walking is categorically perceived, with improved sensitivity at the category boundary and a concomitant repulsion effect.

On the time course of vocal emotion recognition

How quickly do listeners recognize emotions from a speaker's voice, and does the time course for recognition vary by emotion type? To address these questions, we adapted the auditory gating paradigm to estimate how much vocal information is needed for listeners to categorize five basic emotions (anger, disgust, fear, sadness, happiness) and neutral utterances produced by male and female speakers of English. Semantically-anomalous pseudo-utterances (e.g., The rivix jolled the silling) conveying each emotion were divided into seven gate intervals according to the number of syllables that listeners heard from sentence onset. Participants (n = 48) judged the emotional meaning of stimuli presented at each gate duration interval, in a successive, blocked presentation format. Analyses looked at how recognition of each emotion evolves as an utterance unfolds and estimated the “identification point” for each emotion. Results showed that anger, sadness, fear, and neutral expressions are recognized more accurately at short gate intervals than happiness, and particularly disgust; however, as speech unfolds, recognition of happiness improves significantly towards the end of the utterance (and fear is recognized more accurately than other emotions). When the gate associated with the emotion identification point of each stimulus was calculated, data indicated that fear (M = 517 ms), sadness (M = 576 ms), and neutral (M = 510 ms) expressions were identified from shorter acoustic events than the other emotions. These data reveal differences in the underlying time course for conscious recognition of basic emotions from vocal expressions, which should be accounted for in studies of emotional speech processing.

Preliminary Evidence of Pre-Attentive Distinctions of Frequency-Modulated Tones that Convey Affect

Recognizing emotion is an evolutionary imperative. An early stage of auditory scene analysis involves the perceptual grouping of acoustic features, which can be based on both temporal coincidence and spectral features such as perceived pitch. Perceived pitch, or fundamental frequency (F₀), is an especially salient cue for differentiating affective intent through speech intonation (prosody). We hypothesized that: (1) simple frequency-modulated tone abstractions, based on the parameters of actual prosodic stimuli, would be reliably classified as representing differing emotional categories; and (2) that such differences would yield significant mismatch negativities (MMNs) – an index of pre-attentive deviance detection within the auditory environment. We constructed a set of FM tones that approximated the F₀ mean and variation of reliably recognized happy and neutral prosodic stimuli. These stimuli were presented to 13 subjects using a passive listening oddball paradigm. We additionally included stimuli with no frequency modulation (FM) and FM tones with identical carrier frequencies but differing modulation depths as control conditions. Following electrophysiological recording, subjects were asked to identify the sounds they heard as happy, sad, angry, or neutral. We observed that FM tones abstracted from happy and no-expression speech stimuli elicited MMNs. Post hoc behavioral testing revealed that subjects reliably identified the FM tones in a consistent manner. Finally, we also observed that FM tones and no-FM tones elicited equivalent MMNs. MMNs to FM tones that differentiate affect suggests that these abstractions may be sufficient to characterize prosodic distinctions, and that these distinctions can be represented in pre-attentive auditory sensory memory.

Discrete emotion effects on lexical decision response times

Our knowledge about affective processes, especially concerning effects on cognitive demands like word processing, is increasing steadily. Several studies consistently document valence and arousal effects, and although there is some debate on possible interactions and different notions of valence, broad agreement on a two dimensional model of affective space has been achieved. Alternative models like the discrete emotion theory have received little interest in word recognition research so far. Using backward elimination and multiple regression analyses, we show that five discrete emotions (i.e., happiness, disgust, fear, anger and sadness) explain as much variance as two published dimensional models assuming continuous or categorical valence, with the variables happiness, disgust and fear significantly contributing to this account. Moreover, these effects even persist in an experiment with discrete emotion conditions when the stimuli are controlled for emotional valence and arousal levels. We interpret this result as evidence for discrete emotion effects in visual word recognition that cannot be explained by the two dimensional affective space account.

Emotional speech processing: disentangling the effects of prosody and semantic cues

To inform how emotions in speech are implicitly processed and registered in memory, we compared how emotional prosody, emotional semantics, and both cues in tandem prime decisions about conjoined emotional faces. Fifty-two participants rendered facial affect decisions (Pell, 2005a), indicating whether a target face represented an emotion (happiness or sadness) or not (a facial grimace), after passively listening to happy, sad, or neutral prime utterances. Emotional information from primes was conveyed by: (1) prosody only; (2) semantic cues only; or (3) combined prosody and semantic cues. Results indicated that prosody, semantics, and combined prosody-semantic cues facilitate emotional decisions about target faces in an emotion-congruent manner. However, the magnitude of priming did not vary across tasks. Our findings highlight that emotional meanings of prosody and semantic cues are systematically registered during speech processing, but with similar effects on associative knowledge about emotions, which is presumably shared by prosody, semantics, and faces.

When doors of perception close: bottom-up models of disrupted cognition in schizophrenia

Schizophrenia is a major mental disorder that affects approximately 1% of the population worldwide. Cognitive deficits are a key feature of schizophrenia and a primary cause of long-term disability. Current neurophysiological models of schizophrenia focus on distributed brain dysfunction with bottom-up as well as top-down components. Bottom-up deficits in cognitive processing are driven by impairments in basic perceptual processes that localize to primary sensory brain regions. Within the auditory system, deficits are apparent in elemental sensory processing, such as tone matching following brief delay. Such deficits lead to impairments in higher-order processes such as phonological processing and auditory emotion recognition. Within the visual system, deficits are apparent in functioning of the magnocellular visual pathway, leading to higher-order deficits in processes such as perceptual closure, object recognition, and reading. In both auditory and visual systems, patterns of deficit are consistent with underlying impairment of brain N-methyl-d-aspartate receptor systems.