Modality-specific attention and multisensory integration of emotions in schizophrenia: reduced regulatory effects

Increases in sensory noise predict attentional disruptions to audiovisual speech perception

We receive information about the world around us from multiple senses which combine in a process known as multisensory integration. Multisensory integration has been shown to be dependent on attention; however, the neural mechanisms underlying this effect are poorly understood. The current study investigates whether changes in sensory noise explain the effect of attention on multisensory integration and whether attentional modulations to multisensory integration occur via modality-specific mechanisms. A task based on the McGurk Illusion was used to measure multisensory integration while attention was manipulated via a concurrent auditory or visual task. Sensory noise was measured within modality based on variability in unisensory performance and was used to predict attentional changes to McGurk perception. Consistent with previous studies, reports of the McGurk illusion decreased when accompanied with a secondary task; however, this effect was stronger for the secondary visual (as opposed to auditory) task. While auditory noise was not influenced by either secondary task, visual noise increased with the addition of the secondary visual task specifically. Interestingly, visual noise accounted for significant variability in attentional disruptions to the McGurk illusion. Overall, these results strongly suggest that sensory noise may underlie attentional alterations to multisensory integration in a modality-specific manner. Future studies are needed to determine whether this finding generalizes to other types of multisensory integration and attentional manipulations. This line of research may inform future studies of attentional alterations to sensory processing in neurological disorders, such as Schizophrenia, Autism, and ADHD.

Tracing Links Between Early Auditory Information Processing and Negative Symptoms in Schizophrenia: An ERP Study

Background: Negative symptoms represent a heterogeneous dimension with a strong impact on functioning of subjects with schizophrenia (SCZ). Five constructs are included in this dimension: anhedonia, asociality, avolition, blunted affect, and alogia. Factor analyses revealed that these symptoms cluster in two domains: experiential domain (avolition, asociality, and anhedonia) and the expressive deficit (alogia and blunted affect), that might be linked to different neurobiological alterations. Few studies investigated associations between N100, an electrophysiological index of early sensory processing, and negative symptoms, reporting controversial results. However, none of these studies investigated electrophysiological correlates of the two negative symptom domains. Objectives: The aim of our study was to evaluate, within the multicenter study of the Italian Network for Research on Psychoses, the relationships between N100 and negative symptom domains in SCZ. Methods: Auditory N100 was analyzed in 114 chronic stabilized SCZ and 63 healthy controls (HCs). Negative symptoms were assessed with the Brief Negative Symptom Scale (BNSS). Repeated measures ANOVA and correlation analyses were performed to evaluate differences between SCZ and HCs and association of N100 features with negative symptoms. Results: Our findings demonstrated a significant N100 amplitude reduction in SCZ compared with HCs. In SCZ, N100 amplitude for standard stimuli was associated with negative symptoms, in particular with the expressive deficit domain. Within the expressive deficit, blunted affect and alogia had the same pattern of correlation with N100. Conclusion: Our findings revealed an association between expressive deficit and N100, suggesting that these negative symptoms might be related to deficits in early auditory processing in SCZ.

Multisensory Integration of Emotion in Schizophrenic Patients

Multisensory integration (MSI) of emotion has been increasingly recognized as an essential element of schizophrenic patients' impairments, leading to the breakdown of their interpersonal functioning. The present review provides an updated synopsis of schizophrenics' MSI abilities in emotion processing by examining relevant behavioral and neurological research. Existing behavioral studies have adopted well-established experimental paradigms to investigate how participants understand multisensory emotion stimuli, and interpret their reciprocal interactions. Yet it remains controversial with regard to congruence-induced facilitation effects, modality dominance effects, and generalized vs specific impairment hypotheses. Such inconsistencies are likely due to differences and variations in experimental manipulations, participants' clinical symptomatology, and cognitive abilities. Recent electrophysiological and neuroimaging research has revealed aberrant indices in event-related potential (ERP) and brain activation patterns, further suggesting impaired temporal processing and dysfunctional brain regions, connectivity and circuities at different stages of MSI in emotion processing. The limitations of existing studies and implications for future MSI work are discussed in light of research designs and techniques, study samples and stimuli, and clinical applications.

Prosody Dominates Over Semantics in Emotion Word Processing: Evidence From Cross-Channel and Cross-Modal Stroop Effects

Purpose Emotional speech communication involves multisensory integration of linguistic (e.g., semantic content) and paralinguistic (e.g., prosody and facial expressions) messages. Previous studies on linguistic versus paralinguistic salience effects in emotional speech processing have produced inconsistent findings. In this study, we investigated the relative perceptual saliency of emotion cues in cross-channel auditory alone task (i.e., semantics-prosody Stroop task) and cross-modal audiovisual task (i.e., semantics-prosody-face Stroop task). Method Thirty normal Chinese adults participated in two Stroop experiments with spoken emotion adjectives in Mandarin Chinese. Experiment 1 manipulated auditory pairing of emotional prosody (happy or sad) and lexical semantic content in congruent and incongruent conditions. Experiment 2 extended the protocol to cross-modal integration by introducing visual facial expression during auditory stimulus presentation. Participants were asked to judge emotional information for each test trial according to the instruction of selective attention. Results Accuracy and reaction time data indicated that, despite an increase in cognitive demand and task complexity in Experiment 2, prosody was consistently more salient than semantic content for emotion word processing and did not take precedence over facial expression. While congruent stimuli enhanced performance in both experiments, the facilitatory effect was smaller in Experiment 2. Conclusion Together, the results demonstrate the salient role of paralinguistic prosodic cues in emotion word processing and congruence facilitation effect in multisensory integration. Our study contributes tonal language data on how linguistic and paralinguistic messages converge in multisensory speech processing and lays a foundation for further exploring the brain mechanisms of cross-channel/modal emotion integration with potential clinical applications.

Impaired facial and vocal emotion decoding in schizophrenia is underpinned by basic perceptivo-motor deficits

INTRODUCTION

Emotional decoding impairments have been largely demonstrated in schizophrenia for facial and prosodic stimuli, when presented separately. Nevertheless, the exploration of crossmodal integration has been far less considered, despite its omnipresence in daily social interactions. Moreover, the role played by basic visuo-motor impairments in unimodal and crossmodal decoding remains unexplored.

METHODS

Thirty-two patients were compared with 32 matched controls in an emotional decoding task including unimodal (visual and auditory) and crossmodal (congruent and incongruent) conditions. A control perceptive task was also conducted to take potential low-level perceptual deficits into account.

RESULTS

Schizoprenic patients presented lower performance and higher reaction times for both unimodal tasks (visual and auditory) and crossmodal conditions. Moreover, reaction times for the visuo-perceptive task were also significantly longer for patients compared to controls.

CONCLUSIONS

The consistency of the results across unimodal and crossmodal tasks suggests a globalised emotional impairment in schizophrenia, independent of the sensorial modality and crossmodal nature of the stimuli. Centrally, given the results in the visuo-perceptive task, the impairments observed for emotional recognition appears at least partly explained by primary cognitive deficits, namely reduced processing speed.

On the effects of multimodal information integration in multitasking

There have recently been considerable advances in our understanding of the neuronal mechanisms underlying multitasking, but the role of multimodal integration for this faculty has remained rather unclear. We examined this issue by comparing different modality combinations in a multitasking (stop-change) paradigm. In-depth neurophysiological analyses of event-related potentials (ERPs) were conducted to complement the obtained behavioral data. Specifically, we applied signal decomposition using second order blind identification (SOBI) to the multi-subject ERP data and source localization. We found that both general multimodal information integration and modality-specific aspects (potentially related to task difficulty) modulate behavioral performance and associated neurophysiological correlates. Simultaneous multimodal input generally increased early attentional processing of visual stimuli (i.e. P1 and N1 amplitudes) as well as measures of cognitive effort and conflict (i.e. central P3 amplitudes). Yet, tactile-visual input caused larger impairments in multitasking than audio-visual input. General aspects of multimodal information integration modulated the activity in the premotor cortex (BA 6) as well as different visual association areas concerned with the integration of visual information with input from other modalities (BA 19, BA 21, BA 37). On top of this, differences in the specific combination of modalities also affected performance and measures of conflict/effort originating in prefrontal regions (BA 6).

Visual Distractors Disrupt Audiovisual Integration Regardless of Stimulus Complexity

The intricate relationship between multisensory integration and attention has been extensively researched in the multisensory field; however, the necessity of attention for the binding of multisensory stimuli remains contested. In the current study, we investigated whether diverting attention from well-known multisensory tasks would disrupt integration and whether the complexity of the stimulus and task modulated this interaction. A secondary objective of this study was to investigate individual differences in the interaction of attention and multisensory integration. Participants completed a simple audiovisual speeded detection task and McGurk task under various perceptual load conditions: no load (multisensory task while visual distractors present), low load (multisensory task while detecting the presence of a yellow letter in the visual distractors), and high load (multisensory task while detecting the presence of a number in the visual distractors). Consistent with prior studies, we found that increased perceptual load led to decreased reports of the McGurk illusion, thus confirming the necessity of attention for the integration of speech stimuli. Although increased perceptual load led to longer response times for all stimuli in the speeded detection task, participants responded faster on multisensory trials than unisensory trials. However, the increase in multisensory response times violated the race model for no and low perceptual load conditions only. Additionally, a geometric measure of Miller’s inequality showed a decrease in multisensory integration for the speeded detection task with increasing perceptual load. Surprisingly, we found diverging changes in multisensory integration with increasing load for participants who did not show integration for the no load condition: no changes in integration for the McGurk task with increasing load but increases in integration for the detection task. The results of this study indicate that attention plays a crucial role in multisensory integration for both highly complex and simple multisensory tasks and that attention may interact differently with multisensory processing in individuals who do not strongly integrate multisensory information.

Temporal orienting precedes intersensory attention and has opposing effects on early evoked brain activity

Intersensory attention (IA) describes the process of directing attention to a specific modality. Temporal orienting (TO) characterizes directing attention to a specific moment in time. Previously, studies indicated that these two processes could have opposite effects on early evoked brain activity. The exact time-course and processing stages of both processes are still unknown. In this human electroencephalography study, we investigated the effects of IA and TO on visuo-tactile stimulus processing within one paradigm. IA was manipulated by presenting auditory cues to indicate whether participants should detect visual or tactile targets in visuo-tactile stimuli. TO was manipulated by presenting stimuli block-wise at fixed or variable inter-stimulus intervals. We observed that TO affects evoked activity to visuo-tactile stimuli prior to IA. Moreover, we found that TO reduces the amplitude of early evoked brain activity, whereas IA enhances it. Using beamformer source-localization, we observed that IA increases neural responses in sensory areas of the attended modality whereas TO reduces brain activity in widespread cortical areas. Based on these findings we derive an updated working model for the effects of temporal and intersensory attention on early evoked brain activity.

Identifying Core Affect in Individuals from fMRI Responses to Dynamic Naturalistic Audiovisual Stimuli

Recent research has demonstrated that affective states elicited by viewing pictures varying in valence and arousal are identifiable from whole brain activation patterns observed with functional magnetic resonance imaging (fMRI). Identification of affective states from more naturalistic stimuli has clinical relevance, but the feasibility of identifying these states on an individual trial basis from fMRI data elicited by dynamic multimodal stimuli is unclear. The goal of this study was to determine whether affective states can be similarly identified when participants view dynamic naturalistic audiovisual stimuli. Eleven participants viewed 5s audiovisual clips in a passive viewing task in the scanner. Valence and arousal for individual trials were identified both within and across participants based on distributed patterns of activity in areas selectively responsive to audiovisual naturalistic stimuli while controlling for lower level features of the stimuli. In addition, the brain regions identified by searchlight analyses to represent valence and arousal were consistent with previously identified regions associated with emotion processing. These findings extend previous results on the distributed representation of affect to multimodal dynamic stimuli.

Influence of attention on bimodal integration during emotional change decoding: ERP evidence

Recent findings on audiovisual emotional interactions suggest that selective attention affects cross-sensory interaction from an early processing stage. However, the influence of attention manipulation on facial-vocal integration during emotional change perception is still elusive at this point. To address this issue, we asked participants to detect emotional changes conveyed by prosodies (vocal task) or facial expressions (facial task) while facial, vocal, and facial-vocal expressions were presented. At the same time, behavioral responses and electroencephalogram (EEG) were recorded. Behavioral results showed that bimodal emotional changes were detected with shorter response latencies compared to each unimodal condition, suggesting that bimodal emotional cues facilitated emotional change detection. Moreover, while the P3 amplitudes were larger for the bimodal change condition than for the sum of the two unimodal conditions regardless of attention direction, the N1 amplitudes were larger for the bimodal emotional change condition than for the sum of the two unimodal conditions under the attend-voice condition, but not under the attend-face condition. These findings suggest that selective attention modulates facial-vocal integration during emotional change perception in early sensory processing, but not in late cognitive processing stages.

Simultaneous face and voice processing in schizophrenia

While several studies have consistently demonstrated abnormalities in the unisensory processing of face and voice in schizophrenia (SZ), the extent of abnormalities in the simultaneous processing of both types of information remains unclear. To address this issue, we used event-related potentials (ERP) methodology to probe the multisensory integration of face and non-semantic sounds in schizophrenia. EEG was recorded from 18 schizophrenia patients and 19 healthy control (HC) subjects in three conditions: neutral faces (visual condition-VIS); neutral non-semantic sounds (auditory condition-AUD); neutral faces presented simultaneously with neutral non-semantic sounds (audiovisual condition-AUDVIS). When compared with HC, the schizophrenia group showed less negative N170 to both face and face-voice stimuli; later P270 peak latency in the multimodal condition of face-voice relative to unimodal condition of face (the reverse was true in HC); reduced P400 amplitude and earlier P400 peak latency in the face but not in the voice-face condition. Thus, the analysis of ERP components suggests that deficits in the encoding of facial information extend to multimodal face-voice stimuli and that delays exist in feature extraction from multimodal face-voice stimuli in schizophrenia. In contrast, categorization processes seem to benefit from the presentation of simultaneous face-voice information. Timepoint by timepoint tests of multimodal integration did not suggest impairment in the initial stages of processing in schizophrenia.

Facial, vocal and cross-modal emotion processing in early-onset schizophrenia spectrum disorders

Recognition of emotional expressions plays an essential role in children's healthy development. Anomalies in these skills may result in empathy deficits, social interaction difficulties and premorbid emotional problems in children and adolescents with schizophrenia. Twenty-six subjects with early onset schizophrenia spectrum (EOSS) disorders and twenty-eight matched healthy controls (HC) were instructed to identify five basic emotions and a neutral expression. The assessment entailed presenting visual, auditory and congruent cross-modal stimuli. Using a generalized linear mixed model, we found no significant association for handedness, age or gender. However, significant associations emerged for emotion type, perception modality, and group. EOSS patients performed worse than HC in uni- and cross-modal emotional tasks with a specific negative emotion processing impairment pattern. There was no relationship between emotion identification scores and positive or negative symptoms, self-reported empathy traits or a positive history of developmental disorders. However, we found a significant association between emotional identification scores and nonverbal communication impairments. We conclude that cumulative dysfunctions in both nonverbal communication and emotion processing contribute to the social vulnerability and morbidity found in youths who display EOSS disorder.

Mirroring the self: testing neurophysiological correlates of disturbed self-experience in schizophrenia spectrum

BACKGROUND

Self-disorders (SDs) have been described as a core schizophrenia spectrum vulnerability phenotype, both in classic and contemporary psychopathological literature. However, such a core phenotype has not yet been investigated adopting a trans-domain approach that combines the phenomenological and the neurophysiological levels of analysis. The aim of this study is to investigate the relation between SDs and subtle, schizophrenia-specific impairments of emotional resonance that are supposed to reflect abnormalities in the mirror neurons mechanism. Specifically, we tested whether electromyographic response to emotional stimuli (i.e. a proxy for subtle changes in facial mimicry and related motor resonance mechanisms) would predict the occurrence of anomalous subjective experiences (i.e. SDs).

SAMPLING AND METHODS

Eighteen schizophrenia spectrum (SzSp) patients underwent a comprehensive psychopathological examination and were contextually tested with a multimodal paradigm, recording facial electromyographic activity of muscles in response to positive and negative emotional stimuli. Experiential anomalies were explored with the Bonn Scale for the Assessment of Basic Symptoms (BSABS) and then condensed into rational subscales mapping SzSp anomalous self-experiences.

RESULTS

SzSp patients showed an imbalance in emotional motor resonance with a selective bias toward negative stimuli, as well as a multisensory integration impairment. Multiple regression analysis showed that electromyographic facial reactions in response to negative stimuli presented in auditory modality specifically and strongly correlated with SD subscore.

CONCLUSIONS

The study confirms the potential of SDs as target phenotype for neurobiological research and encourages research into disturbed motor/emotional resonance as possible body-level correlate of disturbed subjective experiences in SzSp.

Responses of prefrontal multisensory neurons to mismatching faces and vocalizations

Social communication relies on the integration of auditory and visual information, which are present in faces and vocalizations. Evidence suggests that the integration of information from multiple sources enhances perception compared with the processing of a unimodal stimulus. Our previous studies demonstrated that single neurons in the ventrolateral prefrontal cortex (VLPFC) of the rhesus monkey (Macaca mulatta) respond to and integrate conspecific vocalizations and their accompanying facial gestures. We were therefore interested in how VLPFC neurons respond differentially to matching (congruent) and mismatching (incongruent) faces and vocalizations. We recorded VLPFC neurons during the presentation of movies with congruent or incongruent species-specific facial gestures and vocalizations as well as their unimodal components. Recordings showed that while many VLPFC units are multisensory and respond to faces, vocalizations, or their combination, a subset of neurons showed a significant change in neuronal activity in response to incongruent versus congruent vocalization movies. Among these neurons, we typically observed incongruent suppression during the early stimulus period and incongruent enhancement during the late stimulus period. Incongruent-responsive VLPFC neurons were both bimodal and nonlinear multisensory, fostering their ability to respond to changes in either modality of a face-vocalization stimulus. These results demonstrate that ventral prefrontal neurons respond to changes in either modality of an audiovisual stimulus, which is important in identity processing and for the integration of multisensory communication information.

Perception of emotion-related conflict in human communications: what are the effects of schizophrenia?

Our ability to make sense of emotional cues is of paramount importance for understanding state of mind and communicative intent. However, emotional cues often conflict with each other; this presents a significant challenge for people with schizophrenia. We conducted a theoretical review to determine the extent and types of impaired processing of emotion-related conflict in schizophrenia; we evaluated the relationship with medication and symptoms, and considered possible mediatory mechanisms. The literature established that people with schizophrenia demonstrated impaired function: (i) when passively exposed to emotion cues whilst performing an unrelated task, (ii) when selectively attending to one source of emotion cues whilst trying to ignore interference from another source, and (iii) when trying to resolve conflicting emotion cues and judge meta-communicative intent. These deficits showed associations with both negative and positive symptoms. There was limited evidence for antipsychotic medications attenuating impaired emotion perception when there are conflicting cues, with further direct research needed. Impaired attentional control and context processing may underlie some of the observed impairments. Neuroanatomical correlates are likely to involve interhemispheric transfer via the corpus callosum, limbic regions such as the amygdala, and possibly dorsolateral prefrontal and anterior cingulate cortex through their role in conflict processing.

Multisensory stimuli elicit altered oscillatory brain responses at gamma frequencies in patients with schizophrenia

Deficits in auditory and visual unisensory responses are well documented in patients with schizophrenia; however, potential abnormalities elicited from multisensory audio-visual stimuli are less understood. Further, schizophrenia patients have shown abnormal patterns in task-related and task-independent oscillatory brain activity, particularly in the gamma frequency band. We examined oscillatory responses to basic unisensory and multisensory stimuli in schizophrenia patients (N = 46) and healthy controls (N = 57) using magnetoencephalography (MEG). Time-frequency decomposition was performed to determine regions of significant changes in gamma band power by group in response to unisensory and multisensory stimuli relative to baseline levels. Results showed significant behavioral differences between groups in response to unisensory and multisensory stimuli. In addition, time-frequency analysis revealed significant decreases and increases in gamma-band power in schizophrenia patients relative to healthy controls, which emerged both early and late over both sensory and frontal regions in response to unisensory and multisensory stimuli. Unisensory gamma-band power predicted multisensory gamma-band power differently by group. Furthermore, gamma-band power in these regions predicted performance in select measures of the Measurement and Treatment Research to Improve Cognition in Schizophrenia (MATRICS) test battery differently by group. These results reveal a unique pattern of task-related gamma-band power in schizophrenia patients relative to controls that may indicate reduced inhibition in combination with impaired oscillatory mechanisms in patients with schizophrenia.

The neural bases of crossmodal object recognition in non-human primates and rodents: a review

The ability to integrate information from different sensory modalities to form unique multisensory object representations is a highly adaptive cognitive function. Surprisingly, non-human animal studies of the neural substrates of this form of multisensory integration have been somewhat sparse until very recently, and this may be due in part to a relative paucity of viable testing methods. Here we review the historical development and use of various "crossmodal" cognition tasks for non-human primates and rodents, focusing on tests of "crossmodal object recognition", the ability to recognize an object across sensory modalities. Such procedures have great potential to elucidate the cognitive and neural bases of object representation as it pertains to perception and memory. Indeed, these studies have revealed roles in crossmodal cognition for various brain regions (e.g., prefrontal and temporal cortices) and neurochemical systems (e.g., acetylcholine). A recent increase in behavioral and physiological studies of crossmodal cognition in rodents augurs well for the future of this research area, which should provide essential information about the basic mechanisms of object representation in the brain, in addition to fostering a better understanding of the causes of, and potential treatments for, cognitive deficits in human diseases characterized by atypical multisensory integration.

Crossmodal integration of conspecific vocalizations in rhesus macaques

Crossmodal integration of audio/visual information is vital for recognition, interpretation and appropriate reaction to social signals. Here we examined how rhesus macaques process bimodal species-specific vocalizations by eye tracking, using an unconstrained preferential looking paradigm. Six adult rhesus monkeys (3M, 3F) were presented two side-by-side videos of unknown male conspecifics emitting different vocalizations, accompanied by the audio signal corresponding to one of the videos. The percentage of time animals looked to each video was used to assess crossmodal integration ability and the percentages of time spent looking at each of the six a priori ROIs (eyes, mouth, and rest of each video) were used to characterize scanning patterns. Animals looked more to the congruent video, confirming reports that rhesus monkeys spontaneously integrate conspecific vocalizations. Scanning patterns showed that monkeys preferentially attended to the eyes and mouth of the stimuli, with subtle differences between males and females such that females showed a tendency to differentiate the eye and mouth regions more than males. These results were similar to studies in humans indicating that when asked to assess emotion-related aspects of visual speech, people preferentially attend to the eyes. Thus, the tendency for female monkeys to show a greater differentiation between the eye and mouth regions than males may indicate that female monkeys were slightly more sensitive to the socio-emotional content of complex signals than male monkeys. The current results emphasize the importance of considering both the sex of the observer and individual variability in passive viewing behavior in nonhuman primate research.

Attention and multisensory integration of emotions in schizophrenia

The impairment of multisensory integration in schizophrenia is often explained by deficits of attentional selection. Emotion perception, however, does not always depend on attention because affective stimuli can capture attention automatically. In our study, we specify the role of attention in the multisensory perception of emotional stimuli in schizophrenia. We evaluated attention by interference between conflicting auditory and visual information in two multisensory paradigms in patients with schizophrenia and healthy participants. In the first paradigm, interference occurred between physical features of the dynamic auditory and visual stimuli. In the second paradigm, interference occurred between the emotional content of the auditory and visual stimuli, namely fearful and sad emotions. In patients with schizophrenia, the interference effect was observed in both paradigms. In contrast, in healthy participants, the interference occurred in the emotional paradigm only. These findings indicate that the information leakage between different modalities in patients with schizophrenia occurs at the perceptual level, which is intact in healthy participants. However, healthy participants can have problems with the separation of fearful and sad emotions similar to those of patients with schizophrenia.

Facial reactions in response to dynamic emotional stimuli in different modalities in patients suffering from schizophrenia: a behavioral and EMG study

Emotional facial expression is an important low-level mechanism contributing to the experience of empathy, thereby lying at the core of social interaction. Schizophrenia is associated with pervasive social cognitive impairments, including emotional processing of facial expressions. In this study we test a novel paradigm in order to investigate the evaluation of the emotional content of perceived emotions presented through dynamic expressive stimuli, facial mimicry evoked by the same stimuli, and their functional relation. Fifteen healthy controls and 15 patients diagnosed with schizophrenia were presented with stimuli portraying positive (laugh), negative (cry) and neutral (control) emotional stimuli in visual, auditory modalities in isolation, and congruently or incongruently associated. Participants where requested to recognize and quantitatively rate the emotional value of the perceived stimuli, while electromyographic activity of Corrugator and Zygomaticus muscles was recorded. All participants correctly judged the perceived emotional stimuli and prioritized the visual over the auditory modality in identifying the emotion when they were incongruently associated (Audio-Visual Incongruent condition). The neutral emotional stimuli did not evoke any muscle responses and were judged by all participants as emotionally neutral. Control group responded with rapid and congruent mimicry to emotional stimuli, and in Incongruent condition muscle responses were driven by what participants saw rather than by what they heard. Patient group showed a similar pattern only with respect to negative stimuli, whereas showed a lack of or a non-specific Zygomaticus response when positive stimuli were presented. Finally, we found that only patients with reduced facial mimicry (Internalizers) judged both positive and negative emotions as significantly more neutral than controls. The relevance of these findings for studying emotional deficits in schizophrenia is discussed.

Sensory processing, neurocognition, and social cognition in schizophrenia: towards a cohesive cognitive model

Schizophrenia research has identified deficits in neurocognition, social cognition, and sensory processing. Because a cohesive model of "disturbed cognitive machinery" is currently lacking, we built a conceptual model to integrate neurocognition, social cognition, and sensory processing. In a cross-sectional study, the cognitive performance of participants was measured. In accordance with the Schedules for Clinical Assessment in Neuropsychiatry, the participants were assigned to either the schizophrenia group or the non-schizophrenic psychosis group. Exclusion criteria included substance abuse, serious somatic/neurological illness, and perceptual handicap. The male/female ratio, educational level, and handedness did not differ significantly between the groups. The data were analyzed using structural equation modeling. Based upon the results of all possible pairwise models correlating neurocognition, social cognition, and sensory processing, three omnibus models were analyzed. A statistical analysis of a pairwise model-fit (χ(2), CFI, and RMSEA statistics) revealed poor interrelatedness between sensory processing and neurocognition in schizophrenia patients compared with healthy control participants. The omnibus model that predicted disintegration between sensory processing and neurocognition was statistically confirmed as superior for the schizophrenia group (χ(2)(53) of 56.62, p=0.341, RMSEA=0.04, CFI=0.95). In healthy participants, the model predicting maximal interrelatedness between sensory processing/neurocognition and neurocognition/social cognition gave the best fit (χ(2)(52) of 53.74, p=0.408, RMSEA=0.03, CFI=0.97). The performance of the patients with non-schizophrenic psychosis fell between the schizophrenia patients and control participants. These findings suggest increasing separation between sensory processing and neurocognition along the continuum from mental health to schizophrenia. Our results support a conceptual model that posits disintegration between sensory processing of social stimuli and neurocognitive processing.

Prosody recognition and audiovisual emotion matching in schizophrenia: the contribution of cognition and psychopathology

This study aimed to evaluate the ability to decode emotion in the auditory and audiovisual modality in a group of patients with schizophrenia, and to explore the role of cognition and psychopathology in affecting these emotion recognition abilities. Ninety-four outpatients in a stable phase and 51 healthy subjects were recruited. Patients were assessed through a psychiatric evaluation and a wide neuropsychological battery. All subjects completed the comprehensive affect testing system (CATS), a group of computerized tests designed to evaluate emotion perception abilities. With respect to the controls, patients were not impaired in the CATS tasks involving discrimination of nonemotional prosody, naming of emotional stimuli expressed by voice and judging the emotional content of a sentence, whereas they showed a specific impairment in decoding emotion in a conflicting auditory condition and in the multichannel modality. Prosody impairment was affected by executive functions, attention and negative symptoms, while deficit in multisensory emotion recognition was affected by executive functions and negative symptoms. These emotion recognition deficits, rather than being associated purely with emotion perception disturbances in schizophrenia, are affected by core symptoms of the illness.

Cocaine users manifest impaired prosodic and cross-modal emotion processing

BACKGROUND

A small number of previous studies have provided evidence that cocaine users (CU) exhibit impairments in complex social cognition tasks, while the more basic facial emotion recognition is widely unaffected. However, prosody and cross-modal emotion processing has not been systematically investigated in CU so far. Therefore, the aim of the present study was to assess complex multisensory emotion processing in CU in comparison to controls and to examine a potential association with drug use patterns.

METHOD

The abbreviated version of the comprehensive affect testing system (CATS-A) was used to measure emotion perception across the three channels of facial affect, prosody, and semantic content in 58 CU and 48 healthy control (HC) subjects who were matched for age, sex, verbal intelligence, and years of education.

RESULTS

CU had significantly lower scores than controls in the quotient scales of "emotion recognition" and "prosody recognition" and the subtests "conflicting prosody/meaning - attend to prosody" and "match emotional prosody to emotional face" either requiring to attend to prosody or to integrate cross-modal information. In contrast, no group difference emerged for the "affect recognition quotient." Cumulative cocaine doses and duration of cocaine use correlated negatively with emotion processing.

CONCLUSION

CU show impaired cross-modal integration of different emotion processing channels particularly with regard to prosody, whereas more basic aspects of emotion processing such as facial affect perception are comparable to the performance of HC.

Regulation of Ambient GABA Levels by Neuron-Glia Signaling for Reliable Perception of Multisensory Events

Activities of sensory-specific cortices are known to be suppressed when presented with a different sensory modality stimulus. This is referred to as cross-modal inhibition, for which the conventional synaptic mechanism is unlikely to work. Interestingly, the cross-modal inhibition could be eliminated when presented with multisensory stimuli arising from the same event. To elucidate the underlying neuronal mechanism of cross-modal inhibition and understand its significance for multisensory information processing, we simulated a neural network model. Principal cell to and GABAergic interneuron to glial cell projections were assumed between and within lower-order unimodal networks (X and Y), respectively. Cross-modality stimulation of Y network activated its principal cells, which then depolarized glial cells of X network. This let transporters on the glial cells export GABA molecules into the extracellular space and increased a level of ambient (extrasynaptic) GABA. The ambient GABA molecules were accepted by extrasynaptic GABAa receptors and tonically inhibited principal cells of the X network. Cross-modal inhibition took place in a nonsynaptic manner. Identical modality stimulation of X network activated its principal cells, which then activated interneurons and hyperpolarized glial cells of the X network. This let their transporters import (remove) GABA molecules from the extracellular space and reduced tonic inhibitory current in principal cells, thereby improving their gain function. Top-down signals from a higher-order multimodal network (M) contributed to elimination of the cross-modal inhibition when presented with multisensory stimuli that arose from the same event. Tuning into the multisensory event deteriorated if the cross-modal inhibitory mechanism did not work. We suggest that neuron-glia signaling may regulate local ambient GABA levels in order to coordinate cross-modal inhibition and improve neuronal gain function, thereby achieving reliable perception of multisensory events.

Multisensory integration and attention in autism spectrum disorder: evidence from event-related potentials

Successful integration of various simultaneously perceived perceptual signals is crucial for social behavior. Recent findings indicate that this multisensory integration (MSI) can be modulated by attention. Theories of Autism Spectrum Disorders (ASDs) suggest that MSI is affected in this population while it remains unclear to what extent this is related to impairments in attentional capacity. In the present study Event-related potentials (ERPs) following emotionally congruent and incongruent face-voice pairs were measured in 23 high-functioning, adult ASD individuals and 24 age- and IQ-matched controls. MSI was studied while the attention of the participants was manipulated. ERPs were measured at typical auditory and visual processing peaks, namely, P2 and N170. While controls showed MSI during divided attention and easy selective attention tasks, individuals with ASD showed MSI during easy selective attention tasks only. It was concluded that individuals with ASD are able to process multisensory emotional stimuli, but this is differently modulated by attention mechanisms in these participants, especially those associated with divided attention. This atypical interaction between attention and MSI is also relevant to treatment strategies, with training of multisensory attentional control possibly being more beneficial than conventional sensory integration therapy.

Preattentive processing of audio-visual emotional signals

Previous research has shown that redundant information in faces and voices leads to faster emotional categorization compared to incongruent emotional information even when attending to only one modality. The aim of the present study was to test whether these crossmodal effects are predominantly due to a response conflict rather than interference at earlier, e.g. perceptual processing stages. In Experiment 1, participants had to categorize the valence and rate the intensity of happy, sad, angry and neutral unimodal or bimodal face-voice stimuli. They were asked to rate either the facial or vocal expression and ignore the emotion expressed in the other modality. Participants responded faster and more precisely to emotionally congruent compared to incongruent face-voice pairs in both the Attend Face and in the Attend Voice condition. Moreover, when attending to faces, emotionally congruent bimodal stimuli were more efficiently processed than unimodal visual stimuli. To study the role of a possible response conflict, Experiment 2 used a modified paradigm in which emotional and response conflicts were disentangled. Incongruency effects were significant even in the absence of response conflicts. The results suggest that emotional signals available through different sensory channels are automatically combined prior to response selection.