Eluding the illusion? Schizophrenia, dopamine and the McGurk effect

Abnormal connectivity and activation during audiovisual speech perception in schizophrenia

The unconscious integration of vocal and facial cues during speech perception facilitates face-to-face communication. Recent studies have provided substantial behavioural evidence concerning impairments in audiovisual (AV) speech perception in schizophrenia. However, the specific neurophysiological mechanism underlying these deficits remains unknown. Here, we investigated activities and connectivities centered on the auditory cortex during AV speech perception in schizophrenia. Using magnetoencephalography, we recorded and analysed event-related fields in response to auditory (A: voice), visual (V: face) and AV (voice-face) stimuli in 23 schizophrenia patients (13 males) and 22 healthy controls (13 males). The functional connectivity associated with the subadditive response to AV stimulus (i.e., [AV] < [A] + [V]) was also compared between the two groups. Within the healthy control group, [AV] activity was smaller than the sum of [A] and [V] at latencies of approximately 100 ms in the posterior ramus of the lateral sulcus in only the left hemisphere, demonstrating a subadditive N1m effect. Conversely, the schizophrenia group did not show such a subadditive response. Furthermore, weaker functional connectivity from the posterior ramus of the lateral sulcus of the left hemisphere to the fusiform gyrus of the right hemisphere was observed in schizophrenia. Notably, this weakened connectivity was associated with the severity of negative symptoms. These results demonstrate abnormalities in connectivity between speech- and face-related cortical areas in schizophrenia. This aberrant subadditive response and connectivity deficits for integrating speech and facial information may be the neural basis of social communication dysfunctions in schizophrenia.

Metacognitive awareness in the sound-induced flash illusion

Hundreds (if not thousands) of multisensory studies provide evidence that the human brain can integrate temporally and spatially discrepant stimuli from distinct modalities into a singular event. This process of multisensory integration is usually portrayed in the scientific literature as contributing to our integrated, coherent perceptual reality. However, missing from this account is an answer to a simple question: how do confidence judgements compare between multisensory information that is integrated across multiple sources, and multisensory information that comes from a single, congruent source in the environment? In this paper, we use the sound-induced flash illusion to investigate if confidence judgements are similar across multisensory conditions when the numbers of auditory and visual events are the same, and the numbers of auditory and visual events are different. Results showed that congruent audiovisual stimuli produced higher confidence than incongruent audiovisual stimuli, even when the perceptual report was matched across the two conditions. Integrating these behavioural findings with recent neuroimaging and theoretical work, we discuss the role that prefrontal cortex may play in metacognition, multisensory causal inference and sensory source monitoring in general. This article is part of the theme issue 'Decision and control processes in multisensory perception'.

Deficient Audiovisual Speech Perception in Schizophrenia: An ERP Study

In everyday verbal communication, auditory speech perception is often disturbed by background noise. Especially in disadvantageous hearing conditions, additional visual articulatory information (e.g., lip movement) can positively contribute to speech comprehension. Patients with schizophrenia (SZs) demonstrate an aberrant ability to integrate visual and auditory sensory input during speech perception. Current findings about underlying neural mechanisms of this deficit are inconsistent. Particularly and despite the importance of early sensory processing in speech perception, very few studies have addressed these processes in SZs. Thus, in the present study, we examined 20 adult subjects with SZ and 21 healthy controls (HCs) while presenting audiovisual spoken words (disyllabic nouns) either superimposed by white noise (-12 dB signal-to-noise ratio) or not. In addition to behavioral data, event-related brain potentials (ERPs) were recorded. Our results demonstrate reduced speech comprehension for SZs compared to HCs under noisy conditions. Moreover, we found altered N1 amplitudes in SZ during speech perception, while P2 amplitudes and the N1-P2 complex were similar to HCs, indicating that there may be disturbances in multimodal speech perception at an early stage of processing, which may be due to deficits in auditory speech perception. Moreover, a positive relationship between fronto-central N1 amplitudes and the positive subscale of the Positive and Negative Syndrome Scale (PANSS) has been observed.

Intrinsic neural timescales in autism spectrum disorder and schizophrenia. A replication and direct comparison study

Intrinsic neural timescales (INT) reflect the duration for which brain areas store information. A posterior-anterior hierarchy of increasingly longer INT has been revealed in both typically developed individuals (TD), as well as persons diagnosed with autism spectrum disorder (ASD) and schizophrenia (SZ), though INT are, overall, shorter in both patient groups. In the present study, we aimed to replicate previously reported group differences by comparing INT of TD to ASD and SZ. We partially replicated the previously reported result, showing reduced INT in the left lateral occipital gyrus and the right post-central gyrus in SZ compared to TD. We also directly compared the INT of the two patient groups and found that these same two areas show significantly reduced INT in SZ compared to ASD. Previously reported correlations between INT and symptom severity were not replicated in the current project. Our findings serve to circumscribe the brain areas that can potentially play a determinant role in observed sensory peculiarities in ASD and SZ.

The promise of layer-specific neuroimaging for testing predictive coding theories of psychosis

Predictive coding potentially provides an explanatory model for understanding the neurocognitive mechanisms of psychosis. It proposes that cognitive processes, such as perception and inference, are implemented by a hierarchical system, with the influence of each level being a function of the estimated precision of beliefs at that level. However, predictive coding models of psychosis are insufficiently constrained-any phenomenon can be explained in multiple ways by postulating different changes to precision at different levels of processing. One reason for the lack of constraint in these models is that the core processes are thought to be implemented by the function of specific cortical layers, and the technology to measure layer specific neural activity in humans has until recently been lacking. As a result, our ability to constrain the models with empirical data has been limited. In this review we provide a brief overview of predictive processing models of psychosis and then describe the potential for newly developed, layer specific neuroimaging techniques to test and thus constrain these models. We conclude by discussing the most promising avenues for this research as well as the technical and conceptual challenges which may limit its application.

Direct eye gaze enhances the ventriloquism effect

The “ventriloquism effect” describes an illusory phenomenon where the perceived location of an auditory stimulus is pulled toward the location of a visual stimulus. Ventriloquists use this phenomenon to create an illusion where an inanimate puppet is perceived to speak. Ventriloquists use the expression and suppression of their own and the puppet’s mouth movements as well the direction of their respective eye gaze to maximize the illusion. While the puppet’s often exaggerated mouth movements have been demonstrated to enhance the ventriloquism effect, the contribution of direct eye gaze remains unknown. In Experiment 1, participants viewed an image of a person’s face while hearing a temporally synchronous recording of a voice originating from different locations on the azimuthal plane. The eyes of the facial stimuli were either looking directly at participants or were closed. Participants were more likely to misperceive the location of a range of voice locations as coming from a central position when the eye gaze of the facial stimuli were directed toward them. Thus, direct gaze enhances the ventriloquist effect by attracting participants’ perception of the voice locations toward the location of the face. In an exploratory analysis, we furthermore found no evidence for an other-race effect between White vs Asian listeners. In Experiment 2, we replicated the effect of direct eye gaze on the ventriloquism effect, also showing that faces per se attract perceived sound locations compared with audio-only sound localization. Showing a modulation of the ventriloquism effect by socially-salient eye gaze information thus adds to previous findings reporting top-down influences on this effect.

Effects of pitch and musical sounds on body-representations when moving with sound

The effects of music on bodily movement and feelings, such as when people are dancing or engaged in physical activity, are well-documented—people may move in response to the sound cues, feel powerful, less tired. How sounds and bodily movements relate to create such effects? Here we deconstruct the problem and investigate how different auditory features affect people’s body-representation and feelings even when paired with the same movement. In three experiments, participants executed a simple arm raise synchronised with changing pitch in simple tones (Experiment 1), rich musical sounds (Experiment 2) and within different frequency ranges (Experiment 3), while we recorded indirect and direct measures on their movement, body-representations and feelings. Changes in pitch influenced people’s general emotional state as well as the various bodily dimensions investigated—movement, proprioceptive awareness and feelings about one’s body and movement. Adding harmonic content amplified the differences between ascending and descending sounds, while shifting the absolute frequency range had a general effect on movement amplitude, bodily feelings and emotional state. These results provide new insights in the role of auditory and musical features in dance and exercise, and have implications for the design of sound-based applications supporting movement expression, physical activity, or rehabilitation.

Schizotypal personality traits and multisensory integration: An investigation using the McGurk effect

Multisensory integration, the process by which sensory information from different sensory modalities are bound together, is hypothesized to contribute to perceptual symptomatology in schizophrenia, in whom multisensory integration differences have been consistently found. Evidence is emerging that these differences extend across the schizophrenia spectrum, including individuals in the general population with higher levels of schizotypal traits. In the current study, we used the McGurk task as a measure of multisensory integration. We measured schizotypal traits using the Schizotypal Personality Questionnaire (SPQ), hypothesizing that higher levels of schizotypal traits, specifically Unusual Perceptual Experiences and Odd Speech subscales, would be associated with decreased multisensory integration of speech. Surprisingly, Unusual Perceptual Experiences were not associated with multisensory integration. However, Odd Speech was associated with multisensory integration, and this association extended more broadly across the Disorganized factor of the SPQ, including Odd or Eccentric Behaviour. Individuals with higher levels of Odd or Eccentric Behaviour scores also demonstrated poorer lip-reading abilities, which partially explained performance in the McGurk task. This suggests that aberrant perceptual processes affecting individuals across the schizophrenia spectrum may relate to disorganized symptomatology.

Dopamine Precursor Depletion in Healthy Volunteers Impairs Processing of Duration but Not Temporal Order

Studies in animals and humans have implicated the neurotransmitter dopamine in duration processing. However, very few studies have examined dopamine's involvement in other forms of temporal processing such as temporal order judgments. In a randomized within-subject placebo-controlled design, we used acute phenylalanine/tyrosine depletion (APTD) to reduce availability of the dopamine precursors tyrosine and phenylalanine in healthy human volunteers. As compared to a nutritionally balanced drink, APTD significantly impaired the ability to accurately reproduce interval duration in a temporal reproduction task. In addition, and confirming previous findings, the direction of error differed as a function of individual differences in underlying dopamine function. Specifically, APTD caused participants with low baseline dopamine precursor availability to overestimate the elapse of time, whereas those with high dopamine availability underestimated time. In contrast to these effects on duration processing, there were no significant effects of APTD on the accuracy of discriminating the temporal order of visual stimuli. This pattern of results does not simply represent an effect of APTD on motor, rather than perceptual, measures of timing because APTD had no effect on participants' ability to use temporal cues to speed RT. Our results demonstrate, for the first time in healthy volunteers, a dopaminergic dissociation in judging metrical (duration) versus ordinal (temporal order) aspects of time.

Influence of prior beliefs on perception in early psychosis: Effects of illness stage and hierarchical level of belief

Alterations in the balance between prior expectations and sensory evidence may account for faulty perceptions and inferences leading to psychosis. However, uncertainties remain about the nature of altered prior expectations and the degree to which they vary with the emergence of psychosis. We explored how expectations arising at two different levels—cognitive and perceptual—influenced processing of sensory information and whether relative influences of higher- and lower-level priors differed across people with prodromal symptoms and those with psychotic illness. In two complementary auditory perception experiments, 91 participants (30 with first-episode psychosis, 29 at clinical risk for psychosis, and 32 controls) were required to decipher a phoneme within ambiguous auditory input. Expectations were generated in two ways: an accompanying visual input of lip movements observed during auditory presentation or through written presentation of a phoneme provided prior to auditory presentation. We determined how these different types of information shaped auditory perceptual experience, how this was altered across the prodromal and established phases of psychosis, and how this relates to cingulate glutamate levels assessed by magnetic resonance spectroscopy. The psychosis group relied more on high-level cognitive priors compared to both healthy controls and those at clinical risk for psychosis and relied more on low-level perceptual priors than the clinical risk group. The risk group was marginally less reliant on low-level perceptual priors than controls. The results are consistent with previous theory that influences of prior expectations in perceptions in psychosis differ according to level of prior and illness phase.

What we perceive and believe in any given moment will allow us to form expectations about what we will experience in the next. In psychosis, it is believed that the influence of these so-called perceptual and cognitive “prior” expectations on perception are altered, thereby giving rise to the symptoms seen in psychosis. However, research thus far has found mixed evidence, some suggesting an increase in the influence of priors and some finding a decrease. Here we test the hypothesis that perceptual and cognitive priors are differentially affected in individuals at risk for psychosis and individuals with a first episode of psychosis, thereby partially explaining the mixed findings in the literature. We indeed found evidence in favor of this hypothesis, finding weaker perceptual priors in individuals at risk but stronger cognitive priors in individuals with first-episode psychosis.

Racial bias in face perception is sensitive to instructions but not introspection

Faces with typically African features are perceived as darker than they really are. We investigated how early in processing the bias emerges, whether participants are aware of it, and whether it can be altered by explicit instructions. We presented pairs of faces sequentially, manipulated the luminance and morphological features of each, and asked participants which was lighter, and how confident they were in their responses. In Experiment 1, pre-response mouse cursor trajectories showed that morphology affected motor output just as early as luminance did. Furthermore, participants were not slower to respond or less confident when morphological cues drove them to give a response that conflicted with the actual luminance of the faces. However, Experiment 2 showed that participants could be instructed to reduce their reliance on morphology, even at early stages of processing. All stimuli used, code to run the experiments reported, raw data, and analyses scripts and their outputs can be found at https://osf.io/brssn.

Delusions of control in schizophrenia: Resistant to the mind's best trick?

The existence of a free will is fiercely debated in neuroscience and philosophy. The debate has great impact on society and our self-understanding as human beings. Behavioral and electrophysiological data have challenged the intuitive assumption that human behavior is the result of conscious intentions. This notion has important implications for delusions of control in schizophrenia, where patients experience bodily movements as not being controlled by themselves. Current theories explain control delusions as a deficit to perceive certain aspects of motor control, but many findings are inconsistent with this idea. Here, an alternative view is proposed: Control delusions might be triggered by an even more veridical perception of the temporal order of intentions and actions. This hypothesis is supported by evidence that (a) conscious intentions in healthy subjects are often based on retrospective inferences, (b) temporal recalibrations of conscious percepts occur in healthy subjects and are disturbed in schizophrenia and (c) basic perceptual functions of schizophrenic patients are less influenced by expectations and therefore they can sometimes be closer to physical reality than those of healthy subjects.

Multisensory cortical processing and dysfunction across the neuropsychiatric spectrum

Sensory processing is affected in multiple neuropsychiatric disorders like schizophrenia and autism spectrum disorders. Genetic and environmental factors guide the formation and fine-tuning of brain circuitry necessary to receive, organize, and respond to sensory input in order to behave in a meaningful and consistent manner. During certain developmental stages the brain is sensitive to intrinsic and external factors. For example, disturbed expression levels of certain risk genes during critical neurodevelopmental periods may lead to exaggerated brain plasticity processes within the sensory circuits, and sensory stimulation immediately after birth contributes to fine-tuning of these circuits. Here, the neurodevelopmental trajectory of sensory circuit development will be described and related to some example risk gene mutations that are found in neuropsychiatric disorders. Subsequently, the flow of sensory information through these circuits and the relationship to synaptic plasticity will be described. Research focusing on the combined analyses of neural circuit development and functioning are necessary to expand our understanding of sensory processing and behavioral deficits that are relevant across the neuropsychiatric spectrum.

Forty Years After Hearing Lips and Seeing Voices: the McGurk Effect Revisited

Since its discovery 40 years ago, the McGurk illusion has been usually cited as a prototypical paradigmatic case of multisensory binding in humans, and has been extensively used in speech perception studies as a proxy measure for audiovisual integration mechanisms. Despite the well-established practice of using the McGurk illusion as a tool for studying the mechanisms underlying audiovisual speech integration, the magnitude of the illusion varies enormously across studies. Furthermore, the processing of McGurk stimuli differs from congruent audiovisual processing at both phenomenological and neural levels. This questions the suitability of this illusion as a tool to quantify the necessary and sufficient conditions under which audiovisual integration occurs in natural conditions. In this paper, we review some of the practical and theoretical issues related to the use of the McGurk illusion as an experimental paradigm. We believe that, without a richer understanding of the mechanisms involved in the processing of the McGurk effect, experimenters should be really cautious when generalizing data generated by McGurk stimuli to matching audiovisual speech events.

Neural Mechanisms Underlying Cross-Modal Phonetic Encoding

Audiovisual (AV) integration is essential for speech comprehension, especially in adverse listening situations. Divergent, but not mutually exclusive, theories have been proposed to explain the neural mechanisms underlying AV integration. One theory advocates that this process occurs via interactions between the auditory and visual cortices, as opposed to fusion of AV percepts in a multisensory integrator. Building upon this idea, we proposed that AV integration in spoken language reflects visually induced weighting of phonetic representations at the auditory cortex. EEG was recorded while male and female human subjects watched and listened to videos of a speaker uttering consonant vowel (CV) syllables /ba/ and /fa/, presented in Auditory-only, AV congruent or incongruent contexts. Subjects reported whether they heard /ba/ or /fa/. We hypothesized that vision alters phonetic encoding by dynamically weighting which phonetic representation in the auditory cortex is strengthened or weakened. That is, when subjects are presented with visual /fa/ and acoustic /ba/ and hear /fa/ (illusion-fa), the visual input strengthens the weighting of the phone /f/ representation. When subjects are presented with visual /ba/ and acoustic /fa/ and hear /ba/ (illusion-ba), the visual input weakens the weighting of the phone /f/ representation. Indeed, we found an enlarged N1 auditory evoked potential when subjects perceived illusion-ba, and a reduced N1 when they perceived illusion-fa, mirroring the N1 behavior for /ba/ and /fa/ in Auditory-only settings. These effects were especially pronounced in individuals with more robust illusory perception. These findings provide evidence that visual speech modifies phonetic encoding at the auditory cortex.SIGNIFICANCE STATEMENT The current study presents evidence that audiovisual integration in spoken language occurs when one modality (vision) acts on representations of a second modality (audition). Using the McGurk illusion, we show that visual context primes phonetic representations at the auditory cortex, altering the auditory percept, evidenced by changes in the N1 auditory evoked potential. This finding reinforces the theory that audiovisual integration occurs via visual networks influencing phonetic representations in the auditory cortex. We believe that this will lead to the generation of new hypotheses regarding cross-modal mapping, particularly whether it occurs via direct or indirect routes (e.g., via a multisensory mediator).

A Predictive Coding Account of Psychotic Symptoms in Autism Spectrum Disorder

The co-occurrence of psychotic and autism spectrum disorder (ASD) symptoms represents an important clinical challenge. Here we consider this problem in the context of a computational psychiatry approach that has been applied to both conditions-predictive coding. Some symptoms of schizophrenia have been explained in terms of a failure of top-down predictions or an enhanced weighting of bottom-up prediction errors. Likewise, autism has been explained in terms of similar perturbations. We suggest that this theoretical overlap may explain overlapping symptomatology. Experimental evidence highlights meaningful distinctions and consistencies between these disorders. We hypothesize individuals with ASD may experience some degree of delusions without the presence of any additional impairment, but that hallucinations are likely indicative of a distinct process.

A Predictive Coding Account of Psychotic Symptoms in Autism Spectrum Disorder

The co-occurrence of psychotic and autism spectrum disorder (ASD) symptoms represents an important clinical challenge. Here we consider this problem in the context of a computational psychiatry approach that has been applied to both conditions-predictive coding. Some symptoms of schizophrenia have been explained in terms of a failure of top-down predictions or an enhanced weighting of bottom-up prediction errors. Likewise, autism has been explained in terms of similar perturbations. We suggest that this theoretical overlap may explain overlapping symptomatology. Experimental evidence highlights meaningful distinctions and consistencies between these disorders. We hypothesize individuals with ASD may experience some degree of delusions without the presence of any additional impairment, but that hallucinations are likely indicative of a distinct process.

Reduced susceptibility to the sound-induced flash fusion illusion in schizophrenia

Schizophrenia is characterised by the presence of abnormal complex sensory perceptual experiences. Such experiences could arise as a consequence of dysfunctional multisensory integration. We used the sound-induced flash illusion paradigm, which probes audiovisual integration using elementary visual and auditory cues, in a sample of individuals with schizophrenia (n=40) and matched controls (n=22). Signal detection theory analyses were performed to characterise patients' and controls' sensitivity in distinguishing 1 and 2 flashes under varying auditory conditions. Both groups experienced significant fission illusions (whereby one visual flash, accompanied by two auditory beeps, is misperceived as two flashes) and fusion illusions (whereby two flashes, accompanied by one beep, are perceived as one flash). Patients showed significantly lower fusion illusion rates compared to HC, while the fission illusion occurred similarly frequently in both groups. However, using an SDT approach, we compared illusion conditions with unimodal visual conditions, and found that illusory visual perception was overall more strongly influenced by auditory input in HC compared to patients for both illusions. This suggests that multisensory integration may be impaired on a low perceptual level in SZ.

Skilled musicians are not subject to the McGurk effect

The McGurk effect is a compelling illusion in which humans auditorily perceive mismatched audiovisual speech as a completely different syllable. In this study evidences are provided that professional musicians are not subject to this illusion, possibly because of their finer auditory or attentional abilities. 80 healthy age-matched graduate students volunteered to the study. 40 were musicians of Brescia Luca Marenzio Conservatory of Music with at least 8–13 years of musical academic studies. /la/, /da/, /ta/, /ga/, /ka/, /na/, /ba/, /pa/ phonemes were presented to participants in audiovisual congruent and incongruent conditions, or in unimodal (only visual or only auditory) conditions while engaged in syllable recognition tasks. Overall musicians showed no significant McGurk effect for any of the phonemes. Controls showed a marked McGurk effect for several phonemes (including alveolar-nasal, velar-occlusive and bilabial ones). The results indicate that the early and intensive musical training might affect the way the auditory cortex process phonetic information.

Parallel linear dynamic models can mimic the McGurk effect in clinical populations

One of the most common examples of audiovisual speech integration is the McGurk effect. As an example, an auditory syllable /ba/ recorded over incongruent lip movements that produce "ga" typically causes listeners to hear "da". This report hypothesizes reasons why certain clinical and listeners who are hard of hearing might be more susceptible to visual influence. Conversely, we also examine why other listeners appear less susceptible to the McGurk effect (i.e., they report hearing just the auditory stimulus without being influenced by the visual). Such explanations are accompanied by a mechanistic explanation of integration phenomena including visual inhibition of auditory information, or slower rate of accumulation of inputs. First, simulations of a linear dynamic parallel interactive model were instantiated using inhibition and facilitation to examine potential mechanisms underlying integration. In a second set of simulations, we systematically manipulated the inhibition parameter values to model data obtained from listeners with autism spectrum disorder. In summary, we argue that cross-modal inhibition parameter values explain individual variability in McGurk perceptibility. Nonetheless, different mechanisms should continue to be explored in an effort to better understand current data patterns in the audiovisual integration literature.

Alpha-Band Oscillations Reflect Altered Multisensory Processing of the McGurk Illusion in Schizophrenia

The formation of coherent multisensory percepts requires integration of stimuli across the multiple senses. Patients with schizophrenia (ScZ) often experience a loss of coherent perception and hence, they might also show dysfunctional multisensory processing. In this high-density electroencephalography study, we investigated the neural signatures of the McGurk illusion, as a phenomenon of speech-specific multisensory processing. In the McGurk illusion lip movements are paired with incongruent auditory syllables, which can induce a fused percept. In ScZ patients and healthy controls we compared neural oscillations and event-related potentials (ERPs) to congruent audiovisual speech stimuli and McGurk illusion trials, where a visual /ga/ and an auditory /pa/ was often perceived as /ka/. There were no significant group differences in illusion rates. The EEG data analysis revealed larger short latency ERPs to McGurk illusion compared with congruent trials in controls. The reversed effect pattern was found in ScZ patients, indicating an early audiovisual processing deficit. Moreover, we observed stronger suppression of medio-central alpha-band power (8-10 Hz, 550-700 ms) in response to McGurk illusion compared with control trials in the control group. Again, the reversed pattern was found in SCZ patients. Moreover, within groups, alpha-band suppression was negatively correlated with the McGurk illusion rate in ScZ patients, while the correlation tended to be positive in controls. The topography of alpha-band effects indicated an involvement of auditory and/or frontal structures. Our study suggests that short latency ERPs and long latency alpha-band oscillations reflect abnormal multisensory processing of the McGurk illusion in ScZ.