Grapheme-color synesthetes show enhanced crossmodal processing between auditory and visual modalities

Differences in eccentricity for sound-induced flash illusion in four visual fields

A sound-induced flash illusion (SiFI) is a multisensory illusion dominated by auditory stimuli, in which the individual perceives that the number of visual flashes is equal to the number of auditory stimuli when visual flashes are presented along with an unequal number of auditory stimuli. Although the mechanisms underlying fission and fusion illusions have been documented, there is not yet a consensus on how they vary according to the different eccentricities. In the present study, by incorporating the classic SiFI paradigm into four different eccentricities, we aimed to investigate whether the SiFI varies under the different eccentricities. The results showed that the fission illusion varied significantly across the four eccentricities, with the perifovea (7°) and peripheral (11°) illusions being greater than the fovea and parafovea (3°) illusions. In contrast, the fusion illusion did not vary significantly across the four eccentricities. Our findings revealed that SiFI was affected by different visual fields and that there were differences between the fission and the fusion illusions. Furthermore, by examining the SiFI of eccentricity across visual fields, this study also suggests that bottom-up factors affect the SiFI.

Exploring Group Differences in the Crossmodal Correspondences

There has been a rapid growth of interest amongst researchers in the cross-modal correspondences in recent years. In part, this has resulted from the emerging realization of the important role that the correspondences can sometimes play in multisensory integration. In turn, this has led to an interest in the nature of any differences between individuals, or rather, between groups of individuals, in the strength and/or consensuality of cross-modal correspondences that may be observed in both neurotypically normal groups cross-culturally, developmentally, and across various special populations (including those who have lost a sense, as well as those with autistic tendencies). The hope is that our emerging understanding of such group differences may one day provide grounds for supporting the reality of the various different types of correspondence that have so far been proposed, namely structural, statistical, semantic, and hedonic (or emotionally mediated).

Audiovisual Simultaneity Judgements in Synaesthesia

Synaesthesia is a multimodal phenomenon in which the activation of one sensory modality leads to an involuntary additional experience in another sensory modality. To date, normal multisensory processing has hardly been investigated in synaesthetes. In the present study we examine processes of audiovisual separation in synaesthesia by using a simultaneity judgement task. Subjects were asked to indicate whether an acoustic and a visual stimulus occurred simultaneously or not. Stimulus onset asynchronies (SOA) as well as the temporal order of the stimuli were systematically varied. Our results demonstrate that synaesthetes are better in separating auditory and visual events than control subjects, but only when vision leads.

Synaesthesia and autism: Different developmental outcomes from overlapping mechanisms?

Synaesthesia, a mixing of the senses, is more common in individuals with autism. Here, we review the evidence for the association between synaesthesia and autism with regard to their genetic background, brain connectivity, perception, cognitive mechanisms and their contribution to exceptional talents. Currently, the overlap between synaesthesia and autism is established most convincingly at the level of alterations in sensory sensitivity and perception, with synaesthetes showing autism-like profiles of sensory sensitivity and a bias towards details in perception. Shared features may include a predominance of local over global connectivity in the brain. When autism and synaesthesia co-occur in the same individual, the chance of developing heightened cognitive and memory abilities is increased. We discuss how the same theoretical models could potentially explain both conditions. Given the evidence, we believe the phenotypical overlap between autism and synaesthesia has been established clearly enough to invite future research to confirm overlapping mechanisms.

Synesthesia as (Multimodal) Mental Imagery

It has been repeatedly suggested that synesthesia is intricately connected with unusual ways of exercising one's mental imagery, although it is not always entirely clear what the exact connection is. My aim is to show that all forms of synesthesia are forms of (often very different kinds of) mental imagery and, further, if we consider synesthesia to be a form of mental imagery, we get significant explanatory benefits, especially concerning less central cases of synesthesia where the inducer is not sensory stimulation.

Visual body form and orientation cues do not modulate visuo-tactile temporal integration

Body ownership relies on spatiotemporal correlations between multisensory signals and visual cues specifying oneself such as body form and orientation. The mechanism for the integration of bodily signals remains unclear. One approach to model multisensory integration that has been influential in the multisensory literature is Bayesian causal inference. This specifies that the brain integrates spatial and temporal signals coming from different modalities when it infers a common cause for inputs. As an example, the rubber hand illusion shows that visual form and orientation cues can promote the inference of a common cause (one's body) leading to spatial integration shown by a proprioceptive drift of the perceived location of the real hand towards the rubber hand. Recent studies investigating the effect of visual cues on temporal integration, however, have led to conflicting findings. These could be due to task differences, variation in ecological validity of stimuli and/or small samples. In this pre-registered study, we investigated the influence of visual information on temporal integration using a visuo-tactile temporal order judgement task with realistic stimuli and a sufficiently large sample determined by Bayesian analysis. Participants viewed videos of a touch being applied to plausible or implausible visual stimuli for one's hand (hand oriented plausibly, hand rotated 180 degrees, or a sponge) while also being touched at varying stimulus onset asynchronies. Participants judged which stimulus came first: viewed or felt touch. Results show that visual cues do not modulate visuo-tactile temporal order judgements. This is not in line with the idea that bodily signals indicating oneself influence the integration of multisensory signals in the temporal domain. The current study emphasises the importance of rigour in our methodologies and analyses to advance the understanding of how properties of multisensory events affect the encoding of temporal information in the brain.

A possible key role of vision in the development of schizophrenia

Based on a brief overview of the various aspects of schizophrenia reported by numerous studies, here we hypothesize that schizophrenia may originate (and in part be performed) from visual areas. In other words, it seems that a normal visual system or at least an evanescent visual perception may be an essential prerequisite for the development of schizophrenia as well as of various types of hallucinations. Our study focuses on auditory and visual hallucinations, as they are the most prominent features of schizophrenic hallucinations (and also the most studied types of hallucinations). Here, we evaluate the possible key role of the visual system in the development of schizophrenia.

Inducing synesthesia in non-synesthetes: Short-term visual deprivation facilitates auditory-evoked visual percepts

Sounds can modulate activity in visual cortex, facilitating the detection of visual targets. However, these sound-driven modulations are not thought to evoke conscious visual percepts in the general population. In individuals with synesthesia, however, multisensory interactions do lead to qualitatively different experiences such as sounds evoking flashes of light. Why, if multisensory interactions are present in all individuals, do only synesthetes experience abnormal qualia? Competing models differ in the time required for synesthetic experiences to emerge. The cross-activation model suggests synesthesia arises over months or years from the development of abnormal neural connections. Here we demonstrate that after ∼5 min of visual deprivation, sounds can evoke synesthesia-like percepts (vivid colors and Klüver form-constants) in ∼50% of non-synesthetes. These results challenge aspects of the cross-activation model and suggest that synesthesia exists as a latent feature in all individuals, manifesting when the balance of activity across the senses has been altered.

On the Timing of Signals in Multisensory Integration and Crossmodal Interactions: a Scoping Review

A scoping review was undertaken to explore research investigating early interactions and integration of auditory and visual stimuli in the human brain. The focus was on methods used to study low-level multisensory temporal processing using simple stimuli in humans, and how this research has informed our understanding of multisensory perception. The study of multisensory temporal processing probes how the relative timing between signals affects perception. Several tasks, illusions, computational models, and neuroimaging techniques were identified in the literature search. Research into early audiovisual temporal processing in special populations was also reviewed. Recent research has continued to provide support for early integration of crossmodal information. These early interactions can influence higher-level factors, and vice versa. Temporal relationships between auditory and visual stimuli influence multisensory perception, and likely play a substantial role in solving the 'correspondence problem' (how the brain determines which sensory signals belong together, and which should be segregated).

Gray Bananas and a Red Letter A - From Synesthetic Sensation to Memory Colors

Grapheme-color synesthesia is a condition in which objectively achromatic graphemes induce concurrent color experiences. While it was long thought that the colors emerge during perception, there is growing support for the view that colors are integral to synesthetes' cognitive representations of graphemes. In this work, we review evidence for two opposing theories positing either a perceptual or cognitive origin of concurrent colors: the cross-activation theory and the conceptual-mediation model. The review covers results on inducer and concurrent color processing as well as findings concerning the brain structure and grapheme-color mappings in synesthetes and trained mappings in nonsynesthetes. The results support different aspects of both theories. Finally, we discuss how research on memory colors could provide a new perspective in the debate about the level of processing at which the synesthetic colors occur.

Sounds from seeing silent motion: Who hears them, and what looks loudest?

Some people hear what they see: car indicator lights, flashing neon shop signs, and people's movements as they walk may all trigger an auditory sensation, which we call the visual-evoked auditory response (vEAR or 'visual ear'). We have conducted the first large-scale online survey (N > 4000) of this little-known phenomenon. We analysed the prevalence of vEAR, what induces it, and what other traits are associated with it. We assessed prevalence by asking whether respondents had previously experienced vEAR. Participants then rated silent videos for vividness of evoked auditory sensations, and answered additional trait questions. Prevalence appeared higher relative to other typical synaesthesias. Prior awareness and video ratings were associated with greater frequency of other synaesthesias, including flashes evoked by sounds, and musical imagery. Higher-rated videos often depicted meaningful events that predicted sounds (e.g., collisions). However, even videos containing abstract flickering or moving patterns could also elicit higher ratings, despite having no predictable association with sounds. Such videos had higher levels of raw 'motion energy' (ME), which we quantified using a simple computational model of motion processing in early visual cortex. Critically, only respondents reporting prior awareness of vEAR tended to show a positive correlation between video ratings and ME. This specific sensitivity to ME suggests that in vEAR, signals from visual motion processing may affect audition relatively directly without requiring higher-level interpretative processes. Our other findings challenge the popular assumption that individuals with synaesthesia are rare and have ideosyncratic patterns of brain hyper-connectivity. Instead, our findings of apparently high prevalence and broad associations with other synaesthesias and traits are jointly consistent with a common dependence on normal variations in physiological mechanisms of disinhibition or excitability of sensory brain areas and their functional connectivity. The prevalence of vEAR makes it easier to test such hypotheses further, and makes the results more relevant to understanding not only synaesthetic anomalies but also normal perception.

[i] is Lighter and More Greenish Than [o]: Intrinsic Association Between Vowel Sounds and Colors

It has recently been reported in the synesthesia literature that graphemes sharing the same phonetic feature tend to induce similar synesthetic colors. In the present study, we investigated whether phonetic properties are associated with colors in a specific manner among the general population, even when other visual and linguistic features of graphemes are removed. To test this hypothesis, we presented vowel sounds synthesized by systematically manipulating the position of the tongue body's center. Participants were asked to choose a color after hearing each sound. Results from the main experiment showed that lightness and chromaticity of matched colors exhibited systematic variations along the two axes of the position of the tongue body's center. Some non-random associations between vowel sounds and colors remained effective with pitch and intensity of the sounds equalized in the control experiment, which suggests that other acoustic factors such as inherent pitch of vowels cannot solely account for the current results. Taken together, these results imply that the association between phonetic features and colors is not random, and this synesthesia-like association is shared by people in the general population.

A deafening flash! Visual interference of auditory signal detection

In some people, visual stimulation evokes auditory sensations. How prevalent and how perceptually real is this? 22% of our neurotypical adult participants responded 'Yes' when asked whether they heard faint sounds accompanying flash stimuli, and showed significantly better ability to discriminate visual 'Morse-code' sequences. This benefit might arise from an ability to recode visual signals as sounds, thus taking advantage of superior temporal acuity of audition. In support of this, those who showed better visual relative to auditory sequence discrimination also had poorer auditory detection in the presence of uninformative visual flashes, though this was independent of awareness of visually-evoked sounds. Thus a visually-evoked auditory representation may occur subliminally and disrupt detection of real auditory signals. The frequent natural correlation between visual and auditory stimuli might explain the surprising prevalence of this phenomenon. Overall, our results suggest that learned correspondences between strongly correlated modalities may provide a precursor for some synaesthetic abilities.

Ignoring Irrelevant Information: Enhanced Intermodal Attention in Synaesthetes

Despite the fact that synaesthetes experience additional percepts during their inducer-concurrent associations that are often unrelated or irrelevant to their daily activities, they appear to be relatively unaffected by this potentially distracting information. This might suggest that synaesthetes are particularly good at ignoring irrelevant perceptual information coming from different sensory modalities. To investigate this hypothesis, the performance of a group of synaesthetes was compared to that of a matched non-synaesthete group in two different conflict tasks aimed at assessing participants' abilities to ignore irrelevant information. In order to match the sensory modality of the task-irrelevant distractors (vision) with participants' synaesthetic attentional filtering experience, we tested only synaesthetes experiencing at least one synaesthesia subtype triggering visual concurrents (e.g., grapheme-colour synaesthesia or sequence-space synaesthesia). Synaesthetes and controls performed a classic flanker task (FT) and a visuo-tactile cross-modal congruency task (CCT) in which they had to attend to tactile targets while ignoring visual distractors. While no differences were observed between synaesthetes and controls in the FT, synaesthetes showed reduced interference by the irrelevant distractors of the CCT. These findings provide the first direct evidence that synaesthetes might be more efficient than non-synaesthetes at dissociating conflicting information from different sensory modalities when the irrelevant modality correlates with their synaesthetic concurrent modality (here vision).

Multisensory Decisions: the Test of a Race Model, Its Logic, and Power

The use of separate multisensory signals is often beneficial. A prominent example is the speed-up of responses to two redundant signals relative to the components, which is known as the redundant signals effect (RSE). A convenient explanation for the effect is statistical facilitation, which is inherent in the basic architecture of race models (Raab, 1962,Trans. N. Y. Acad. Sci.24, 574–590). However, this class of models has been largely rejected in multisensory research, which we think results from an ambiguity in definitions and misinterpretations of the influential race model test (Miller, 1982,Cogn. Psychol.14, 247–279). To resolve these issues, we here discuss four main items. First, we clarify definitions and ask how successful models of perceptual decision making can be extended from uni- to multisensory decisions. Second, we review the race model test and emphasize elements leading to confusion with its interpretation. Third, we introduce a new approach to study the RSE. As a major change of direction, our working hypothesis is that the basic race model architecture is correct even if the race model test seems to suggest otherwise. Based on this approach, we argue that understanding the variability of responses is the key to understand the RSE. Finally, we highlight the critical role of model testability to advance research on multisensory decisions. Despite being largely rejected, it should be recognized that race models, as part of a broader class of parallel decision models, demonstrate, in fact, a convincing explanatory power in a range of experimental paradigms. To improve research consistency in the future, we conclude with a short checklist for RSE studies.

Cue Integration for Continuous and Categorical Dimensions by Synesthetes

For synesthetes, sensory or cognitive stimuli induce the perception of an additional sensory or cognitive stimulus. Grapheme-color synesthetes, for instance, consciously and consistently experience particular colors (e.g., fluorescent pink) when perceiving letters (e.g., u). As a phenomenon involving multiple stimuli within or across modalities, researchers have posited that synesthetes may integrate sensory cues differently than non-synesthetes. However, findings to date present mixed results concerning this hypothesis, with researchers reporting enhanced, depressed, or normal sensory integration for synesthetes. In this study we quantitatively evaluated the multisensory integration process of synesthetes and non-synesthetes using Bayesian principles, rather than employing multisensory illusions, to make inferences about the sensory integration process. In two studies we investigated synesthetes' sensory integration by comparing human behavior to that of an ideal observer. We found that synesthetes integrated cues for both continuous and categorical dimensions in a statistically optimal manner, matching the sensory integration behavior of controls. These findings suggest that synesthetes and controls utilize similar cue integration mechanisms, despite differences in how they perceive unimodal stimuli.

Synesthesia strengthens sound-symbolic cross-modal correspondences

Synesthesia is a phenomenon in which an experience in one domain is accompanied by an involuntary secondary experience in another, unrelated domain; in classical synesthesia, these associations are arbitrary and idiosyncratic. Cross-modal correspondences refer to universal associations between seemingly unrelated sensory features, e.g., auditory pitch and visual size. Some argue that these phenomena form a continuum, with classical synesthesia being an exaggeration of universal cross-modal correspondences, whereas others contend that the two are quite different, since cross-modal correspondences are non-arbitrary, non-idiosyncratic, and do not involve secondary experiences. Here, we used the implicit association test to compare synesthetes' and non-synesthetes' sensitivity to cross-modal correspondences. We tested the associations between auditory pitch and visual elevation, auditory pitch and visual size, and sound-symbolic correspondences between auditory pseudowords and visual shapes. Synesthetes were more sensitive than non-synesthetes to cross-modal correspondences involving sound-symbolic, but not low-level sensory, associations. We conclude that synesthesia heightens universally experienced cross-modal correspondences, but only when these involve sound symbolism. This is only partly consistent with the idea of a continuum between synesthesia and cross-modal correspondences, but accords with the idea that synesthesia is a high-level, post-perceptual phenomenon, with spillover of the abilities of synesthetes into domains outside their synesthesias. To our knowledge, this is the first demonstration that synesthetes, relative to non-synesthetes, experience stronger cross-modal correspondences outside their synesthetic domains.

Multisensory integration and cross-modal learning in synaesthesia: A unifying model

Recent research into synaesthesia has highlighted the role of learning, yet synaesthesia is clearly a genetic condition. Here we ask how can the idea that synaesthesia reflects innate, genetic differences be reconciled with models that suggest it is driven by learning. A number of lines of evidence suggest that synaesthesia relies on, or at least interacts with, processes of multisensory integration that are common across all people. These include multisensory activations that arise in early regions of the brain as well as feedback from longer-term cross-modal associations generated in memory. These cognitive processes may interact independently to influence the phenomenology of the synaesthetic experience, as well as the individual differences within particular types of synaesthesia. The theoretical framework presented here is consistent with both an innate difference as the fundamental driver of the condition of synaesthesia, and with experiential and semantic influences on the eventual phenotype that emerges. In particular, it proposes that the internally generated synaesthetic percepts are treated similarly to other sensory information as the brain is learning the multisensory attributes of objects and developing cross-modal associations that merge in the concept of the object.

What is the link between synaesthesia and sound symbolism?

Sound symbolism is a property of certain words which have a direct link between their phonological form and their semantic meaning. In certain instances, sound symbolism can allow non-native speakers to understand the meanings of etymologically unfamiliar foreign words, although the mechanisms driving this are not well understood. We examined whether sound symbolism might be mediated by the same types of cross-modal processes that typify synaesthetic experiences. Synaesthesia is an inherited condition in which sensory or cognitive stimuli (e.g., sounds, words) cause additional, unusual cross-modal percepts (e.g., sounds trigger colours, words trigger tastes). Synaesthesia may be an exaggeration of normal cross-modal processing, and if so, there may be a link between synaesthesia and the type of cross-modality inherent in sound symbolism. To test this we predicted that synaesthetes would have superior understanding of unfamiliar (sound symbolic) foreign words. In our study, 19 grapheme-colour synaesthetes and 57 non-synaesthete controls were presented with 400 adjectives from 10 unfamiliar languages and were asked to guess the meaning of each word in a two-alternative forced-choice task. Both groups showed superior understanding compared to chance levels, but synaesthetes significantly outperformed controls. This heightened ability suggests that sound symbolism may rely on the types of cross-modal integration that drive synaesthetes' unusual experiences. It also suggests that synaesthesia endows or co-occurs with heightened multi-modal skills, and that this can arise in domains unrelated to the specific form of synaesthesia.

N1 enhancement in synesthesia during visual and audio-visual perception in semantic cross-modal conflict situations: an ERP study

Synesthesia entails a special kind of sensory perception, where stimulation in one sensory modality leads to an internally generated perceptual experience of another, not stimulated sensory modality. This phenomenon can be viewed as an abnormal multisensory integration process as here the synesthetic percept is aberrantly fused with the stimulated modality. Indeed, recent synesthesia research has focused on multimodal processing even outside of the specific synesthesia-inducing context and has revealed changed multimodal integration, thus suggesting perceptual alterations at a global level. Here, we focused on audio–visual processing in synesthesia using a semantic classification task in combination with visually or auditory–visually presented animated and in animated objects in an audio–visual congruent and incongruent manner. Fourteen subjects with auditory-visual and/or grapheme-color synesthesia and 14 control subjects participated in the experiment. During presentation of the stimuli, event-related potentials were recorded from 32 electrodes. The analysis of reaction times and error rates revealed no group differences with best performance for audio-visually congruent stimulation indicating the well-known multimodal facilitation effect. We found enhanced amplitude of the N1 component over occipital electrode sites for synesthetes compared to controls. The differences occurred irrespective of the experimental condition and therefore suggest a global influence on early sensory processing in synesthetes.

Cross-linguistic sound symbolism and crossmodal correspondence: Evidence from fMRI and DTI

Non-arbitrary correspondences between spoken words and categories of meanings exist in natural language, with mounting evidence that listeners are sensitive to this sound symbolic information. Native English speakers were asked to choose the meaning of spoken foreign words from one of four corresponding antonym pairs selected from a previously developed multi-language stimulus set containing both sound symbolic and non-symbolic stimuli. In behavioral (n=9) and fMRI (n=15) experiments, participants showed reliable sensitivity to the sound symbolic properties of the stimulus set, selecting the consistent meaning for the sound symbolic words at above chances rates. There was increased activation for sound symbolic relative to non-symbolic words in left superior parietal cortex, and a cluster in left superior longitudinal fasciculus showed a positive correlation between fractional anisotropy (FA) and an individual's sensitivity to sound symbolism. These findings support the idea that crossmodal correspondences underlie sound symbolism in spoken language.

Real color captures attention and overrides spatial cues in grapheme-color synesthetes but not in controls

Grapheme-color synesthetes perceive color when reading letters or digits. We investigated oscillatory brain signals of synesthetes vs. controls using magnetoencephalography. Brain oscillations specifically in the alpha band (∼10Hz) have two interesting features: alpha has been linked to inhibitory processes and can act as a marker for attention. The possible role of reduced inhibition as an underlying cause of synesthesia, as well as the precise role of attention in synesthesia is widely discussed. To assess alpha power effects due to synesthesia, synesthetes as well as matched controls viewed synesthesia-inducing graphemes, colored control graphemes, and non-colored control graphemes while brain activity was recorded. Subjects had to report a color change at the end of each trial which allowed us to assess the strength of synesthesia in each synesthete. Since color (synesthetic or real) might allocate attention we also included an attentional cue in our paradigm which could direct covert attention. In controls the attentional cue always caused a lateralization of alpha power with a contralateral decrease and ipsilateral alpha increase over occipital sensors. In synesthetes, however, the influence of the cue was overruled by color: independent of the attentional cue, alpha power decreased contralateral to the color (synesthetic or real). This indicates that in synesthetes color guides attention. This was confirmed by reaction time effects due to color, i.e. faster RTs for the color side independent of the cue. Finally, the stronger the observed color dependent alpha lateralization, the stronger was the manifestation of synesthesia as measured by congruency effects of synesthetic colors on RTs. Behavioral and imaging results indicate that color induces a location-specific, automatic shift of attention towards color in synesthetes but not in controls. We hypothesize that this mechanism can facilitate coupling of grapheme and color during the development of synesthesia.

Principles of multisensory behavior

The combined use of multisensory signals is often beneficial. Based on neuronal recordings in the superior colliculus of cats, three basic rules were formulated to describe the effectiveness of multisensory signals: the enhancement of neuronal responses to multisensory compared with unisensory signals is largest when signals occur at the same location ("spatial rule"), when signals are presented at the same time ("temporal rule"), and when signals are rather weak ("principle of inverse effectiveness"). These rules are also considered with respect to multisensory benefits as observed with behavioral measures, but do they capture these benefits best? To uncover the principles that rule benefits in multisensory behavior, we here investigated the classical redundant signal effect (RSE; i.e., the speedup of response times in multisensory compared with unisensory conditions) in humans. Based on theoretical considerations using probability summation, we derived two alternative principles to explain the effect. First, the "principle of congruent effectiveness" states that the benefit in multisensory behavior (here the speedup of response times) is largest when behavioral performance in corresponding unisensory conditions is similar. Second, the "variability rule" states that the benefit is largest when performance in corresponding unisensory conditions is unreliable. We then tested these predictions in two experiments, in which we manipulated the relative onset and the physical strength of distinct audiovisual signals. Our results, which are based on a systematic analysis of response time distributions, show that the RSE follows these principles very well, thereby providing compelling evidence in favor of probability summation as the underlying combination rule.

Parietal connectivity mediates multisensory facilitation

Our senses interact in daily life through multisensory integration, facilitating perceptual processes and behavioral responses. The neural mechanisms proposed to underlie this multisensory facilitation include anatomical connections directly linking early sensory areas, indirect connections to higher-order multisensory regions, as well as thalamic connections. Here we examine the relationship between white matter connectivity, as assessed with diffusion tensor imaging, and individual differences in multisensory facilitation and provide the first demonstration of a relationship between anatomical connectivity and multisensory processing in typically developed individuals. Using a whole-brain analysis and contrasting anatomical models of multisensory processing we found that increased connectivity between parietal regions and early sensory areas was associated with the facilitation of reaction times to multisensory (auditory-visual) stimuli. Furthermore, building on prior animal work suggesting the involvement of the superior colliculus in this process, using probabilistic tractography we determined that the strongest cortical projection area connected with the superior colliculus includes the region of connectivity implicated in our independent whole-brain analysis.

Enhanced mental rotation ability in time-space synesthesia

Time-space synesthesia is a variant of sequence-space synesthesia and involves the involuntary association of months of the year with 2D and 3D spatial forms, such as arcs, circles, and ellipses. Previous studies have revealed conflicting results regarding the association between time-space synesthesia and enhanced spatial processing ability. Here, we tested 15 time-space synesthetes, and 15 non-synesthetic controls matched for age, education, and gender on standard tests of mental rotation ability, spatial working memory, and verbal working memory. Synesthetes performed better than controls on our test of mental rotation, but similarly to controls on tests of spatial and verbal working memory. Results support a dissociation between visuo-spatial imagery and spatial working memory capacity, and suggest time-space synesthesia is associated only with enhanced visuo-spatial imagery. These data are consistent with the time-space connectivity thesis that time-space synesthesia results from enhanced connectivity in the parietal lobe between regions supporting the representation of temporal sequences and those underlying visuo-spatial imagery.

Reduced audiovisual integration in synesthesia--evidence from bimodal speech perception

Recent research suggests synesthesia as a result of a hypersensitive multimodal binding mechanism. To address the question whether multimodal integration is altered in synesthetes in general, grapheme-colour and auditory-visual synesthetes were investigated using speech-related stimulation in two behavioural experiments. First, we used the McGurk illusion to test the strength and number of illusory perceptions in synesthesia. In a second step, we analysed the gain in speech perception coming from seen articulatory movements under acoustically noisy conditions. We used disyllabic nouns as stimulation and varied signal-to-noise ratio of the auditory stream presented concurrently to a matching video of the speaker. We hypothesized that if synesthesia is due to a general hyperbinding mechanism this group of subjects should be more susceptible to McGurk illusions and profit more from the visual information during audiovisual speech perception. The results indicate that there are differences between synesthetes and controls concerning multisensory integration--but in the opposite direction as hypothesized. Synesthetes showed a reduced number of illusions and had a reduced gain in comprehension by viewing matching articulatory movements in comparison to control subjects. Our results indicate that rather than having a hypersensitive binding mechanism, synesthetes show weaker integration of vision and audition.

Crossmodal correspondences between sounds and tastes

In this article, the rapidly growing body of research that has been published recently on the topic of crossmodal correspondences that involve auditory and gustatory/flavor stimuli is critically reviewed. The evidence demonstrates that people reliably match different tastes/flavors to auditory stimuli varying in both their psychoacoustic (e.g., pitch) and musical (e.g., timbre) properties. In order to stimulate further progress in this relatively young research field, the present article aims at consolidating prior findings concerning specific auditory-gustatory mappings, whereby special attention is given to highlighting (1) any conflicts in the existing experimental evidence and (2) any potential caveats with regard to the most appropriate interpretation of prior studies. Next, potential mechanisms underlying auditory-gustatory crossmodal correspondences are discussed. Finally, a number of potentially fruitful avenues for future research are outlined.

Reduced audio-visual integration in synaesthetes indicated by the double-flash illusion

It has been suggested that synaesthesia is the result of a hyper-sensitive multimodal binding-mechanism. To address the question whether multi-modal integration is altered in synaesthetes in general, grapheme-colour and auditory-visual synaesthetes were studied using the double-flash illusion. This illusion is induced by a single light flash presented together with multiple beep sounds, which is then perceived as multiple flashes. By varying the separation of auditory and visual stimuli, the hypothesis of a widened temporal window of audio-visual integration in synaesthetes was tested. As hypothesised, the results show differences between synaesthetes and controls concerning multisensory integration, but surprisingly other than expected synaesthetes perceive a reduced number of illusions and have a smaller time-window of audio-visual integration compared to controls. This indicates that they do not have a hyper-sensitive binding mechanism. On the contrary, synaesthetes seem to integrate even less than controls between vision and audition.

Survival of the synesthesia gene: why do people hear colors and taste words?

This Unsolved Mystery reviews the biological evidence for why synesthesia, a condition in which stimuli presented through one modality spontaneously evoke sensations in an unrelated modality, may have been conserved in the population.

Synesthesia is a perceptual experience in which stimuli presented through one modality will spontaneously evoke sensations in an unrelated modality. The condition occurs from increased communication between sensory regions and is involuntary, automatic, and stable over time. While synesthesia can occur in response to drugs, sensory deprivation, or brain damage, research has largely focused on heritable variants comprising roughly 4% of the general population. Genetic research on synesthesia suggests the phenomenon is heterogeneous and polygenetic, yet it remains unclear whether synesthesia ever provided a selective advantage or is merely a byproduct of some other useful selected trait. Progress in uncovering the genetic basis of synesthesia will help us understand why synesthesia has been conserved in the population.

Contextual Priming in Grapheme–Color Synesthetes and Yoked Controls: 400 msec in the Life of a Synesthete

Grapheme–color synesthesia is a heritable trait where graphemes (“2”) elicit the concurrent perception of specific colors (red). Researchers have questioned whether synesthetic experiences are meaningful or simply arbitrary associations and whether these associations are perceptual or conceptual. To address these fundamental questions, ERPs were recorded as 12 synesthetes read statements such as “The Coca-Cola logo is white and 2,” in which the final grapheme induced a color that was either contextually congruous (red) or incongruous (“…white and 7,” for a synesthetes who experienced 7 as green). Grapheme congruity was found to modulate the amplitude of the N1, P2, N300, and N400 components in synesthetes, suggesting that synesthesia impacts perceptual as well as conceptual aspects of processing. To evaluate whether observed ERP effects required the experience of colored graphemes versus knowledge of grapheme–color pairings, we ran three separate groups of controls on a similar task. Controls trained to a synesthete's associations elicited N400 modulation, indicating that knowledge of grapheme–color mappings was sufficient to modulate this component. Controls trained to synesthetic associations and given explicit visualization instructions elicited both N300 and N400 modulations. Lastly, untrained controls who viewed physically colored graphemes (“2” printed in red) elicited N1 and N400 modulations. The N1 grapheme congruity effect began earlier in synesthetes than colored grapheme controls but had similar scalp topography. Data suggest that, in synesthetes, achromatic graphemes engage similar visual processing networks as colored graphemes in nonsynesthetes and are in keeping with models of synesthesia that posit early feed-forward connections between form and color processing areas in extrastriate cortex. The P2 modulation was unique to the synesthetes and may reflect neural activity that underlies the conscious experience of the synesthetic induction.