Neural responses in parietal and occipital areas in response to visual events are modulated by prior multisensory stimuli

Self as a prior: The malleability of Bayesian multisensory integration to social salience

Our everyday perceptual experiences are grounded in the integration of information within and across our senses. Due to this direct behavioural relevance, cross-modal integration retains a certain degree of contextual flexibility, even to social relevance. However, how social relevance modulates cross-modal integration remains unclear. To investigate possible mechanisms, Experiment 1 tested the principles of audio-visual integration for numerosity estimation by deriving a Bayesian optimal observer model with perceptual prior from empirical data to explain perceptual biases. Such perceptual priors may shift towards locations of high salience in the stimulus space. Our results showed that the tendency to over- or underestimate numerosity, expressed in the frequency and strength of fission and fusion illusions, depended on the actual event numerosity. Experiment 2 replicated the effects of social relevance on multisensory integration from Scheller & Sui, 2022 JEP:HPP, using a lower number of events, thereby favouring the opposite illusion through enhanced influences of the prior. In line with the idea that the self acts like a prior, the more frequently observed illusion (more malleable to prior influences) was modulated by self-relevance. Our findings suggest that the self can influence perception by acting like a prior in cue integration, biasing perceptual estimates towards areas of high self-relevance.

Two are better than one: Differences in cortical EEG patterns during auditory and visual verbal working memory processing between Unilateral and Bilateral Cochlear Implanted children

Despite the proven effectiveness of cochlear implant (CI) in the hearing restoration of deaf or hard-of-hearing (DHH) children, to date, extreme variability in verbal working memory (VWM) abilities is observed in both unilateral and bilateral CI user children (CIs). Although clinical experience has long observed deficits in this fundamental executive function in CIs, the cause to date is still unknown. Here, we have set out to investigate differences in brain functioning regarding the impact of monaural and binaural listening in CIs compared with normal hearing (NH) peers during a three-level difficulty n-back task undertaken in two sensory modalities (auditory and visual). The objective of this pioneering study was to identify electroencephalographic (EEG) marker pattern differences in visual and auditory VWM performances in CIs compared to NH peers and possible differences between unilateral cochlear implant (UCI) and bilateral cochlear implant (BCI) users. The main results revealed differences in theta and gamma EEG bands. Compared with hearing controls and BCIs, UCIs showed hypoactivation of theta in the frontal area during the most complex condition of the auditory task and a correlation of the same activation with VWM performance. Hypoactivation in theta was also observed, again for UCIs, in the left hemisphere when compared to BCIs and in the gamma band in UCIs compared to both BCIs and NHs. For the latter two, a correlation was found between left hemispheric gamma oscillation and performance in the audio task. These findings, discussed in the light of recent research, suggest that unilateral CI is deficient in supporting auditory VWM in DHH. At the same time, bilateral CI would allow the DHH child to approach the VWM benchmark for NH children. The present study suggests the possible effectiveness of EEG in supporting, through a targeted approach, the diagnosis and rehabilitation of VWM in DHH children.

Sound-induced flash illusions at different spatial locations were affected by personality traits

Sound-induced flash illusion (SiFI) is an auditory-dominated effect in which observers will misperceive the number of flashes due to simultaneously presented beeps, which includes fission and fusion illusions. Although several individual differences have been found in SiFI, little is known about the effect of personality traits. In the present study, we presented flashes in near space and beeps in far space (Vnear_Afar) and flashes in far space and beeps in near space (Vfar_Anear) to better approximate the real world. We collected 103 participants' Big Five questionnaire results and their SiFI task performance to investigate the difference in trait level on the SiFI in the performance of accuracy, d' and c. The results show that all five personality traits had certain effects on the SiFI to different degrees, and different personality traits played different roles in the fission illusion and fusion illusion. The high agreeableness group was more prone to the fission illusion, and the report criteria were less strict. The report criteria of the low neuroticism group were stricter for the fusion illusion. The extraversion, conscientiousness and low openness groups were more prone to the fusion illusion in the Vnear_Afar condition than in the Vfar_Anear condition. The study indicated that personality traits were important but easily overlooked factors in multisensory illusion, which might make a difference between the fission illusion and the fusion illusion.

Differences in eccentricity for sound-induced flash illusion in four visual fields

A sound-induced flash illusion (SiFI) is a multisensory illusion dominated by auditory stimuli, in which the individual perceives that the number of visual flashes is equal to the number of auditory stimuli when visual flashes are presented along with an unequal number of auditory stimuli. Although the mechanisms underlying fission and fusion illusions have been documented, there is not yet a consensus on how they vary according to the different eccentricities. In the present study, by incorporating the classic SiFI paradigm into four different eccentricities, we aimed to investigate whether the SiFI varies under the different eccentricities. The results showed that the fission illusion varied significantly across the four eccentricities, with the perifovea (7°) and peripheral (11°) illusions being greater than the fovea and parafovea (3°) illusions. In contrast, the fusion illusion did not vary significantly across the four eccentricities. Our findings revealed that SiFI was affected by different visual fields and that there were differences between the fission and the fusion illusions. Furthermore, by examining the SiFI of eccentricity across visual fields, this study also suggests that bottom-up factors affect the SiFI.

Compensation Mechanisms May Not Always Account for Enhanced Multisensory Illusion in Older Adults: Evidence from Sound-Induced Flash Illusion

Sound-induced flash illusion (SiFI) is typical auditory dominance phenomenon in multisensory illusion. Although a number of studies have explored the SiFI in terms of age-related effects, the reasons for the enhanced SiFI in older adults are still controversial. In the present study, older and younger adults with equal visual discrimination were selected to explore age differences in SiFI effects, and to explore the neural indicators by resting-state functional magnetic resonance imaging (rs-fMRI) signals. A correlation analysis was calculated to examine the relationship between regional homogeneity (ReHo) and the SiFI. The results showed that both younger and older adults experienced significant fission and fusion illusions, and fission illusions of older adults were greater than that of younger adults. In addition, our results showed ReHo values of the left middle frontal gyrus (MFG), the right inferior frontal gyrus (IFG) and right superior frontal gyrus (SFG) were significantly positively correlated with the SiFI in older adults. More importantly, the comparison between older and younger adults showed that ReHo values of the right superior temporal gyrus (STG) decreased in older adults, and this was independent of the SiFI. The results indicated that when there was no difference in unisensory ability, the enhancement of multisensory illusion in older adults may not always be explained by compensation mechanisms.

Reward reduces the fission illusion in the sound-induced flash illusion

Pairing a single visual stimulus with multiple auditory stimuli will lead to the illusory perception of multiple visual stimuli, which is known as sound-induced flash illusion (SIFI). The present study adopted the classic SIFI paradigm to investigate whether value-associated tasks could affect the SIFI. By adjusting the sequence of reward and nonreward conditions, we also examined the effect of reward history on SIFI. The results showed that the fission illusion was reduced when associated with momentary reward, demonstrating significantly higher accuracy and discriminability than the nonreward condition. However, the fusion illusion was not affected by the momentary reward, and the explanation was that the fusion illusion was not as stable as the fission illusion and disappeared across different trials and conditions. Moreover, the robustness of reward history in the present study was not as strong as previous studies have suggested, indicating that the effect of sound on the perceptual representation of visual stimuli is strong and robust to reward history. These findings demonstrated that the reward could reduce the SIFI and broaden the existing dichotomy of SIFI. New evidence for the operation of value-driven attention mechanisms is also provided, suggesting that the underlying value-driven attention operates across multiple sensory systems.

The impact of joint attention on the sound-induced flash illusions

Humans coordinate their focus of attention with others, either by gaze following or prior agreement. Though the effects of joint attention on perceptual and cognitive processing tend to be examined in purely visual environments, they should also show in multisensory settings. According to a prevalent hypothesis, joint attention enhances visual information encoding and processing, over and above individual attention. If two individuals jointly attend to the visual components of an audiovisual event, this should affect the weighing of visual information during multisensory integration. We tested this prediction in this preregistered study, using the well-documented sound-induced flash illusions, where the integration of an incongruent number of visual flashes and auditory beeps results in a single flash being seen as two (fission illusion) and two flashes as one (fusion illusion). Participants were asked to count flashes either alone or together, and expected to be less prone to both fission and fusion illusions when they jointly attended to the visual targets. However, illusions were as frequent when people attended to the flashes alone or with someone else, even though they responded faster during joint attention. Our results reveal the limitations of the theory that joint attention enhances visual processing as it does not affect temporal audiovisual integration.

Complex sensorimotor transformation processes required for response selection are facilitated by the striatum

Both fronto-parietal networks and the basal ganglia play an important role in action cascading. It is well-known that cortical structures mediate sensorimotor transformation for this purpose. The striatum receives extensive input from those cortical structures and has been shown to be modulated by the predictability of cortical input. Until today, it has however remained unclear whether the processing of spatial codes or even sensorimotor transformation processes for the purpose of action cascading involve the striatum. We therefore examined this question by means of fMRI using a stop-change task that varied the predictability as well as the complexity of sensorimotor transformations required for correct responding in the context of action cascading. On the behavioral level, we found that the complexity of sensorimotor transformation processes only prolonged reaction times when the requirement for this transformation was predictable. fMRI results matched this effect showing enhanced activity of the caudate in case a complex sensorimotor transformation could be anticipated. Irrespective of the complexity of the required transformations, the putamen was furthermore involved in the prediction of imminent action cascading demands. Taken together, our findings give rise to a conceptual advance regarding basal ganglia function by showing that the anticipation and, more importantly, processing of complex sensorimotor transformation processes involves the striatum.