Auditory and tactile signals combine to influence vision during binocular rivalry

Auditory guidance of eye movements toward threat-related images in the absence of visual awareness

The human brain is sensitive to threat-related information even when we are not aware of this information. For example, fearful faces attract gaze in the absence of visual awareness. Moreover, information in different sensory modalities interacts in the absence of awareness, for example, the detection of suppressed visual stimuli is facilitated by simultaneously presented congruent sounds or tactile stimuli. Here, we combined these two lines of research and investigated whether threat-related sounds could facilitate visual processing of threat-related images suppressed from awareness such that they attract eye gaze. We suppressed threat-related images of cars and neutral images of human hands from visual awareness using continuous flash suppression and tracked observers' eye movements while presenting congruent or incongruent sounds (finger snapping and car engine sounds). Indeed, threat-related car sounds guided the eyes toward suppressed car images, participants looked longer at the hidden car images than at any other part of the display. In contrast, neither congruent nor incongruent sounds had a significant effect on eye responses to suppressed finger images. Overall, our results suggest that only in a danger-related context semantically congruent sounds modulate eye movements to images suppressed from awareness, highlighting the prioritisation of eye responses to threat-related stimuli in the absence of visual awareness.

Binocular rivalry under naturalistic geometry: Evidence from worlds simulated in virtual reality

Binocular rivalry is a fascinating, widely studied visual phenomenon in which perception alternates between two competing images. This experience, however, is generally restricted to laboratory settings where two irreconcilable images are presented separately to the two eyes, an implausible geometry where two objects occupy the same physical location. Such laboratory experiences are in stark contrast to everyday visual behavior, where rivalry is almost never encountered, casting doubt on whether rivalry is relevant to our understanding of everyday binocular vision. To investigate the external validity of binocular rivalry, we manipulated the geometric plausibility of rival images using a naturalistic, cue-rich, 3D-corridor model created in virtual reality. Rival stimuli were presented in geometrically implausible, semi-plausible, or plausible layouts. Participants tracked rivalry fluctuations in each of these three layouts and for both static and moving rival stimuli. Results revealed significant and canonical binocular rivalry alternations regardless of geometrical plausibility and stimulus type. Rivalry occurred for layouts that mirrored the unnatural geometry used in laboratory studies and for layouts that mimicked real-world occlusion geometry. In a complementary 3D modeling analysis, we show that interocular conflict caused by geometrically plausible occlusion is a common outcome in a visual scene containing multiple objects. Together, our findings demonstrate that binocular rivalry can reliably occur for both geometrically implausible interocular conflicts and conflicts caused by a common form of naturalistic occlusion. Thus, key features of binocular rivalry are not simply laboratory artifacts but generalize to conditions that match the geometry of everyday binocular vision.

Only visible flicker helps flutter: Tactile-visual integration breaks in the absence of visual awareness

Combining information from multiple senses enhances our perception of the world. Whether we need to be aware of all stimuli to benefit from multisensory integration, however, is still under investigation. Here, we tested whether tactile frequency perception benefits from the presence of congruent visual flicker even if the flicker is so rapid that it is perceptually fused into a steady light and therefore invisible. Our participants completed a tactile frequency discrimination task given either unisensory tactile or congruent tactile-visual stimulation. Tactile and tactile-visual test frequencies ranged from far below to far above participants' flicker fusion threshold (determined separately). For frequencies distinctively below their flicker fusion threshold, participants performed significantly better given tactile-visual stimulation than when presented with only tactile stimuli. Yet, for frequencies above their flicker fusion threshold, participants' tactile frequency perception did not profit from the presence of congruent but likely fused and thus invisible visual flicker. The results matched the predictions of an ideal-observer model in which tactile-visual integration is conditional on awareness of both stimuli. In contrast, it was impossible to reproduce the observed results with a model that assumed tactile-visual integration proceeds irrespective of stimulus awareness. In sum, we revealed that the benefits of congruent visual stimulation for tactile flutter frequency perception depend on the visibility of the visual flicker, suggesting that multisensory integration requires awareness.

The Dynamic Target Motion Perception Mechanism of Tactile-Assisted Vision in MR Environments

In the mixed reality (MR) environment, the task of target motion perception is usually undertaken by vision. This approach suffers from poor discrimination and high cognitive load when the tasks are complex. This cannot meet the needs of the air traffic control field for rapid capture and precise positioning of the dynamic targets in the air. Based on this problem, we conducted a multimodal optimization study on target motion perception judgment by controlling the hand tactile sensor to achieve the use of tactile sensation to assist vision in MR environment. This allows it to adapt to the requirements of future development-led interactive tasks under the mixed reality holographic aviation tower. Motion perception tasks are usually divided into urgency sensing for multiple targets and precise position tracking for single targets according to the number of targets and task division. Therefore, in this paper, we designed experiments to investigate the correlation between tactile intensity-velocity correspondence and target urgency, and the correlation between the PRS (position, rhythm, sequence) tactile indication scheme and position tracking. We also evaluated it through comprehensive experiment. We obtained the following conclusions: (1) high, higher, medium, lower, and low tactile intensities would bias human visual cognitive induction to fast, faster, medium, slower, and slow motion targets. Additionally, this correspondence can significantly improve the efficiency of the participants' judgment of target urgency; (2) under the PRS tactile indication scheme, position-based rhythm and sequence cues can improve the judgment effect of human tracking target dynamic position, and the effect of adding rhythm cues is better. However, when adding rhythm and sequence cues at the same time, it can cause clutter; (3) tactile assisted vision has a good improvement effect on the comprehensive perception of dynamic target movement. The above findings are useful for the study of target motion perception in MR environments and provide a theoretical basis for subsequent research on the cognitive mechanism and quantitative of tactile indication in MR environment.

Direction-selective modulation of visual motion rivalry by collocated tactile motion

Early models of multisensory integration posited that cross-modal signals only converged in higher-order association cortices and that vision automatically dominates. However, recent studies have challenged this view. In this study, the significance of the alignment of motion axes and spatial alignment across visual and tactile stimuli, as well as the effect of hand visibility on visuo-tactile interactions were examined. Using binocular rivalry, opposed motions were presented to each eye and participants were required to track the perceived visual direction. A tactile motion that was either a leftward or rightward sweep across the fingerpad was intermittently presented. Results showed that tactile effects on visual percepts were dependent on the alignment of motion axes: rivalry between up/down visual motions was not modulated at all by left/right tactile motion. On the other hand, visual percepts could be altered by tactile motion signals when both modalities shared a common axis of motion: a tactile stimulus could maintain the dominance duration of a congruent visual stimulus and shorten its suppression period. The effects were also conditional on the spatial alignment of the visual and tactile stimuli, being eliminated when the tactile device was displaced 15 cm away to the right of the visual stimulus. In contrast, visibility of the hand touching the tactile stimulus facilitated congruent switches relative to a visual-only baseline but did not present a significant advantage overall. In sum, these results show a low-level sensory interaction that is conditional on visual and tactile stimuli sharing a common motion axis and location in space.

Effect of Impaired Stereoscopic Vision on Large-Scale Resting-State Functional Network Connectivity in Comitant Exotropia Patients

Background

Comitant exotropia (CE) is a common eye movement disorder, characterized by impaired eye movements and stereoscopic vision. CE patients reportedly exhibit changes in the central nervous system. However, it remains unclear whether large-scale brain network changes occur in CE patients.

Purpose

This study investigated the effects of exotropia and stereoscopic vision dysfunction on large-scale brain networks in CE patients via independent component analysis (ICA).

Methods

Twenty-eight CE patients (mean age, 15.80 ± 2.46 years) and 27 healthy controls (HCs; mean age, 16.00 ± 2.68 years; closely matched for age, sex, and education) underwent resting-state magnetic resonance imaging. ICA was applied to extract resting-state networks (RSNs) in both groups. Two-sample’s t-tests were conducted to investigate intranetwork functional connectivity (FC) within RSNs and interactions among RSNs between the two groups.

Results

Compared with the HC group, the CE group showed increased intranetwork FC in the bilateral postcentral gyrus of the sensorimotor network (SMN). The CE group also showed decreased intranetwork FC in the right cerebellum_8 of the cerebellum network (CER), the right superior temporal gyrus of the auditory network (AN), and the right middle occipital gyrus of the visual network (VN). Moreover, functional network connectivity (FNC) analysis showed that CER-AN, SMN-VN, SN-DMN, and DMN-VN connections were significantly altered between the two groups.

Conclusion

Comitant exotropia patients had abnormal brain networks related to the CER, SMN, AN, and VN. Our results offer important insights into the neural mechanisms of eye movements and stereoscopic vision dysfunction in CE patients.

The body-ownership is unconsciously distorted in the brain: An event-related potential study of rubber hand illusion

Many studies have reported that bottom-up multisensory integration of visual, tactile, and proprioceptive information can distort our sense of body-ownership, producing rubber hand illusion (RHI). There is less evidence about when and how the body-ownership is distorted in the brain during RHI. To examine whether this illusion effect occurs preattentively at an early stage of processing, we monitored the visual mismatch negativity (vMMN) component (the index of automatic deviant detection) and N2 (the index for conflict monitoring). Participants first performed an RHI elicitation task in a synchronous or asynchronous setting and then finished a passive visual oddball task in which the deviant stimuli were unrelated to the explicit task. A significant interaction between Deviancy (deviant hand vs. standard hand) and Group (synchronous vs. asynchronous) was found. The asynchronous group showed clear mismatch effects in both vMMN and N2, while the synchronous group had such effect only in N2. The results indicate that after the elicitation of RHI bottom-up integration could be retrieved at the early stage of sensory processing before top-down processing, providing evidence for the priority of the bottom-up processes after the generation of RHI and revealing the mechanism of how the body-ownership is unconsciously distorted in the brain.

Vestibular and active self-motion signals drive visual perception in binocular rivalry

Multisensory integration helps the brain build reliable models of the world and resolve ambiguities. Visual interactions with sound and touch are well established but vestibular influences on vision are less well studied. Here, we test the vestibular influence on vision using horizontally opposed motions presented one to each eye so that visual perception is unstable and alternates irregularly. Passive, whole-body rotations in the yaw plane stabilized visual alternations, with perceived direction oscillating congruently with rotation (leftward motion during leftward rotation, and vice versa). This demonstrates a purely vestibular signal can resolve ambiguous visual motion and determine visual perception. Active self-rotation following the same sinusoidal profile also entrained vision to the rotation cycle – more strongly and with a lesser time lag, likely because of efference copy and predictive internal models. Both experiments show that visual ambiguity provides an effective paradigm to reveal how vestibular and motor inputs can shape visual perception.

•

Binocular rivalry between left/right motions is stabilized by congruent head movement

•

Left/right head rotations entrain rivalry dynamics so matching direction is perceived

•

Active and passive rotations both drive rivalry dominance to match rotation direction

•

Resolving ambiguous vision occurs in a broader vestibular and action-based context

Consciousness in active inference: Deep self-models, other minds, and the challenge of psychedelic-induced ego-dissolution

Predictive processing approaches to brain function are increasingly delivering promise for illuminating the computational underpinnings of a wide range of phenomenological states. It remains unclear, however, whether predictive processing is equipped to accommodate a theory of consciousness itself. Furthermore, objectors have argued that without specification of the core computational mechanisms of consciousness, predictive processing is unable to inform the attribution of consciousness to other non-human (biological and artificial) systems. In this paper, I argue that an account of consciousness in the predictive brain is within reach via recent accounts of phenomenal self-modelling in the active inference framework. The central claim here is that phenomenal consciousness is underpinned by 'subjective valuation'-a deep inference about the precision or 'predictability' of the self-evidencing ('fitness-promoting') outcomes of action. Based on this account, I argue that this approach can critically inform the distribution of experience in other systems, paying particular attention to the complex sensory attenuation mechanisms associated with deep self-models. I then consider an objection to the account: several recent papers argue that theories of consciousness that invoke self-consciousness as constitutive or necessary for consciousness are undermined by states (or traits) of 'selflessness'; in particular the 'totally selfless' states of ego-dissolution occasioned by psychedelic drugs. Drawing on existing work that accounts for psychedelic-induced ego-dissolution in the active inference framework, I argue that these states do not threaten to undermine an active inference theory of consciousness. Instead, these accounts corroborate the view that subjective valuation is the constitutive facet of experience, and they highlight the potential of psychedelic research to inform consciousness science, computational psychiatry and computational phenomenology.

Perceptual Awareness of Optic Flows Paced Optimally and Non-optimally to Walking Speed

Previous research suggests that visual processing depends strongly on locomotor activity and is tuned to optic flows consistent with self-motion speed. Here, we used a binocular rivalry paradigm to investigate whether perceptual access to optic flows depends on their optimality in relation to walking velocity. Participants walked at two different speeds on a treadmill while viewing discrepant visualizations of a virtual tunnel in each eye. We hypothesized that visualizations paced appropriately to the walking speeds will be perceived longer than non optimal (too fast/slow) ones. The presented optic flow speeds were predetermined individually in a task based on matching visual speed to both walking velocities. In addition, perceptual preference for optimal optic flows was expected to increase with proprioceptive ability to detect threshold-level changes in walking speed. Whereas faster (more familiar) optic flows showed enhanced access to awareness during faster compared with slower walking conditions, for slower visual flows, only a nonsignificant tendency for the analogous effect was observed. These effects were not dependent on individual proprioceptive sensitivity. Our findings concur with the emerging view that the velocity of one's locomotion is used to calibrate visual perception of self-motion and extend the scope of reported action effects on visual awareness.

Forward optic flow is prioritised in visual awareness independently of walking direction

When two different images are presented separately to each eye, one experiences smooth transitions between them-a phenomenon called binocular rivalry. Previous studies have shown that exposure to signals from other senses can enhance the access of stimulation-congruent images to conscious perception. However, despite our ability to infer perceptual consequences from bodily movements, evidence that action can have an analogous influence on visual awareness is scarce and mainly limited to hand movements. Here, we investigated whether one's direction of locomotion affects perceptual access to optic flow patterns during binocular rivalry. Participants walked forwards and backwards on a treadmill while viewing highly-realistic visualisations of self-motion in a virtual environment. We hypothesised that visualisations congruent with walking direction would predominate in visual awareness over incongruent ones, and that this effect would increase with the precision of one's active proprioception. These predictions were not confirmed: optic flow consistent with forward locomotion was prioritised in visual awareness independently of walking direction and proprioceptive abilities. Our findings suggest the limited role of kinaesthetic-proprioceptive information in disambiguating visually perceived direction of self-motion and indicate that vision might be tuned to the (expanding) optic flow patterns prevalent in everyday life.

Growing evidence for separate neural mechanisms for attention and consciousness

Our conscious experience of the world seems to go in lockstep with our attentional focus: We tend to see, hear, taste, and feel what we attend to, and vice versa. This tight coupling between attention and consciousness has given rise to the idea that these two phenomena are indivisible. In the late 1950s, the honoree of this special issue, Charles Eriksen, was among a small group of early pioneers that sought to investigate whether a transient increase in overall level of attention (alertness) in response to a noxious stimulus can be decoupled from conscious perception using experimental techniques. Recent years saw a similar debate regarding whether attention and consciousness are two dissociable processes. Initial evidence that attention and consciousness are two separate processes primarily rested on behavioral data. However, the past couple of years witnessed an explosion of studies aimed at testing this conjecture using neuroscientific techniques. Here we provide an overview of these and related empirical studies on the distinction between the neuronal correlates of attention and consciousness, and detail how advancements in theory and technology can bring about a more detailed understanding of the two. We argue that the most promising approach will combine ever-evolving neurophysiological and interventionist tools with quantitative, empirically testable theories of consciousness that are grounded in a mathematically formalized understanding of phenomenology.

Rotating One's Head Modulates the Perceived Velocity of Motion Aftereffect

As a prominent illusion, the motion aftereffect (MAE) has traditionally been considered a visual phenomenon. Recent neuroimaging work has revealed increased activities in MT+ and decreased activities in vestibular regions during the MAE, supporting the notion of visual-vestibular interaction on the MAE. Since the head had to remain stationary in fMRI experiments, vestibular self-motion signals were absent in those studies. Accordingly, more direct evidence is still lacking in terms of whether and how vestibular signals modulate the MAE. By developing a virtual reality approach, the present study for the first time demonstrates that horizontal head rotation affects the perceived velocity of the MAE. We found that the MAE was predominantly perceived as moving faster when its direction was opposite to the direction of head rotation than when its direction was the same as head rotation. The magnitude of this effect was positively correlated with the velocity of head rotation. Similar result patterns were not observed for the real motion stimuli. Our findings support a 'cross-modal bias' hypothesis that after living in a multisensory environment long-term the brain develops a strong association between signals from the visual and vestibular pathways. Consequently, weak biasing visual signals in the associated direction can spontaneously emerge with the input of vestibular signals in the multisensory brain areas, substantially modulating the illusory visual motion represented in those areas as well. The hypothesis can also be used to explain other multisensory integration phenomena.

Tactile priming accelerates conscious access to continuous flash-suppressed characters

NEW FINDINGS

What is the central question of this study? Research has reported that some sensory input, such as auditory and olfactory input, can affect subliminal visual processing. However, it is important to address whether tactile input, another form of elementary sensory input, could influence the interocular rivalry process. What is the main finding and its importance? We present several pieces of evidence regarding the influences of familiar tactile shapes and temperature on continuous flash suppression. Our findings provide support for the hypothesis that there is a cross-modal effect on subconscious visual semantic processing of Chinese characters. More specifically, tactile sensations affect subliminal processing of visual information.

ABSTRACT

Tactile and visual sensations are the most vital human functions for obtaining environmental information. However, whether tactile information influences visual processing remains unclear. In this study, a breaking continuous flash suppression (b-CFS) protocol was used to measure the extent to which tactile sensations facilitate visual processing subconsciously. In experiment 1, finger stimulation with cold and hot temperatures served as primers for the words 'cold' and 'hot', which were in turn suppressed by CFS. In experiment 2, subjects viewed the upright or inverted word 'cell phone', with or without tactile priming of holding a cell phone in their hand. Results demonstrated that the tactile primer significantly shortened the reaction time in the touch group compared with the control group in both experiments. Thus, the tactile sensation of a familiar article and/or temperature appears to facilitate corresponding visual semantic recognition to break CFS earlier.

Effects of Multimodal Association on Ambiguous Perception in Binocular Rivalry

When two eyes view dissimilar images, an observer typically reports ambiguous perception called binocular rivalry where the subjective perception fluctuates between the two inputs. This perceptual instability is often comprised of exclusive dominance of each image and a transition state called piecemeal state where the two images are intermingled in patchwork manner. Herein, we investigated the effects of multimodal association of sensory congruent pair, arbitrary pair, and reverse pair on piecemeal state in order to see how each level of association affects the ambiguous perception during binocular rivalry. To induce the multisensory associations, we designed a matching task with audiovisual feedback where subjects were required to respond according to given pairing rules. We found that explicit audiovisual associations can substantially affect the piecemeal state during binocular rivalry and that this congruency effect that reduces the amount of visual ambiguity originates primarily from explicit audiovisual association training rather than common sensory features. Furthermore, when one information is associated with multiple information, recent and preexisting associations work collectively to influence the perceptual ambiguity during rivalry. Our findings show that learned multimodal association directly affects the temporal dynamics of ambiguous perception during binocular rivalry by modulating not only the exclusive dominance but also the piecemeal state in a systematic manner.

Sensorimotor contingency modulates breakthrough of virtual 3D objects during a breaking continuous flash suppression paradigm

To investigate how embodied sensorimotor interactions shape subjective visual experience, we developed a novel combination of Virtual Reality (VR) and Augmented Reality (AR) within an adapted breaking continuous flash suppression (bCFS) paradigm. In a first experiment, participants manipulated novel virtual 3D objects, viewed through a head-mounted display, using three interlocking cogs. This setup allowed us to manipulate the sensorimotor contingencies governing interactions with virtual objects, while characterising the effects on subjective visual experience by measuring breakthrough times from bCFS. We contrasted the effects of the congruency (veridical versus reversed sensorimotor coupling) and contingency (live versus replayed interactions) using a motion discrimination task. The results showed that the contingency but not congruency of sensorimotor coupling affected breakthrough times, with live interactions displaying faster breakthrough times. In a second experiment, we investigated how the contingency of sensorimotor interactions affected object category discrimination within a more naturalistic setting, using a motion tracker that allowed object interactions with increased degrees of freedom. We again found that breakthrough times were faster for live compared to replayed interactions (contingency effect). Together, these data demonstrate that bCFS breakthrough times for unfamiliar 3D virtual objects are modulated by the contingency of the dynamic causal coupling between actions and their visual consequences, in line with theories of perception that emphasise the influence of sensorimotor contingencies on visual experience. The combination of VR/AR and motion tracking technologies with bCFS provides a novel methodology extending the use of binocular suppression paradigms into more dynamic and realistic sensorimotor environments.

Tactile motion biases visual motion perception in binocular rivalry

There is an ongoing debate whether or not multisensory interactions require awareness of the sensory signals. Static visual and tactile stimuli have been shown to influence each other even in the absence of visual awareness. However, it is unclear if this finding generalizes to dynamic contexts. In the present study, we presented visual and tactile motion stimuli and induced fluctuations of visual awareness by means of binocular rivalry: two gratings which drifted in opposite directions were displayed, one to each eye. One visual motion stimulus dominated and reached awareness while the other visual stimulus was suppressed from awareness. Tactile motion stimuli were presented at random time points during the visual stimulation. The motion direction of a tactile stimulus always matched the direction of one of the concurrently presented visual stimuli. The visual gratings were differently tinted, and participants reported the color of the currently seen stimulus. Tactile motion delayed perceptual switches that ended dominance periods of congruently moving visual stimuli compared to switches during visual-only stimulation. In addition, tactile motion fostered the return to dominance of suppressed, congruently moving visual stimuli, but only if the tactile motion started at a late stage of the ongoing visual suppression period. At later stages, perceptual suppression is typically decreasing. These results suggest that visual awareness facilitates but does not gate multisensory interactions between visual and tactile motion signals.

Reciprocal Interactions Between Audition and Touch in Flutter Frequency Perception

In both audition and touch, sensory cues comprising repeating events are perceived either as a continuous signal or as a stream of temporally discrete events (flutter), depending on the events' repetition rate. At high repetition rates (>100 Hz), auditory and tactile cues interact reciprocally in pitch processing. The frequency of a cue experienced in one modality systematically biases the perceived frequency of a cue experienced in the other modality. Here, we tested whether audition and touch also interact in the processing of low-frequency stimulation. We also tested whether multisensory interactions occurred if the stimulation in one modality comprised click trains and the stimulation in the other modality comprised amplitude-modulated signals. We found that auditory cues bias touch and tactile cues bias audition on a flutter discrimination task. Even though participants were instructed to attend to a single sensory modality and ignore the other cue, the flutter rate in the attended modality is perceived to be similar to that of the distractor modality. Moreover, we observed similar interaction patterns regardless of stimulus type and whether the same stimulus types were experienced by both senses. Combined with earlier studies, our results suggest that the nervous system extracts and combines temporal rate information from multisensory environmental signals, regardless of stimulus type, in both the low- and high temporal frequency domains. This function likely reflects the importance of temporal frequency as a fundamental feature of our multisensory experience.

Attention periodically samples competing stimuli during binocular rivalry

The attentional sampling hypothesis suggests that attention rhythmically enhances sensory processing when attending to a single (~8 Hz), or multiple (~4 Hz) objects. Here, we investigated whether attention samples sensory representations that are not part of the conscious percept during binocular rivalry. When crossmodally cued toward a conscious image, subsequent changes in consciousness occurred at ~8 Hz, consistent with the rates of undivided attentional sampling. However, when attention was cued toward the suppressed image, changes in consciousness slowed to ~3.5 Hz, indicating the division of attention away from the conscious visual image. In the electroencephalogram, we found that at attentional sampling frequencies, the strength of inter-trial phase-coherence over fronto-temporal and parieto-occipital regions correlated with changes in perception. When cues were not task-relevant, these effects disappeared, confirming that perceptual changes were dependent upon the allocation of attention, and that attention can flexibly sample away from a conscious image in a task-dependent manner.

The Complex Interplay Between Multisensory Integration and Perceptual Awareness

The integration of information has been considered a hallmark of human consciousness, as it requires information being globally available via widespread neural interactions. Yet the complex interdependencies between multisensory integration and perceptual awareness, or consciousness, remain to be defined. While perceptual awareness has traditionally been studied in a single sense, in recent years we have witnessed a surge of interest in the role of multisensory integration in perceptual awareness. Based on a recent IMRF symposium on multisensory awareness, this review discusses three key questions from conceptual, methodological and experimental perspectives: (1) What do we study when we study multisensory awareness? (2) What is the relationship between multisensory integration and perceptual awareness? (3) Which experimental approaches are most promising to characterize multisensory awareness? We hope that this review paper will provoke lively discussions, novel experiments, and conceptual considerations to advance our understanding of the multifaceted interplay between multisensory integration and consciousness.

Upper nasal hemifield location and nonspatial auditory tones accelerate visual detection during dichoptic viewing

Visual performance is asymmetric across the visual field, but locational biases that occur during dichoptic viewing are not well understood. In this study, we characterized horizontal, vertical and naso-temporal biases in visual target detection during dichoptic stimulation and explored whether the detection was facilitated by non-spatial auditory tones associated with the target’s location.

The detection time for single monocular targets that were suppressed from view with a 10 Hz dynamic noise mask presented to the other eye was measured at the 4° intercardinal location of each eye with the breaking Continuous Flash Suppression (b-CFS) technique. Each target was either combined with a sound (i.e., high or low pitch tone) that was congruent or incongruent with its vertical location (i.e., upper or lower visual field) or presented without a sound. The results indicated faster detection of targets in the upper rather than lower visual field and faster detection of targets in the nasal than temporal hemifield of each eye. Sounds generally accelerated target detection, but the tone pitch-elevation congruency did not further enhance performance. These findings suggest that visual detection during dichoptic viewing differs from standard viewing conditions with respect to location-related perceptual biases and crossmodal modulation of visual perception. These differences should be carefully considered in experimental designs employing dichoptic stimulation techniques and in display applications that utilize dichoptic viewing.

Probing Electrophysiological Indices of Perceptual Awareness across Unisensory and Multisensory Modalities

The neural underpinnings of perceptual awareness have been extensively studied using unisensory (e.g., visual alone) stimuli. However, perception is generally multisensory, and it is unclear whether the neural architecture uncovered in these studies directly translates to the multisensory domain. Here, we use EEG to examine brain responses associated with the processing of visual, auditory, and audiovisual stimuli presented near threshold levels of detectability, with the aim of deciphering similarities and differences in the neural signals indexing the transition into perceptual awareness across vision, audition, and combined visual-auditory (multisensory) processing. More specifically, we examine (1) the presence of late evoked potentials (∼>300 msec), (2) the across-trial reproducibility, and (3) the evoked complexity associated with perceived versus nonperceived stimuli. Results reveal that, although perceived stimuli are associated with the presence of late evoked potentials across each of the examined sensory modalities, between-trial variability and EEG complexity differed for unisensory versus multisensory conditions. Whereas across-trial variability and complexity differed for perceived versus nonperceived stimuli in the visual and auditory conditions, this was not the case for the multisensory condition. Taken together, these results suggest that there are fundamental differences in the neural correlates of perceptual awareness for unisensory versus multisensory stimuli. Specifically, the work argues that the presence of late evoked potentials, as opposed to neural reproducibility or complexity, most closely tracks perceptual awareness regardless of the nature of the sensory stimulus. In addition, the current findings suggest a greater similarity between the neural correlates of perceptual awareness of unisensory (visual and auditory) stimuli when compared with multisensory stimuli.

Recent cross-modal statistical learning influences visual perceptual selection

Incoming sensory signals are often ambiguous and consistent with multiple perceptual interpretations. Information from one sensory modality can help to resolve ambiguity in another modality, but the mechanisms by which multisensory associations come to influence the contents of conscious perception are unclear. We asked whether and how novel statistical information about the coupling between sounds and images influences the early stages of awareness of visual stimuli. We exposed subjects to consistent, arbitrary pairings of sounds and images and then measured the impact of this recent passive statistical learning on subjects' initial conscious perception of a stimulus by employing binocular rivalry, a phenomenon in which incompatible images presented separately to the two eyes result in a perceptual alternation between the two images. On each trial of the rivalry test, subjects were presented with a pair of rivalrous images (one of which had been consistently paired with a specific sound during exposure while the other had not) and an accompanying sound. We found that, at the onset of binocular rivalry, an image was significantly more likely to be perceived, and was perceived for a longer duration, when it was presented with its paired sound than when presented with other sounds. Our results indicate that recently acquired multisensory information helps resolve sensory ambiguity, and they demonstrate that statistical learning is a fast, flexible mechanism that facilitates this process.

Error-related cardiac response as information for visibility judgements

Interoception provides information about the saliency of external or internal sensory events and thus may inform perceptual decision-making. Error in performance is an example of a motivationally significant internal event that evokes autonomic nervous system response resembling the orienting response: heart rate deceleration, increased skin conductance response, and pupil dilation. Here, we investigate whether error-related cardiac activity may serve as a source of information when making metacognitive judgments in an orientation discrimination backward masking task. In the first experiment, we found that the heart accelerates less after an incorrect stimuli discrimination than after a correct one. Moreover, this difference becomes more pronounced with increasing subjective visibility of the stimuli. In the second experiment, this accuracy-dependent pattern of cardiac activity was found only when participants listened to their own heartbeats, but not someone else's. We propose that decision accuracy coded in cardiac activity may be fed as a cue to subjective visibility judgments.

Does direction of walking impact binocular rivalry between competing patterns of optic flow?

When dissimilar monocular images are viewed simultaneously by the two eyes, stable binocular vision gives way to unstable vision characterized by alternations in dominance between the two images in a phenomenon called binocular rivalry. These alternations in perception reveal the existence of inhibitory interactions between neural representations associated with conflicting visual inputs. Binocular rivalry has been studied since the days of Wheatstone, but one recent strategy is to investigate its susceptibility to influences caused by one's own motor activity. This paper focused on the activity of walking, which produces an expected, characteristic direction of optic flow dependent upon the direction of one's walking. In a set of experiments, we employed virtual reality technology to present dichoptic stimuli to observers who walked forward, backward, or were sitting. Optic flow was presented to a given eye, and was sometimes congruent with the direction of walking, sometimes incongruent, and sometimes random, except when the participant was sitting. Our results indicate that, while walking had a reliable influence on rivalry dynamics, the predominance of congruent or incongruent motion did not.

The role of expectation in multisensory body representation - neural evidence

Sensory events contribute to body ownership, the feeling that the body belongs to me. However, the encoding of sensory events is not only reactive, but also proactive in that our brain generates prediction about forthcoming stimuli. In previous studies, we have shown that prediction of sensory events is a sufficient condition to induce the sense of body ownership. In this study, we investigated the underlying neural mechanisms. Participants were seated with their right arm resting upon a table just below another smaller table. Hence, the real hand was hidden from the participant's view and a life-sized rubber model of a right hand was placed on the small table in front of them. Participants observed a wooden plank while approaching - without touching - the rubber hand. We measured the phenomenology of the illusion by means of questionnaire. Neural activity was recorded by means of near-infrared spectroscopy (fNIRS). Results showed higher activation of multisensory parietal cortices in the rubber hand illusion induced by touch expectation. Furthermore, such activity was correlated with the subjective feeling of owning the rubber hand. Our results enrich current models of body ownership suggesting that our multisensory brain regions generate prediction on what could be my body and what could not. This finding might have interesting implications in all those cases in which body representation is altered, anorexia, bulimia nervosa and obesity, among others.

Consciousness is more than meets the eye: a call for a multisensory study of subjective experience

Over the last 30 years, our understanding of the neurocognitive bases of consciousness has improved, mostly through studies employing vision. While studying consciousness in the visual modality presents clear advantages, we believe that a comprehensive scientific account of subjective experience must not neglect other exteroceptive and interoceptive signals as well as the role of multisensory interactions for perceptual and self-consciousness. Here, we briefly review four distinct lines of work which converge in documenting how multisensory signals are processed across several levels and contents of consciousness. Namely, how multisensory interactions occur when consciousness is prevented because of perceptual manipulations (i.e. subliminal stimuli) or because of low vigilance states (i.e. sleep, anesthesia), how interactions between exteroceptive and interoceptive signals give rise to bodily self-consciousness, and how multisensory signals are combined to form metacognitive judgments. By describing the interactions between multisensory signals at the perceptual, cognitive, and metacognitive levels, we illustrate how stepping out the visual comfort zone may help in deriving refined accounts of consciousness, and may allow cancelling out idiosyncrasies of each sense to delineate supramodal mechanisms involved during consciousness.

Sounds can boost the awareness of visual events through attention without cross-modal integration

Cross-modal interactions can lead to enhancement of visual perception, even for visual events below awareness. However, the underlying mechanism is still unclear. Can purely bottom-up cross-modal integration break through the threshold of awareness? We used a binocular rivalry paradigm to measure perceptual switches after brief flashes or sounds which, sometimes, co-occurred. When flashes at the suppressed eye coincided with sounds, perceptual switches occurred the earliest. Yet, contrary to the hypothesis of cross-modal integration, this facilitation never surpassed the assumption of probability summation of independent sensory signals. A follow-up experiment replicated the same pattern of results using silent gaps embedded in continuous noise, instead of sounds. This manipulation should weaken putative sound-flash integration, although keep them salient as bottom-up attention cues. Additional results showed that spatial congruency between flashes and sounds did not determine the effectiveness of cross-modal facilitation, which was again not better than probability summation. Thus, the present findings fail to fully support the hypothesis of bottom-up cross-modal integration, above and beyond the independent contribution of two transient signals, as an account for cross-modal enhancement of visual events below level of awareness.

Interplay between Narrative and Bodily Self in Access to Consciousness: No Difference between Self- and Non-self Attributes

The construct of the “self” is conceived as being fundamental in promoting survival. As such, extensive studies have documented preferential processing of self-relevant stimuli. For example, attributes that relate to the self are better encoded and retrieved, and are more readily consciously perceived. The preferential processing of self-relevant information, however, appears to be especially true for physical (e.g., faces), as opposed to psychological (e.g., traits), conceptions of the self. Here, we test whether semantic attributes that participants judge as self-relevant are further processed unconsciously than attributes that were not judged as self-relevant. In Experiment 1, a continuous flash suppression paradigm was employed with “self” and “non-self” attribute words being presented subliminally, and we asked participants to categorize unseen words as either self-related or not. In a second experiment, we attempted to boost putative preferential self-processing by relation to its physical conception, that is, one’s own body. To this aim, we repeated Experiment 1 while administrating acoustic stimuli either close or far from the body, i.e., within or outside peripersonal space. Results of both Experiment 1 and 2 demonstrate no difference in breaking suppression for self and non-self words. Additionally, we found that while participants were able to process the physical location of the unseen words (above or below fixation) they were not able to categorize these as self-relevant or not. Finally, results showed that sounds presented in the extra-personal space elicited a more stringent response criterion for “self” in the process of categorizing unseen visual stimuli. This shift in criterion as a consequence of sound location was restricted to the self, as no such effect was observed in the categorization of attributes occurring above or below fixation. Overall, our findings seem to indicate that subliminally presented stimuli are not semantically processed, at least inasmuch as to be categorized as self-relevant or not. However, we do demonstrate that the distance at which acoustic stimuli are presented may alter the balance between self- and non-self biases.

Gamification in Physical Therapy: More Than Using Games

The implementation of computer games in physical therapy is motivated by characteristics such as attractiveness, motivation, and engagement, but these do not guarantee the intended therapeutic effect of the interventions. Yet, these characteristics are important variables in physical therapy interventions because they involve reward-related dopaminergic systems in the brain that are known to facilitate learning through long-term potentiation of neural connections. In this perspective we propose a way to apply game design approaches to therapy development by "designing" therapy sessions in such a way as to trigger physical and cognitive behavioral patterns required for treatment and neurological recovery. We also advocate that improving game knowledge among therapists and improving communication between therapists and game designers may lead to a novel avenue in designing applied games with specific therapeutic input, thereby making gamification in therapy a realistic and promising future that may optimize clinical practice.

Touch Accelerates Visual Awareness

To efficiently interact with the external environment, our nervous system combines information arising from different sensory modalities. Recent evidence suggests that cross-modal interactions can be automatic and even unconscious, reflecting the ecological relevance of cross-modal processing. Here, we use continuous flash suppression (CFS) to directly investigate whether haptic signals can interact with visual signals outside of visual awareness. We measured suppression durations of visual gratings rendered invisible by CFS either during visual stimulation alone or during visuo-haptic stimulation. We found that active exploration of a haptic grating congruent in orientation with the suppressed visual grating reduced suppression durations both compared with visual-only stimulation and to incongruent visuo-haptic stimulation. We also found that the facilitatory effect of touch on visual suppression disappeared when the visual and haptic gratings were mismatched in either spatial frequency or orientation. Together, these results demonstrate that congruent touch can accelerate the rise to consciousness of a suppressed visual stimulus and that this unconscious cross-modal interaction depends on visuo-haptic congruency. Furthermore, since CFS suppression is thought to occur early in visual cortical processing, our data reinforce the evidence suggesting that visuo-haptic interactions can occur at the earliest stages of cortical processing.

Vestibular signals of self-motion modulate global motion perception

Certain visual stimuli can have two possible interpretations. These perceptual interpretations may alternate stochastically, a phenomenon known as bistability. Some classes of bistable stimuli, including binocular rivalry, are sensitive to bias from input through other modalities, such as sound and touch. Here, we address the question whether bistable visual motion stimuli, known as plaids, are affected by vestibular input that is caused by self-motion. In Experiment 1, we show that a vestibular self-motion signal biases the interpretation of the bistable plaid, increasing or decreasing the likelihood of the plaid being perceived as globally coherent or transparently sliding depending on the relationship between self-motion and global visual motion directions. In Experiment 2, we find that when the vestibular direction is orthogonal to the visual direction, the vestibular self-motion signal also biases the direction of one-dimensional motion. This interaction suggests that the effect in Experiment 1 is due to the self-motion vector adding to the visual motion vectors. Together, this demonstrates that the perception of visual motion direction can be systematically affected by concurrent but uninformative and task-irrelevant vestibular input caused by self-motion.

Early Cross-modal Plasticity in Adults

It is known that, after a prolonged period of visual deprivation, the adult visual cortex can be recruited for nonvisual processing, reflecting cross-modal plasticity. Here, we investigated whether cross-modal plasticity can occur at short timescales in the typical adult brain by comparing the interaction between vision and touch during binocular rivalry before and after a brief period of monocular deprivation, which strongly alters ocular balance favoring the deprived eye. While viewing dichoptically two gratings of orthogonal orientation, participants were asked to actively explore a haptic grating congruent in orientation to one of the two rivalrous stimuli. We repeated this procedure before and after 150 min of monocular deprivation. We first confirmed that haptic stimulation interacted with vision during rivalry promoting dominance of the congruent visuo-haptic stimulus and that monocular deprivation increased the deprived eye and decreased the nondeprived eye dominance. Interestingly, after deprivation, we found that the effect of touch did not change for the nondeprived eye, whereas it disappeared for the deprived eye, which was potentiated after deprivation. The absence of visuo-haptic interaction for the deprived eye lasted for over 1 hr and was not attributable to a masking induced by the stronger response of the deprived eye as confirmed by a control experiment. Taken together, our results demonstrate that the adult human visual cortex retains a high degree of cross-modal plasticity, which can occur even at very short timescales.

The temporal frequency tuning of continuous flash suppression reveals peak suppression at very low frequencies

Continuous flash suppression (CFS) is a psychophysical technique where a rapidly changing Mondrian pattern viewed by one eye suppresses the target in the other eye for several seconds. Despite the widespread use of CFS to study unconscious visual processes, the temporal tuning of CFS suppression is currently unknown. In the present study we used spatiotemporally filtered dynamic noise as masking stimuli to probe the temporal characteristics of CFS. Surprisingly, we find that suppression in CFS peaks very prominently at approximately 1 Hz, well below the rates typically used in CFS studies (10 Hz or more). As well as a strong bias to low temporal frequencies, CFS suppression is greater for high spatial frequencies and increases with increasing masker contrast, indicating involvement of parvocellular/ventral mechanisms in the suppression process. These results are reminiscent of binocular rivalry, and unifies two phenomenon previously thought to require different explanations.

The COGs (context, object, and goals) in multisensory processing

Our understanding of how perception operates in real-world environments has been substantially advanced by studying both multisensory processes and "top-down" control processes influencing sensory processing via activity from higher-order brain areas, such as attention, memory, and expectations. As the two topics have been traditionally studied separately, the mechanisms orchestrating real-world multisensory processing remain unclear. Past work has revealed that the observer's goals gate the influence of many multisensory processes on brain and behavioural responses, whereas some other multisensory processes might occur independently of these goals. Consequently, other forms of top-down control beyond goal dependence are necessary to explain the full range of multisensory effects currently reported at the brain and the cognitive level. These forms of control include sensitivity to stimulus context as well as the detection of matches (or lack thereof) between a multisensory stimulus and categorical attributes of naturalistic objects (e.g. tools, animals). In this review we discuss and integrate the existing findings that demonstrate the importance of such goal-, object- and context-based top-down control over multisensory processing. We then put forward a few principles emerging from this literature review with respect to the mechanisms underlying multisensory processing and discuss their possible broader implications.

When audiovisual correspondence disturbs visual processing

Multisensory integration is known to create a more robust and reliable perceptual representation of one's environment. Specifically, a congruent auditory input can make a visual stimulus more salient, consequently enhancing the visibility and detection of the visual target. However, it remains largely unknown whether a congruent auditory input can also impair visual processing. In the current study, we demonstrate that temporally congruent auditory input disrupts visual processing, consequently slowing down visual target detection. More importantly, this cross-modal inhibition occurs only when the contrast of visual targets is high. When the contrast of visual targets is low, enhancement of visual target detection is observed, consistent with the prediction based on the principle of inverse effectiveness (PIE) in cross-modal integration. The switch of the behavioral effect of audiovisual interaction from benefit to cost further extends the PIE to encompass the suppressive cross-modal interaction.

An invisible touch: Body-related multisensory conflicts modulate visual consciousness

The majority of scientific studies on consciousness have focused on vision, exploring the cognitive and neural mechanisms of conscious access to visual stimuli. In parallel, studies on bodily consciousness have revealed that bodily (i.e. tactile, proprioceptive, visceral, vestibular) signals are the basis for the sense of self. However, the role of bodily signals in the formation of visual consciousness is not well understood. Here we investigated how body-related visuo-tactile stimulation modulates conscious access to visual stimuli. We used a robotic platform to apply controlled tactile stimulation to the participants' back while they viewed a dot moving either in synchrony or asynchrony with the touch on their back. Critically, the dot was rendered invisible through continuous flash suppression. Manipulating the visual context by presenting the dot moving on either a body form, or a non-bodily object we show that: (i) conflict induced by synchronous visuo-tactile stimulation in a body context is associated with a delayed conscious access compared to asynchronous visuo-tactile stimulation, (ii) this effect occurs only in the context of a visual body form, and (iii) is not due to detection or response biases. The results indicate that body-related visuo-tactile conflicts impact visual consciousness by facilitating access of non-conflicting visual information to awareness, and that these are sensitive to the visual context in which they are presented, highlighting the interplay between bodily signals and visual experience.

Melodic sound enhances visual awareness of congruent musical notes, but only if you can read music

Predictive influences of auditory information on resolution of visual competition were investigated using music, whose visual symbolic notation is familiar only to those with musical training. Results from two experiments using different experimental paradigms revealed that melodic congruence between what is seen and what is heard impacts perceptual dynamics during binocular rivalry. This bisensory interaction was observed only when the musical score was perceptually dominant, not when it was suppressed from awareness, and it was observed only in people who could read music. Results from two ancillary experiments showed that this effect of congruence cannot be explained by differential patterns of eye movements or by differential response sluggishness associated with congruent score/melody combinations. Taken together, these results demonstrate robust audiovisual interaction based on high-level, symbolic representations and its predictive influence on perceptual dynamics during binocular rivalry.

Suppressed visual looming stimuli are not integrated with auditory looming signals: Evidence from continuous flash suppression

Previous studies using binocular rivalry have shown that signals in a modality other than the visual can bias dominance durations depending on their congruency with the rivaling stimuli. More recently, studies using continuous flash suppression (CFS) have reported that multisensory integration influences how long visual stimuli remain suppressed. In this study, using CFS, we examined whether the contrast thresholds for detecting visual looming stimuli are influenced by a congruent auditory stimulus. In Experiment 1, we show that a looming visual stimulus can result in lower detection thresholds compared to a static concentric grating, but that auditory tone pips congruent with the looming stimulus did not lower suppression thresholds any further. In Experiments 2, 3, and 4, we again observed no advantage for congruent multisensory stimuli. These results add to our understanding of the conditions under which multisensory integration is possible, and suggest that certain forms of multisensory integration are not evident when the visual stimulus is suppressed from awareness using CFS.

Intersensory selective attention and temporal orienting operate in parallel and are instantiated in spatially distinct sensory and motor cortices

Knowledge about the sensory modality in which a forthcoming event might occur permits anticipatory intersensory attention. Information as to when exactly an event occurs enables temporal orienting. Intersensory and temporal attention mechanisms are often deployed simultaneously, but as yet it is unknown whether these processes operate interactively or in parallel. In this human electroencephalography study, we manipulated intersensory attention and temporal orienting in the same paradigm. A continuous stream of bisensory visuo-tactile inputs was presented, and a preceding auditory cue indicated to which modality participants should attend (visual or tactile). Temporal orienting was manipulated blockwise by presenting stimuli either at regular or irregular intervals. Using linear beamforming, we examined neural oscillations at virtual channels in sensory and motor cortices. Both attentional processes simultaneously modulated the power of anticipatory delta- and beta-band oscillations, as well as delta-band phase coherence. Modulations in sensory cortices reflected intersensory attention, indicative of modality-specific gating mechanisms. Modulations in motor and partly in somatosensory cortex reflected temporal orienting, indicative of a supramodal preparatory mechanism. We found no evidence for interactions between intersensory attention and temporal orienting, suggesting that these two mechanisms act in parallel and largely independent of each other in sensory and motor cortices.

Congruent tactile stimulation reduces the strength of visual suppression during binocular rivalry

Presenting different images to each eye triggers ‘binocular rivalry’ in which one image is visible and the other suppressed, with the visible image alternating every second or so. We previously showed that binocular rivalry between cross-oriented gratings is altered when the fingertip explores a grooved stimulus aligned with one of the rivaling gratings: the matching visual grating's dominance duration was lengthened and its suppression duration shortened. In a more robust test, we here measure visual contrast sensitivity during rivalry dominance and suppression, with and without exploration of the grooved surface, to determine if rivalry suppression strength is modulated by touch. We find that a visual grating undergoes 45% less suppression when observers touch an aligned grating, compared to a cross-oriented one. Touching an aligned grating also improved visual detection thresholds for the ‘invisible’ suppressed grating by 2.4 dB, relative to a vision-only condition. These results show that congruent haptic stimulation prevents a visual stimulus from becoming deeply suppressed in binocular rivalry. Moreover, because congruent touch acted on the phenomenally invisible grating, this visuo-haptic interaction must precede awareness and likely occurs early in visual processing.

A spatially collocated sound thrusts a flash into awareness

To interact effectively with the environment the brain integrates signals from multiple senses. It is currently unclear to what extent spatial information can be integrated across different senses in the absence of awareness. Combining dynamic continuous flash suppression (CFS) and spatial audiovisual stimulation, the current study investigated whether a sound facilitates a concurrent visual flash to elude flash suppression and enter perceptual awareness depending on audiovisual spatial congruency. Our results demonstrate that a concurrent sound boosts unaware visual signals into perceptual awareness. Critically, this process depended on the spatial congruency of the auditory and visual signals pointing towards low level mechanisms of audiovisual integration. Moreover, the concurrent sound biased the reported location of the flash as a function of flash visibility. The spatial bias of sounds on reported flash location was strongest for flashes that were judged invisible. Our results suggest that multisensory integration is a critical mechanism that enables signals to enter conscious perception.

A phonologically congruent sound boosts a visual target into perceptual awareness

Capacity limitations of attentional resources allow only a fraction of sensory inputs to enter our awareness. Most prominently, in the attentional blink the observer often fails to detect the second of two rapidly successive targets that are presented in a sequence of distractor items. To investigate how auditory inputs enable a visual target to escape the attentional blink, this study presented the visual letter targets T1 and T2 together with phonologically congruent or incongruent spoken letter names. First, a congruent relative to an incongruent sound at T2 rendered visual T2 more visible. Second, this T2 congruency effect was amplified when the sound was congruent at T1 as indicated by a T1 congruency × T2 congruency interaction. Critically, these effects were observed both when the sounds were presented in synchrony with and prior to the visual target letters suggesting that the sounds may increase visual target identification via multiple mechanisms such as audiovisual priming or decisional interactions. Our results demonstrate that a sound around the time of T2 increases subjects' awareness of the visual target as a function of T1 and T2 congruency. Consistent with Bayesian causal inference, the brain may thus combine (1) prior congruency expectations based on T1 congruency and (2) phonological congruency cues provided by the audiovisual inputs at T2 to infer whether auditory and visual signals emanate from a common source and should hence be integrated for perceptual decisions.