Attention regulates the plasticity of multisensory timing

Neural correlates of temporal recalibration to delayed auditory feedback of active and passive movements

When we perform an action, its sensory outcomes usually follow shortly after. This characteristic temporal relationship aids in distinguishing self- from externally generated sensory input. To preserve this ability under dynamically changing environmental conditions, our expectation of the timing between action and outcome must be able to recalibrate, for example, when the outcome is consistently delayed. Until now, it remains unclear whether this process, known as sensorimotor temporal recalibration, can be specifically attributed to recalibration of sensorimotor (action-outcome) predictions, or whether it may be partly due to the recalibration of expectations about the intersensory (e.g., audio-tactile) timing. Therefore, we investigated the behavioral and neural correlates of temporal recalibration and differences in sensorimotor and intersensory contexts. During fMRI, subjects were exposed to delayed or undelayed tones elicited by actively or passively generated button presses. While recalibration of the expected intersensory timing (i.e., between the tactile sensation during the button movement and the tones) can be expected to occur during both active and passive movements, recalibration of sensorimotor predictions should be limited to active movement conditions. Effects of this procedure on auditory temporal perception and the modality-transfer to visual perception were tested in a delay detection task. Across both contexts, we found recalibration to be associated with activations in hippocampus and cerebellum. Context-dependent differences emerged in terms of stronger behavioral recalibration effects in sensorimotor conditions and were captured by differential activation pattern in frontal cortices, cerebellum, and sensory processing regions. These findings highlight the role of the hippocampus in encoding and retrieving newly acquired temporal stimulus associations during temporal recalibration. Furthermore, recalibration-related activations in the cerebellum may reflect the retention of multiple representations of temporal stimulus associations across both contexts. Finally, we showed that sensorimotor predictions modulate recalibration-related processes in frontal, cerebellar, and sensory regions, which potentially account for the perceptual advantage of sensorimotor versus intersensory temporal recalibration.

Musical training refines audiovisual integration but does not influence temporal recalibration

When the brain is exposed to a temporal asynchrony between the senses, it will shift its perception of simultaneity towards the previously experienced asynchrony (temporal recalibration). It is unknown whether recalibration depends on how accurately an individual integrates multisensory cues or on experiences they have had over their lifespan. Hence, we assessed whether musical training modulated audiovisual temporal recalibration. Musicians (n = 20) and non-musicians (n = 18) made simultaneity judgements to flash-tone stimuli before and after adaptation to asynchronous (± 200 ms) flash-tone stimuli. We analysed these judgements via an observer model that described the left and right boundaries of the temporal integration window (decisional criteria) and the amount of sensory noise that affected these judgements. Musicians’ boundaries were narrower (closer to true simultaneity) than non-musicians’, indicating stricter criteria for temporal integration, and they also exhibited enhanced sensory precision. However, while both musicians and non-musicians experienced cumulative and rapid recalibration, these recalibration effects did not differ between the groups. Unexpectedly, cumulative recalibration was caused by auditory-leading but not visual-leading adaptation. Overall, these findings suggest that the precision with which observers perceptually integrate audiovisual temporal cues does not predict their susceptibility to recalibration.

The impact of cerebellar transcranial direct current stimulation (tDCS) on sensorimotor and inter-sensory temporal recalibration

The characteristic temporal relationship between actions and their sensory outcomes allows us to distinguish self- from externally generated sensory events. However, the complex sensory environment can cause transient delays between action and outcome calling for flexible recalibration of predicted sensorimotor timing. Since the neural underpinnings of this process are largely unknown this study investigated the involvement of the cerebellum by means of cerebellar transcranial direct current stimulation (ctDCS). While receiving anodal, cathodal, dual-hemisphere or sham ctDCS, in an adaptation phase, participants were exposed to constant delays of 150 ms between actively or passively generated button presses and visual sensory outcomes. Recalibration in the same (visual outcome) and in another sensory modality (auditory outcome) was assessed in a subsequent test phase during which variable delays between button press and visual or auditory outcome had to be detected. Results indicated that temporal recalibration occurred in audition after anodal ctDCS while it was absent in vision. As the adaptation modality was visual, effects in audition suggest that recalibration occurred on a supra-modal level. In active conditions, anodal ctDCS improved sensorimotor recalibration at the delay level closest to the adaptation delay, suggesting a precise cerebellar-dependent temporal recalibration mechanism. In passive conditions, the facilitation of inter-sensory recalibration by anodal ctDCS was overall stronger and tuned to larger delays. These findings point to a role of the cerebellum in supra-modal temporal recalibration across sensorimotor and perceptual domains, but the differential manifestation of the effect across delay levels in active and passive conditions points to differences in the underlying mechanisms depending on the availability of action-based predictions. Furthermore, these results suggest that anodal ctDCS can be a promising tool for facilitating effects of temporal recalibration in sensorimotor and inter-sensory contexts.

The Role of Awareness on Motor-Sensory Temporal Recalibration

Temporal recalibration (TR) may arise to realign asynchronous stimuli after exposure to a short, constant delay between voluntary movement and sensory stimulus. The objective of this study was to determine if awareness of the temporal lag between a motor response (i.e., a keypress) and a sensory event (i.e., a visual flash) is necessary for TR to occur. We further investigated whether manipulating the required motor and perceptual judgment tasks modified the influence of awareness on TR. Participants (n = 48) were randomly divided between two groups (Group 1: Aware and Group 2: Unaware). The Aware group was told of the temporal lag between their keypress and visual flash at the beginning of the experiment, whereas the Unaware group was not. All participants completed eight blocks of trials, in which the motor task (single or repetitive tap), perceptual judgment task (judging the temporal order of the keypress in relation to the visual flash or judging whether the two stimuli were simultaneous or not), and fixed temporal lag between keypress and visual flash (0 or 100 ms) varied. TR was determined by comparing judgments between corresponding blocks of trials in which the temporal lag was 0 or 100 ms. Results revealed that both the Aware and Unaware groups demonstrated a similar magnitude of TR across all motor and perceptual judgment tasks, such that the magnitude of TR did not vary between Aware and Unaware participants. These results suggest that awareness of a temporal lag does not influence the magnitude of TR achieved and that motor and perceptual judgment task demands do not modulate the influence of awareness on TR.

Perception of Happening: How the Brain Deals with the No-History Problem

In physics, the temporal dimension has units of infinitesimally brief duration. Given this, how is it possible to perceive things, such as motion, music, and vibrotactile stimulation, that involve extension across many units of time? To address this problem, it is proposed that there is what is termed an "information construct of happening" (ICOH), a simultaneous representation of recent, temporally differentiated perceptual information on the millisecond time scale. The main features of the ICOH are (i) time marking, semantic labeling of all information in the ICOH with ordinal temporal information and distance from what is informationally identified as the present moment, (ii) vector informational features that specify kind, direction, and rate of change for every feature in a percept, and (iii) connectives, information relating vector informational features at adjacent temporal locations in the ICOH. The ICOH integrates products of perceptual processing with recent historical information in sensory memory on the subsecond time scale. Perceptual information about happening in informational sensory memory is encoded in semantic form that preserves connected semantic trails of vector and timing information. The basic properties of the ICOH must be supported by a general and widespread timing mechanism that generates ordinal and interval timing information and it is suggested that state-dependent networks may suffice for that purpose. Happening, therefore, is perceived at a moment and is constituted by an information structure of connected recent historical information.

Effects of Audiovisual Presentations on Visual Localization Errors: One or Several Multisensory Mechanisms?

The present study examines the extent to which temporal and spatial properties of sound modulate visual motion processing in spatial localization tasks. Participants were asked to locate the place at which a moving visual target unexpectedly vanished. Across different tasks, accompanying sounds were factorially varied within subjects as to their onset and offset times and/or positions relative to visual motion. Sound onset had no effect on the localization error. Sound offset was shown to modulate the perceived visual offset location, both for temporal and spatial disparities. This modulation did not conform to attraction toward the timing or location of the sounds but, demonstrably in the case of temporal disparities, to bimodal enhancement instead. Favorable indications to a contextual effect of audiovisual presentations on interspersed visual-only trials were also found. The short sound-leading offset asynchrony had equivalent benefits to audiovisual offset synchrony, suggestive of the involvement of early-level mechanisms, constrained by a temporal window, at these conditions. Yet, we tentatively hypothesize that the whole of the results and how they compare with previous studies requires the contribution of additional mechanisms, including learning-detection of auditory-visual associations and cross-sensory spread of endogenous attention.

The perceived present: What is it, and what is it there for?

It is proposed that the perceived present is not a moment in time, but an information structure comprising an integrated set of products of perceptual processing. All information in the perceived present carries an informational time marker identifying it as "present". This marker is exclusive to information in the perceived present. There are other kinds of time markers, such as ordinality ("this stimulus occurred before that one") and duration ("this stimulus lasted for 50 ms"). These are different from the "present" time marker and may be attached to information regardless of whether it is in the perceived present or not. It is proposed that the perceived present is a very short-term and very high-capacity holding area for perceptual information. The maximum holding time for any given piece of information is ~100 ms: This is affected by the need to balance the value of informational persistence for further processing against the problem of obsolescence of the information. The main function of the perceived present is to facilitate access by other specialized, automatic processes.

Alpha Activity Reflects the Magnitude of an Individual Bias in Human Perception

Biases in sensory perception can arise from both experimental manipulations and personal trait-like features. These idiosyncratic biases and their neural underpinnings are often overlooked in studies on the physiology underlying perception. A potential candidate mechanism reflecting such idiosyncratic biases could be spontaneous alpha band activity, a prominent brain rhythm known to influence perceptual reports in general. Using a temporal order judgment task, we here tested the hypothesis that alpha power reflects the overcoming of an idiosyncratic bias. Importantly, to understand the interplay between idiosyncratic biases and contextual (temporary) biases induced by experimental manipulations, we quantified this relation before and after temporal recalibration. Using EEG recordings in human participants (male and female), we find that prestimulus frontal alpha power correlates with the tendency to respond relative to an own idiosyncratic bias, with stronger α leading to responses matching the bias. In contrast, alpha power does not predict response correctness. These results also hold after temporal recalibration and are specific to the alpha band, suggesting that alpha band activity reflects, directly or indirectly, processes that help to overcome an individual's momentary bias in perception. We propose that combined with established roles of parietal α in the encoding of sensory information frontal α reflects complementary mechanisms influencing perceptual decisions.SIGNIFICANCE STATEMENT The brain is a biased organ, frequently generating systematically distorted percepts of the world, leading each of us to evolve in our own subjective reality. However, such biases are often overlooked or considered noise when studying the neural mechanisms underlying perception. We show that spontaneous alpha band activity predicts the degree of biasedness of human choices in a time perception task, suggesting that alpha activity indexes processes needed to overcome an individual's idiosyncratic bias. This result provides a window onto the neural underpinnings of subjective perception, and offers the possibility to quantify or manipulate such priors in future studies.

Attention Gates the Selective Encoding of Duration

The abundance of temporal information in our environment calls for the effective selection and utilization of temporal information that is relevant for our behavior. Here we investigated whether visual attention gates the selective encoding of relevant duration information when multiple sources of duration information are present. We probed the encoding of duration by using a duration-adaptation paradigm. Participants adapted to two concurrently presented streams of stimuli with different durations, while detecting oddballs in one of the streams. We measured the resulting duration after-effect (DAE) and found that the DAE reflects stronger relative adaptation to attended durations, compared to unattended durations. Additionally, we demonstrate that unattended durations do not contribute to the measured DAE. These results suggest that attention plays a crucial role in the selective encoding of duration: attended durations are encoded, while encoding of unattended durations is either weak or absent.

Still no Evidence for Sustained Effects of Multisensory Integration of Duration

In studies on temporal order perception, immediate as well as sustained effects of multisensory integration have been demonstrated repeatedly. Regarding duration perception, the corresponding literature reports clear immediate effects of multisensory integration, but evidence on sustained effects of multisensory duration integration is scarce. In fact, a single study [Heron, J. et al. (2013). A neural hierarchy for illusions of time: Duration adaptation precedes multisensory integration, J. Vis. 13, 1-12.] investigated adaptation to multisensory conflicting intervals, and found no sustained effects of the audiovisual conflict on perceived duration of subsequently presented unimodal visual intervals. In two experiments, we provide independent evidence in support of this finding. In Experiment 1, we demonstrate that adaptation to audiovisual conflict does not alter perceived duration of subsequently presented visual test intervals. Thus, replicating the results of Heron et al. (2013), we observed no sustained effect of multisensory duration integration. However, one might argue that the prolonged exposure to consistent multisensory conflict might have prevented or hampered multisensory integration per se. In Experiment 2, we rule out this alternative explanation by showing that multisensory integration of audiovisual conflicting intervals is still effective after exposure to audiovisual conflict. This further strengthens the conclusion that multisensory integration of interval duration affects perception in an immediate, but not in a sustained manner.

Spatial and frequency specificity of the ventriloquism aftereffect revisited

Exposure to audiovisual stimuli with a consistent spatial misalignment seems to result in a recalibration of unisensory auditory spatial representations. The previous studies have suggested that this so-called ventriloquism aftereffect is confined to the trained region of space, but yielded inconsistent results as to whether or not recalibration generalizes to untrained sound frequencies. Here, we reassessed the spatial and frequency specificity of the ventriloquism aftereffect by testing whether auditory spatial perception can be independently recalibrated for two different sound frequencies and/or at two different spatial locations. Recalibration was confined to locations within the trained hemifield, suggesting that spatial representations were independently adjusted for the two hemifields. The frequency specificity of the ventriloquism aftereffect depended on the presence or the absence of conflicting audiovisual adaptation stimuli within the same hemifield. Moreover, adaptation of two different sound frequencies in opposite directions (leftward vs. rightward) resulted in a selective suppression of leftward recalibration, even when the adapting stimuli were presented in different hemifields. Thus, instead of representing a fixed stimulus-driven process, cross-modal recalibration seems to critically depend on the sensory context and takes into account inconsistencies in the cross-modal input.

Audio-motor but not visuo-motor temporal recalibration speeds up sensory processing

Perception of synchrony between one's own action (a finger tap) and the sensory feedback thereof (a visual flash or an auditory pip) can be recalibrated after exposure to an artificially inserted delay between them (temporal recalibration effect: TRE). TRE might be mediated by a compensatory shift of motor timing (when did I tap?) and/or the sensory timing of the feedback (when did I hear/see the feedback?). To examine this, we asked participants to voluntarily tap their index finger at a constant pace while receiving visual or auditory feedback (a flash or pip) that was either synced or somewhat delayed relative to the tap. Following this exposure phase, they then performed a simple reaction time (RT) task to measure the sensory timing of the exposure stimulus, and a sensorimotor synchronization (SMS) task (tapping in synchrony with a flash or pip as pacing stimulus) to measure the point of subjective synchrony between the tap and pacing stimulus. The results showed that after exposure to delayed auditory feedback, participants tapped earlier (~21.5 ms) relative to auditory pacing stimuli (= temporal recalibration) and reacted faster (~5.6 ms) to auditory stimuli. For visual exposure and test stimuli, there were no such compensatory effects. These results indicate that adjustments of audio-motor synchrony can to some extent be explained by a change in the speed of auditory sensory processing. We discuss this in terms of an attentional modulation of sensory processing.

Age-Related Changes in Sensorimotor Temporal Binding

The causal relationship between a voluntary movement and a sensory event is crucial for experiencing agency. Sensory events must occur within a certain delay from a voluntary movement to be perceived as self-generated. Therefore, temporal sensitivity, i.e., the ability to discriminate temporal asynchronies between motor and sensory events, is important for sensorimotor binding. Moreover, differences in the physical propagation of external stimuli can sometimes challenge sensorimotor binding, generating illusory asynchrony. To overcome this problem, the brain adjusts the perceptual timing of sensory and motor events. This mechanism, named sensorimotor recalibration, helps keeping causality judgments accurate. As humans age, the broad decline in sensory and motor processing may reduce temporal sensitivity, and compromise sensorimotor recalibration. In the current study, we investigated the effect of aging on sensorimotor temporal binding by measuring changes in both temporal sensitivity and recalibration. Young and elderly adults were exposed to a prolonged physical delay between a voluntary movement (a keypress) and its perceptual consequence (a visual stimulus). Before and after this exposure, participants performed a sensorimotor temporal order judgment (TOJ) task. As expected, elderly adults showed reduced sensorimotor recalibration and sensitivity as compared to young adults, suggesting that aging affects sensorimotor temporal binding.

Spatial modulation of motor-sensory recalibration in early deaf individuals

Audition dominates other senses in temporal processing, and in the absence of auditory cues, temporal perception can be compromised. Moreover, after auditory deprivation, visual attention is selectively enhanced for peripheral visual stimuli. In this study, we assessed whether early hearing loss affects motor-sensory recalibration, the ability to adjust the timing of an action and its sensory effect based on the recent experience. Early deaf participants and hearing controls were asked to discriminate the temporal order between a motor action (a keypress) and a visual stimulus (a white circle) before and after adaptation to a delay between the two events. To examine the effects of spatial modulation, we presented visual stimuli in both central and peripheral visual fields. Results showed overall higher temporal JNDs (Just Noticeable Difference) for deaf participants as compared to hearing controls suggesting that the auditory information is important for the calibration of motor-sensory timing. Adaptation to a motor-sensory delay induced distinctive effect in the two groups of participants, with hearing controls showing a recalibration effect for central stimuli only whereas deaf individuals for peripheral visual stimuli only. Our results suggest that auditory deprivation affects motor-sensory recalibration and that the mechanism underlying motor-sensory recalibration is susceptible to spatial modulation.

Grouping and Segregation of Sensory Events by Actions in Temporal Audio-Visual Recalibration

Perception in multi-sensory environments involves both grouping and segregation of events across sensory modalities. Temporal coincidence between events is considered a strong cue to resolve multisensory perception. However, differences in physical transmission and neural processing times amongst modalities complicate this picture. This is illustrated by cross-modal recalibration, whereby adaptation to audio-visual asynchrony produces shifts in perceived simultaneity. Here, we examined whether voluntary actions might serve as a temporal anchor to cross-modal recalibration in time. Participants were tested on an audio-visual simultaneity judgment task after an adaptation phase where they had to synchronize voluntary actions with audio-visual pairs presented at a fixed asynchrony (vision leading or vision lagging). Our analysis focused on the magnitude of cross-modal recalibration to the adapted audio-visual asynchrony as a function of the nature of the actions during adaptation, putatively fostering cross-modal grouping or, segregation. We found larger temporal adjustments when actions promoted grouping than segregation of sensory events. However, a control experiment suggested that additional factors, such as attention to planning/execution of actions, could have an impact on recalibration effects. Contrary to the view that cross-modal temporal organization is mainly driven by external factors related to the stimulus or environment, our findings add supporting evidence for the idea that perceptual adjustments strongly depend on the observer's inner states induced by motor and cognitive demands.

A model-based comparison of three theories of audiovisual temporal recalibration

Observers change their audio-visual timing judgements after exposure to asynchronous audiovisual signals. The mechanism underlying this temporal recalibration is currently debated. Three broad explanations have been suggested. According to the first, the time it takes for sensory signals to propagate through the brain has changed. The second explanation suggests that decisional criteria used to interpret signal timing have changed, but not time perception itself. A final possibility is that a population of neurones collectively encode relative times, and that exposure to a repeated timing relationship alters the balance of responses in this population. Here, we simplified each of these explanations to its core features in order to produce three corresponding six-parameter models, which generate contrasting patterns of predictions about how simultaneity judgements should vary across four adaptation conditions: No adaptation, synchronous adaptation, and auditory leading/lagging adaptation. We tested model predictions by fitting data from all four conditions simultaneously, in order to assess which model/explanation best described the complete pattern of results. The latency-shift and criterion-change models were better able to explain results for our sample as a whole. The population-code model did, however, account for improved performance following adaptation to a synchronous adapter, and best described the results of a subset of observers who reported least instances of synchrony.

The effects of attention on the temporal integration of multisensory stimuli

In unisensory contexts, spatially-focused attention tends to enhance perceptual processing. How attention influences the processing of multisensory stimuli, however, has been of much debate. In some cases, attention has been shown to be important for processes related to the integration of audio-visual stimuli, but in other cases such processes have been reported to occur independently of attention. To address these conflicting results, we performed three experiments to examine how attention interacts with a key facet of multisensory processing: the temporal window of integration (TWI). The first two experiments used a novel cued-spatial-attention version of the bounce/stream illusion, wherein two moving visual stimuli with intersecting paths tend to be perceived as bouncing off rather than streaming through each other when a brief sound occurs near in time. When the task was to report whether the visual stimuli appeared to bounce or stream, attention served to narrow this measure of the TWI and bias perception toward “streaming”. When the participants’ task was to explicitly judge the simultaneity of the sound with the intersection of the moving visual stimuli, however, the results were quite different. Specifically, attention served to mainly widen the TWI, increasing the likelihood of simultaneity perception, while also substantially increasing the simultaneity judgment accuracy when the stimuli were actually physically simultaneous. Finally, in Experiment 3, where the task was to judge the simultaneity of a simple, temporally discrete, flashed visual stimulus and the same brief tone pip, attention had no effect on the measured TWI. These results highlight the flexibility of attention in enhancing multisensory perception and show that the effects of attention on multisensory processing are highly dependent on the task demands and observer goals.

Hysteresis in audiovisual synchrony perception

The effect of stimulation history on the perception of a current event can yield two opposite effects, namely: adaptation or hysteresis. The perception of the current event thus goes in the opposite or in the same direction as prior stimulation, respectively. In audiovisual (AV) synchrony perception, adaptation effects have primarily been reported. Here, we tested if perceptual hysteresis could also be observed over adaptation in AV timing perception by varying different experimental conditions. Participants were asked to judge the synchrony of the last (test) stimulus of an AV sequence with either constant or gradually changing AV intervals (constant and dynamic condition, respectively). The onset timing of the test stimulus could be cued or not (prospective vs. retrospective condition, respectively). We observed hysteretic effects for AV synchrony judgments in the retrospective condition that were independent of the constant or dynamic nature of the adapted stimuli; these effects disappeared in the prospective condition. The present findings suggest that knowing when to estimate a stimulus property has a crucial impact on perceptual simultaneity judgments. Our results extend beyond AV timing perception, and have strong implications regarding the comparative study of hysteresis and adaptation phenomena.

Multiple concurrent temporal recalibrations driven by audiovisual stimuli with apparent physical differences

Out-of-synchrony experiences can easily recalibrate one's subjective simultaneity point in the direction of the experienced asynchrony. Although temporal adjustment of multiple audiovisual stimuli has been recently demonstrated to be spatially specific, perceptual grouping processes that organize separate audiovisual stimuli into distinctive "objects" may play a more important role in forming the basis for subsequent multiple temporal recalibrations. We investigated whether apparent physical differences between audiovisual pairs that make them distinct from each other can independently drive multiple concurrent temporal recalibrations regardless of spatial overlap. Experiment 1 verified that reducing the physical difference between two audiovisual pairs diminishes the multiple temporal recalibrations by exposing observers to two utterances with opposing temporal relationships spoken by one single speaker rather than two distinct speakers at the same location. Experiment 2 found that increasing the physical difference between two stimuli pairs can promote multiple temporal recalibrations by complicating their non-temporal dimensions (e.g., disks composed of two rather than one attribute and tones generated by multiplying two frequencies); however, these recalibration aftereffects were subtle. Experiment 3 further revealed that making the two audiovisual pairs differ in temporal structures (one transient and one gradual) was sufficient to drive concurrent temporal recalibration. These results confirm that the more audiovisual pairs physically differ, especially in temporal profile, the more likely multiple temporal perception adjustments will be content-constrained regardless of spatial overlap. These results indicate that multiple temporal recalibrations are based secondarily on the outcome of perceptual grouping processes.

Selective attention modulates the direction of audio-visual temporal recalibration

Temporal recalibration of cross-modal synchrony has been proposed as a mechanism to compensate for timing differences between sensory modalities. However, far from the rich complexity of everyday life sensory environments, most studies to date have examined recalibration on isolated cross-modal pairings. Here, we hypothesize that selective attention might provide an effective filter to help resolve which stimuli are selected when multiple events compete for recalibration. We addressed this question by testing audio-visual recalibration following an adaptation phase where two opposing audio-visual asynchronies were present. The direction of voluntary visual attention, and therefore to one of the two possible asynchronies (flash leading or flash lagging), was manipulated using colour as a selection criterion. We found a shift in the point of subjective audio-visual simultaneity as a function of whether the observer had focused attention to audio-then-flash or to flash-then-audio groupings during the adaptation phase. A baseline adaptation condition revealed that this effect of endogenous attention was only effective toward the lagging flash. This hints at the role of exogenous capture and/or additional endogenous effects producing an asymmetry toward the leading flash. We conclude that selective attention helps promote selected audio-visual pairings to be combined and subsequently adjusted in time but, stimulus organization exerts a strong impact on recalibration. We tentatively hypothesize that the resolution of recalibration in complex scenarios involves the orchestration of top-down selection mechanisms and stimulus-driven processes.

Intersensory binding across space and time: a tutorial review

Spatial ventriloquism refers to the phenomenon that a visual stimulus such as a flash can attract the perceived location of a spatially discordant but temporally synchronous sound. An analogous example of mutual attraction between audition and vision has been found in the temporal domain, where temporal aspects of a visual event, such as its onset, frequency, or duration, can be biased by a slightly asynchronous sound. In this review, we examine various manifestations of spatial and temporal attraction between the senses (both direct effects and aftereffects), and we discuss important constraints on the occurrence of these effects. Factors that potentially modulate ventriloquism-such as attention, synesthetic correspondence, and other cognitive factors-are described. We trace theories and models of spatial and temporal ventriloquism, from the traditional unity assumption and modality appropriateness hypothesis to more recent Bayesian and neural network approaches. Finally, we summarize recent evidence probing the underlying neural mechanisms of spatial and temporal ventriloquism.

Concurrent sensorimotor temporal recalibration to different lags for the left and right hand

Perception of temporal synchrony between one’s own action and the sensory feedback of that action is quite flexible. We examined whether sensorimotor temporal recalibration (TR) involves central or motor-specific components by concurrently exposing the left and right hands to different lags. The experiment was composed of a pre-test, an adaptation phase, and a post-test. During the adaptation phase, participants tapped their left and right index fingers in alternating fashion while each tap induced an auditory feedback signal (a short click sound). One hand was exposed to a long delay between the tap and the sound (~150 ms), while the other hand was exposed to a subjective no-delay (~50 ms). Before and after the adaptation phase (the pre- and post-test), participants tried to tap in synchrony with pacer tones (ISI = 1000 ms). The results showed that the hand that was exposed to the delayed sound corrected for this delay by tapping earlier (a larger anticipation error) than the no-delay hand, indicating TR. Different amounts of TR were found when the left and right hand were concurrently exposed to the same versus different delays. With different exposure- delays for the two hands, there was a TR even for the hand that did not experience any delay in the feedback signal. However, it is not the case with the same exposure delay for the two hands. TR of the hand that experienced delayed feedback also occurred faster and was more complete (~40% greater than that of the hand with no subjective delay) if the two hands were exposed to the same rather than different delays (~20% greater than that of the hand with no subjective delay). These results suggest the existence of cross-talk between the hands, where both central and motor-specific components might be involved.

A neural hierarchy for illusions of time: duration adaptation precedes multisensory integration

Perceived time is inherently malleable. For example, adaptation to relatively long or short sensory events leads to a repulsive aftereffect such that subsequent events appear to be contracted or expanded (duration adaptation). Perceived visual duration can also be distorted via concurrent presentation of discrepant auditory durations (multisensory integration). The neural loci of both distortions remain unknown. In the current study we use a psychophysical approach to establish their relative positioning within the sensory processing hierarchy. We show that audiovisual integration induces marked distortions of perceived visual duration. We proceed to use these distorted durations as visual adapting stimuli yet find subsequent visual duration aftereffects to be consistent with physical rather than perceived visual duration. Conversely, the concurrent presentation of adapted auditory durations with nonadapted visual durations results in multisensory integration patterns consistent with perceived, rather than physical, auditory duration. These results demonstrate that recent sensory history modifies human duration perception prior to the combination of temporal information across sensory modalities and provides support for adaptation mechanisms mediated by duration selective neurons situated in early areas of the visual and auditory nervous system (Aubie, Sayegh, & Faure, 2012; Duysens, Schaafsma, & Orban, 1996; Leary, Edwards, & Rose, 2008).

Recalibration of the multisensory temporal window of integration results from changing task demands

The notion of the temporal window of integration, when applied in a multisensory context, refers to the breadth of the interval across which the brain perceives two stimuli from different sensory modalities as synchronous. It maintains a unitary perception of multisensory events despite physical and biophysical timing differences between the senses. The boundaries of the window can be influenced by attention and past sensory experience. Here we examined whether task demands could also influence the multisensory temporal window of integration. We varied the stimulus onset asynchrony between simple, short-lasting auditory and visual stimuli while participants performed two tasks in separate blocks: a temporal order judgment task that required the discrimination of subtle auditory-visual asynchronies, and a reaction time task to the first incoming stimulus irrespective of its sensory modality. We defined the temporal window of integration as the range of stimulus onset asynchronies where performance was below 75% in the temporal order judgment task, as well as the range of stimulus onset asynchronies where responses showed multisensory facilitation (race model violation) in the reaction time task. In 5 of 11 participants, we observed audio-visual stimulus onset asynchronies where reaction time was significantly accelerated (indicating successful integration in this task) while performance was accurate in the temporal order judgment task (indicating successful segregation in that task). This dissociation suggests that in some participants, the boundaries of the temporal window of integration can adaptively recalibrate in order to optimize performance according to specific task demands.

Audio-Visual Temporal Recalibration Can be Constrained by Content Cues Regardless of Spatial Overlap

It has now been well established that the point of subjective synchrony for audio and visual events can be shifted following exposure to asynchronous audio-visual presentations, an effect often referred to as temporal recalibration. Recently it was further demonstrated that it is possible to concurrently maintain two such recalibrated estimates of audio-visual temporal synchrony. However, it remains unclear precisely what defines a given audio-visual pair such that it is possible to maintain a temporal relationship distinct from other pairs. It has been suggested that spatial separation of the different audio-visual pairs is necessary to achieve multiple distinct audio-visual synchrony estimates. Here we investigated if this is necessarily true. Specifically, we examined whether it is possible to obtain two distinct temporal recalibrations for stimuli that differed only in featural content. Using both complex (audio visual speech; see Experiment 1) and simple stimuli (high and low pitch audio matched with either vertically or horizontally oriented Gabors; see Experiment 2) we found concurrent, and opposite, recalibrations despite there being no spatial difference in presentation location at any point throughout the experiment. This result supports the notion that the content of an audio-visual pair alone can be used to constrain distinct audio-visual synchrony estimates regardless of spatial overlap.

Interval timing in children: effects of auditory and visual pacing stimuli and relationships with reading and attention variables

Motor timing tasks have been employed in studies of neurodevelopmental disorders such as developmental dyslexia and ADHD, where they provide an index of temporal processing ability. Investigations of these disorders have used different stimulus parameters within the motor timing tasks that are likely to affect performance measures. Here we assessed the effect of auditory and visual pacing stimuli on synchronised motor timing performance and its relationship with cognitive and behavioural predictors that are commonly used in the diagnosis of these highly prevalent developmental disorders. Twenty-one children (mean age 9.6 years) completed a finger tapping task in two stimulus conditions, together with additional psychometric measures. As anticipated, synchronisation to the beat (ISI 329 ms) was less accurate in the visually paced condition. Decomposition of timing variance indicated that this effect resulted from differences in the way that visual and auditory paced tasks are processed by central timekeeping and associated peripheral implementation systems. The ability to utilise an efficient processing strategy on the visual task correlated with both reading and sustained attention skills. Dissociations between these patterns of relationship across task modality suggest that not all timing tasks are equivalent.

The Build-Up and Transfer of Sensorimotor Temporal Recalibration Measured via a Synchronization Task

The timing relation between a motor action and the sensory consequences of that action can be adapted by exposing participants to artificially delayed feedback (temporal recalibration). Here, we demonstrate that a sensorimotor synchronization task (i.e., tapping the index finger in synchrony with a pacing signal) can be used as a measure of temporal recalibration. Participants were first exposed to a constant delay (~150 ms) between a voluntary action (a finger tap) and an external feedback stimulus of that action (a visual flash or auditory tone). A subjective "no-delay" condition (~50 ms) served as baseline. After a short exposure phase to delayed feedback participants performed the tapping task in which they tapped their finger in synchrony with a flash or tone. Temporal recalibration manifested itself in that taps were given ~20 ms earlier after exposure to 150 ms delays than in the case of 50 ms delays. This effect quickly built up (within 60 taps) and was bigger for auditory than visual adapters. In Experiment 2, we tested whether temporal recalibration would transfer across modalities by switching the modality of the adapter and pacing signal. Temporal recalibration transferred from visual adapter to auditory test, but not from auditory adapter to visual test. This asymmetric transfer suggests that sensory-specific effects are at play.

Audiovisual time perception is spatially specific

Our sensory systems face a daily barrage of auditory and visual signals whose arrival times form a wide range of audiovisual asynchronies. These temporal relationships constitute an important metric for the nervous system when surmising which signals originate from common external events. Internal consistency is known to be aided by sensory adaptation: repeated exposure to consistent asynchrony brings perceived arrival times closer to simultaneity. However, given the diverse nature of our audiovisual environment, functionally useful adaptation would need to be constrained to signals that were generated together. In the current study, we investigate the role of two potential constraining factors: spatial and contextual correspondence. By employing an experimental design that allows independent control of both factors, we show that observers are able to simultaneously adapt to two opposing temporal relationships, provided they are segregated in space. No such recalibration was observed when spatial segregation was replaced by contextual stimulus features (in this case, pitch and spatial frequency). These effects provide support for dedicated asynchrony mechanisms that interact with spatially selective mechanisms early in visual and auditory sensory pathways.

Multisensory simultaneity recalibration: storage of the aftereffect in the absence of counterevidence

Recent studies show that repeated exposure to an asynchrony between auditory and visual stimuli shifts the point of subjective simultaneity. Usually, the measurement stimuli used to assess this aftereffect are interleaved with short re-exposures to the asynchrony. In a first experiment, we show that the aftereffect declines during measurement in spite of the use of re-exposures. In a second experiment, we investigate whether the observed decline is either due to a dissipation of the aftereffect with the passage of time, or the result of using measurement stimuli with a distribution of asynchronies different from the exposure stimulus. To this end, we introduced a delay before measuring the aftereffects and we compared the magnitude of the aftereffect with and without delay. We find that the aftereffect does not dissipate during the delay but instead is stored until new sensory information in the form of measurement stimuli is presented as counterevidence (i.e., stimuli with an asynchrony that differs from the one used during exposure).

Shifts of criteria or neural timing? The assumptions underlying timing perception studies

In timing perception studies, the timing of one event is usually manipulated relative to another, and participants are asked to judge if the two events were synchronous, or to judge which of the two events occurred first. Responses are analyzed to determine a measure of central tendency, which is taken as an estimate of the timing at which the two events are perceptually synchronous. When these estimates do not coincide with physical synchrony, it is often assumed that the sensory signals are asynchronous, as though the transfer of information concerning one input has been accelerated or decelerated relative to the other. Here we show that, while this is a viable interpretation, it is equally plausible that such effects are driven by shifts in the criteria used to differentiate simultaneous from asynchronous inputs. Our analyses expose important ambiguities concerning the interpretation of simultaneity judgement data, which have hitherto been underappreciated.

Asynchrony adaptation reveals neural population code for audio-visual timing

The relative timing of auditory and visual stimuli is a critical cue for determining whether sensory signals relate to a common source and for making inferences about causality. However, the way in which the brain represents temporal relationships remains poorly understood. Recent studies indicate that our perception of multisensory timing is flexible--adaptation to a regular inter-modal delay alters the point at which subsequent stimuli are judged to be simultaneous. Here, we measure the effect of audio-visual asynchrony adaptation on the perception of a wide range of sub-second temporal relationships. We find distinctive patterns of induced biases that are inconsistent with the previous explanations based on changes in perceptual latency. Instead, our results can be well accounted for by a neural population coding model in which: (i) relative audio-visual timing is represented by the distributed activity across a relatively small number of neurons tuned to different delays; (ii) the algorithm for reading out this population code is efficient, but subject to biases owing to under-sampling; and (iii) the effect of adaptation is to modify neuronal response gain. These results suggest that multisensory timing information is represented by a dedicated population code and that shifts in perceived simultaneity following asynchrony adaptation arise from analogous neural processes to well-known perceptual after-effects.