Sound reduces saccadic chronostasis illusion

The saccadic chronostasis illusion refers to the duration overestimation of the first visual stimulation after saccadic eye movement, which is also known as "stopped clock illusion." The present study investigated whether saccadic chronostasis would be observed in the auditory modality and whether the saccade-induced time dilation in the visual modality would be reduced by a synchronously presented sound. In each trial, a unisensory visual stimulus, unisensory sound, or bimodal audio-visual stimulus with a duration of 200-800 ms (probe stimulus) was presented at the saccade target location and temporally around the offset of the saccade, followed by a unisensory visual or auditory standard stimulus for a fixed 500 ms. Participants were required to identify which of the two stimuli (probe or standard) presented in the target modality (visual or auditory) was perceived as longer. The results showed that no saccadic chronostasis was observed in the auditory modality, regardless of whether the sound was presented alone or synchronously accompanied by a visual stimulus. Interestingly, the magnitude of the saccadic chronostasis illusion was reduced by the synchronously presented sound. Moreover, the combined effect of the saccade and sound on visual time perception fits well with the standard scalar model, and the weight of the cross-modal effect was higher than that of saccadic visual time dilation. These results suggest that sound dominates vision in time processing during saccades and linearly modulates saccadic chronostasis, which follows the Scalar Expectancy Theory.

Slowing the body slows down time perception

Interval timing is a fundamental component of action and is susceptible to motor-related temporal distortions. Previous studies have shown that concurrent movement biases temporal estimates, but have primarily considered self-modulated movement only. However, real-world encounters often include situations in which movement is restricted or perturbed by environmental factors. In the following experiments, we introduced viscous movement environments to externally modulate movement and investigated the resulting effects on temporal perception. In two separate tasks, participants timed auditory intervals while moving a robotic arm that randomly applied four levels of viscosity. Results demonstrated that higher viscosity led to shorter perceived durations. Using a drift-diffusion model and a Bayesian observer model, we confirmed these biasing effects arose from perceptual mechanisms, instead of biases in decision making. These findings suggest that environmental perturbations are an important factor in movement-related temporal distortions, and enhance the current understanding of the interactions of motor activity and cognitive processes.

Temporal sequence discrimination within and across senses: do we really hear what we see?

Previous evidence suggests that people "hear" visual stimuli when encoding temporal information. This suggestion is based on the observation that auditory distractor information can strongly affect discrimination performance for visual temporal sequences. The present study aimed to replicate and extend this finding by investigating sequence discrimination within and across the two modalities. In two experimental series, participants judged whether two subsequently presented temporal sequences, a standard sequence followed by a comparison sequence, were identical or not. In Experimental Series A, irrelevant distractor information was presented simultaneously with the standard sequence. In Series B, the distraction appeared in the retention interval between the standard sequence and the comparison sequence. The results showed that auditory distraction impaired performance irrespective of whether the target sequences were auditory or visual, whereas visual distraction only impaired the discrimination of visual target sequences. Furthermore, auditory distraction was always at least as effective as visual distraction, irrespective of standard modality. Generally, discrimination performance was much better for auditory than for visual sequences. Overall, the present results are consistent with the idea that people code visual temporal information in the auditory modality. Moreover, the present study also suggests that such cross-modal interference effects should be interpreted cautiously with respect to their underlying timing mechanism because of the basic differences in temporal sensitivity between the two modalities.

On the Timing of Signals in Multisensory Integration and Crossmodal Interactions: a Scoping Review

A scoping review was undertaken to explore research investigating early interactions and integration of auditory and visual stimuli in the human brain. The focus was on methods used to study low-level multisensory temporal processing using simple stimuli in humans, and how this research has informed our understanding of multisensory perception. The study of multisensory temporal processing probes how the relative timing between signals affects perception. Several tasks, illusions, computational models, and neuroimaging techniques were identified in the literature search. Research into early audiovisual temporal processing in special populations was also reviewed. Recent research has continued to provide support for early integration of crossmodal information. These early interactions can influence higher-level factors, and vice versa. Temporal relationships between auditory and visual stimuli influence multisensory perception, and likely play a substantial role in solving the 'correspondence problem' (how the brain determines which sensory signals belong together, and which should be segregated).

Age-Related Effects on Cross-Modal Duration Perception

Reliable duration perception of external events is necessary to coordinate perception with action, precisely discriminate speech, and for other daily functions. Visual duration perception can be heavily influenced by concurrent auditory signals; however, age-related effects on this process have received minimal attention. In the present study, we examined the effect of aging on duration perception by quantifying (1) duration discrimination thresholds, (2) auditory temporal dominance, and (3) visual duration expansion/compression percepts induced by an accompanying auditory stimulus of longer/shorter duration. Duration discrimination thresholds were significantly greater for visual than auditory tasks in both age groups, however there was no effect of age. While the auditory modality retained dominance in duration perception with age, older adults still performed worse than young adults when comparing durations of two target stimuli (e.g., visual) in the presence of distractors from the other modality (e.g., auditory). Finally, both age groups perceived similar visual duration compression, whereas older adults exhibited visual duration expansion over a wider range of auditory durations compared to their younger counterparts. Results are discussed in terms of multisensory integration and possible decision strategies that change with age.

Enhancement of letter identification by concurrent auditory stimuli of varying duration

Previously it has been shown that the concurrent presentation of a sound can improve processing of visual information at higher perceptual levels, for example, in letter identification tasks. Moreover, increasing the duration of the concurrent sounds can enhance performance in low-level tasks as contrast detection, which has been attributed to a sustained visual activation corresponding to the duration of the sound. Yet, the role of sound duration has so far not been investigated in higher-level visual processing. In a series of five Experiments, we again demonstrated that the mere presence of a concurrent sound can enhance the identification of a masked, centrally presented letter compared to unimodal presentation, even though this benefit was absent in one experiment for high-contrast letters yielding an especially high level of task-performance. In general, however, the sound-induced benefit was not modulated by a variation of target contrast or by the duration of the target-to-mask interstimulus interval. Taking individual performance differences into account, a further analysis suggested that the sound-induced facilitation effect may nevertheless be most pronounced at specific performance levels. Beyond this general sound-induced facilitation, letter identification performance was not further affected by the duration of the concurrent sounds, even though in a control experiment it could be established that letter identification performance improved with increasing letter duration, and perceived letter duration was prolonged with increasing auditory duration. The results and their interpretation with respect to the large observed interindividual performance differences are discussed in terms of potential underlying mechanisms of multisensory facilitation, as preparedness enhancement, signal enhancement, and object enhancement.

Still no Evidence for Sustained Effects of Multisensory Integration of Duration

In studies on temporal order perception, immediate as well as sustained effects of multisensory integration have been demonstrated repeatedly. Regarding duration perception, the corresponding literature reports clear immediate effects of multisensory integration, but evidence on sustained effects of multisensory duration integration is scarce. In fact, a single study [Heron, J. et al. (2013). A neural hierarchy for illusions of time: Duration adaptation precedes multisensory integration, J. Vis. 13, 1-12.] investigated adaptation to multisensory conflicting intervals, and found no sustained effects of the audiovisual conflict on perceived duration of subsequently presented unimodal visual intervals. In two experiments, we provide independent evidence in support of this finding. In Experiment 1, we demonstrate that adaptation to audiovisual conflict does not alter perceived duration of subsequently presented visual test intervals. Thus, replicating the results of Heron et al. (2013), we observed no sustained effect of multisensory duration integration. However, one might argue that the prolonged exposure to consistent multisensory conflict might have prevented or hampered multisensory integration per se. In Experiment 2, we rule out this alternative explanation by showing that multisensory integration of audiovisual conflicting intervals is still effective after exposure to audiovisual conflict. This further strengthens the conclusion that multisensory integration of interval duration affects perception in an immediate, but not in a sustained manner.

On the interplay of visuospatial and audiotemporal dominance: Evidence from a multimodal kappa effect

When participants judge multimodal audiovisual stimuli, the auditory information strongly dominates temporal judgments, whereas the visual information dominates spatial judgments. However, temporal judgments are not independent of spatial features. For example, in the kappa effect, the time interval between two marker stimuli appears longer when they originate from spatially distant sources rather than from the same source. We investigated the kappa effect for auditory markers presented with accompanying irrelevant visual stimuli. The spatial sources of the markers were varied such that they were either congruent or incongruent across modalities. In two experiments, we demonstrated that the spatial layout of the visual stimuli affected perceived auditory interval duration. This effect occurred although the visual stimuli were designated to be task-irrelevant for the duration reproduction task in Experiment 1, and even when the visual stimuli did not contain sufficient temporal information to perform a two-interval comparison task in Experiment 2. We conclude that the visual and auditory marker stimuli were integrated into a combined multisensory percept containing temporal as well as task-irrelevant spatial aspects of the stimulation. Through this multisensory integration process, visuospatial information affected even temporal judgments, which are typically dominated by the auditory modality.

Representations of temporal information in short-term memory: Are they modality-specific?

Rattat and Picard (2012) reported that the coding of temporal information in short-term memory is modality-specific, that is, temporal information received via the visual (auditory) modality is stored as a visual (auditory) code. This conclusion was supported by modality-specific interference effects on visual and auditory duration discrimination, which were induced by secondary tasks (visual tracking or articulatory suppression), presented during a retention interval. The present study assessed the stability of these modality-specific interference effects. Our study did not replicate the selective interference pattern but rather indicated that articulatory suppression not only impairs short-term memory for auditory but also for visual durations. This result pattern supports a crossmodal or an abstract view of temporal encoding.

Temporal judgments in multi-sensory space

To successfully interact with the environment requires a combination of stimulus recognition as well as localization in both space and time, with information moreover coming from multiple senses. Several studies have shown that auditory stimuli last subjectively longer than visual ones of equal duration. Recently, it has also been suggested that stimulus position affects duration perception. The present study investigated how lateral spatial presentation influences sub-second visual and auditory duration judgments. Five experiments were conducted using the duration discrimination paradigm, wherein two stimuli are presented sequentially and participants are asked to judge whether the second stimulus (comparison) is shorter or longer in duration than the first (standard). The number of stimulus positions and the way in which different modality trials were presented (mixed or blocked) varied. Additionally, comparisons were made either within or across modalities. No stable effect of location itself was found. However, in mixed modality experiments there was a clear over-estimation of duration in visual trials when the location of the comparison was different from the standard. This effect was reversed in the same location trials. Auditory judgments were unaffected by location manipulations. Based on these results, we propose the existence of an error-mechanism, according to which a specific duration is added in order to compensate for the loss of duration perception caused by spatial attention shifts between different locations. This mechanism is revealed in spatial and modality-mixed circumstances wherein its over-activation results in a systematic temporal bias.

Sounds Modulate the Perceived Duration of Visual Stimuli via Crossmodal Integration

Previous studies have shown that the perceived duration of visual stimuli can be strongly distorted by auditory stimuli presented simultaneously. In this study, we examine whether sounds presented separately from target visual stimuli alter the perceived duration of the target’s presentation. The participants’ task was to classify the duration of the target visual stimuli as perceived by them into four categories. Our results demonstrate that a sound presented before and after a visual target increases or decreases the perceived visual duration depending on the inter-stimulus interval between the sounds and the visual stimulus. In addition, three tones presented before and after a visual target did not increase or decrease the perceived visual duration. This indicates that auditory perceptual grouping prevents intermodal perceptual grouping, and eliminates crossmodal effects. These findings suggest that the auditory–visual integration, rather than a high arousal state caused by the presentation of the preceding sound, can induce distortions of perceived visual duration, and that inter- and intramodal perceptual grouping plays an important role in crossmodal time perception. These findings are discussed with reference to the Scalar Expectancy Theory.