Sounds Modulate the Perceived Duration of Visual Stimuli via Crossmodal Integration

Measuring the perception and metacognition of time

The ability of humans to identify and reproduce short time intervals (in the region of a second) may be affected by many factors ranging from the gender and personality of the individual observer, through the attentional state, to the precise spatiotemporal structure of the stimulus. The relative roles of these very different factors are a challenge to describe and define; several methodological approaches have been used to achieve this to varying degrees of success. Here we describe and model the results of a paradigm affording not only a first-order measurement of the perceived duration of an interval but also a second-order metacognitive judgement of perceived time. This approach, we argue, expands the form of the data generally collected in duration-judgements and allows more detailed comparison of psychophysical behavior to the underlying theory. We also describe a hierarchical Bayesian measurement model that performs a quantitative analysis of the trial-by-trial data calculating the variability of the temporal estimates and the metacognitive judgments allowing direct comparison between an actual and an ideal observer. We fit the model to data collected for judgements of 750 ms (bisecting 1500 ms) and 1500 ms (bisecting 3000 ms) intervals across three stimulus modalities (visual, audio, and audiovisual). This enhanced form of data on a given interval judgement and the ability to track its progression on a trial-by-trial basis offers a way of looking at the different roles that subject-based, task-based and stimulus-based factors have on the perception of time.

Stimulus (dis)similarity can modify the effect of a task-irrelevant sandwiching stimulus on the perceived duration of brief visual stimuli

The perceived duration of a target visual stimulus is shorter when a brief non-target visual stimulus precedes and trails the target than when it appears alone. This time compression requires spatiotemporal proximity of the target and non-target stimuli, which is one of the perceptual grouping rules. The present study examined whether and how another grouping rule, stimulus (dis)similarity, modulated this effect. In Experiment 1, time compression occurred only when the preceding and trailing stimuli (black-white checkerboard) were dissimilar from the target (unfilled round or triangle) with spatiotemporal proximity. In contrast, it was reduced when the preceding or trailing stimuli (filled rounds or triangles) were similar to the target. Experiment 2 revealed time compression with dissimilar stimuli, independent of the intensity or saliency of the target and non-target stimuli. Experiment 3 replicated the findings of Experiment 1 by manipulating the luminance similarity between target and non-target stimuli. Furthermore, time dilation occurred when the non-target stimuli were indistinguishable from the target stimuli. These results indicate that stimulus dissimilarity with spatiotemporal proximity induces time compression, whereas stimulus similarity with spatiotemporal proximity does not. These findings were discussed in relation to the neural readout model.

Auditory affective content facilitates time-to-contact estimation of visual affective targets

Reacting to a moving object requires an ability to estimate when a moving object reaches its destination, also referred to as the time-to-contact (TTC) estimation. Although the TTC estimation of threatening visually moving objects is known to be underestimated, the effect of the affective content of auditory information on visual TTC estimation remains unclear. We manipulated the velocity and presentation time to investigate the TTC of a threat or non-threat target with the addition of auditory information. In the task, a visual or an audiovisual target moved from right to left and disappeared behind an occluder. Participants' task was to estimate the TTC of the target, they needed to press a button when they thought that the target contacted a destination behind the occluder. Behaviorally, the additional auditory affective content facilitated TTC estimation; velocity was a more critical factor than presentation time in determining the audiovisual threat facilitation effect. Overall, the results indicate that exposure to auditory affective content can influence TTC estimation and that the effect of velocity on TTC estimation will provide more information than presentation time.

Temporal Alignment but not Complexity of Audiovisual Stimuli Influences Crossmodal Duration Percepts

Reliable duration perception is an integral aspect of daily life that impacts everyday perception, motor coordination, and subjective passage of time. The Scalar Expectancy Theory (SET) is a common model that explains how an internal pacemaker, gated by an external stimulus-driven switch, accumulates pulses during sensory events and compares these accumulated pulses to a reference memory duration for subsequent duration estimation. Second-order mechanisms, such as multisensory integration (MSI) and attention, can influence this model and affect duration perception. For instance, diverting attention away from temporal features could delay the switch closure or temporarily open the accumulator, altering pulse accumulation and distorting duration perception. In crossmodal duration perception, auditory signals of unequal duration can induce perceptual compression and expansion of durations of visual stimuli, presumably via auditory influence on the visual clock. The current project aimed to investigate the role of temporal (stimulus alignment) and nontemporal (stimulus complexity) features on crossmodal, specifically auditory over visual, duration perception. While temporal alignment revealed a larger impact on the strength of crossmodal duration percepts compared to stimulus complexity, both features showcase auditory dominance in processing visual duration.

Incongruent Audiovisual Inducer Information and Fission/Fusion Illusions

In research studies on how people perceive simultaneously presented audiovisual information, researchers have often shown that the number of visual flashes participants perceive on a computer screen can be altered by varying the number of accompanying auditory, visual, or combined audiovisual cues or inducers. In the present study, we examined the effects of number-incongruent audiovisual inducer stimuli on the participants' perceived number of target flashes. We instructed 16 participants (eight males and eight females; M_age = 21.56; SD_age = 1.93) to report their perceived number of target flashes while ignoring the visual and auditory inducers. Across 18 different experimental conditions, we presented one or two target flashes in association with varied numbers (0, 1, 2) of auditory and visual inducer stimuli. In the condition with one target flash paired with one visual and two auditory inducers, the number of visual inducers (i.e., one) had a greater influence on the number of perceived target flashes than did the number of auditory inducers (i.e., two). Under all other number incongruent audiovisual inducer conditions, the participants' perceived number of target flashes was influenced more by the number of auditory than the number of visual inducers. We discuss these findings in the context of perceptual grouping and perceptual temporal uncertainty.

Sandwiched visual stimuli are perceived as shorter than the stimulus alone

A visual stimulus is perceived as shorter when a short sound is presented immediately before and after the visual target than when the visual target appears alone. It remains unclear whether the time compression occurs in an intramodal condition. Therefore, the present study examined how and when non-target sandwiching stimuli affect the perceived filled duration of target visual stimuli. We further hypothesized that this effect could be modulated by temporal and spatial proximity between the target and non-target stimuli. Experiments 1a, 1b, and 2 showed that non-target stimuli could decrease the perceived duration only when the inter-stimulus interval between these stimuli was 0 ms, using time reproduction and category estimation methods. Experiments 3 revealed that the time compression effect did not occur when both the non-target preceding and trailing stimuli were spatially distinct from the target. Experiment 4 demonstrated that either the preceding or trailing stimulus induced the time compression effect when the non-target stimuli were presented at the same position as the target stimuli. We discuss the implications of the time compression effect induced by non-target sandwiching stimuli with reference to the Scalar Expectancy Theory and the Neural Readout Model. We speculated that the attenuation of neural responses to the target via visual masking or perceptual grouping may be attributable to the time compression effect.

Enhancement of letter identification by concurrent auditory stimuli of varying duration

Previously it has been shown that the concurrent presentation of a sound can improve processing of visual information at higher perceptual levels, for example, in letter identification tasks. Moreover, increasing the duration of the concurrent sounds can enhance performance in low-level tasks as contrast detection, which has been attributed to a sustained visual activation corresponding to the duration of the sound. Yet, the role of sound duration has so far not been investigated in higher-level visual processing. In a series of five Experiments, we again demonstrated that the mere presence of a concurrent sound can enhance the identification of a masked, centrally presented letter compared to unimodal presentation, even though this benefit was absent in one experiment for high-contrast letters yielding an especially high level of task-performance. In general, however, the sound-induced benefit was not modulated by a variation of target contrast or by the duration of the target-to-mask interstimulus interval. Taking individual performance differences into account, a further analysis suggested that the sound-induced facilitation effect may nevertheless be most pronounced at specific performance levels. Beyond this general sound-induced facilitation, letter identification performance was not further affected by the duration of the concurrent sounds, even though in a control experiment it could be established that letter identification performance improved with increasing letter duration, and perceived letter duration was prolonged with increasing auditory duration. The results and their interpretation with respect to the large observed interindividual performance differences are discussed in terms of potential underlying mechanisms of multisensory facilitation, as preparedness enhancement, signal enhancement, and object enhancement.

Still no Evidence for Sustained Effects of Multisensory Integration of Duration

In studies on temporal order perception, immediate as well as sustained effects of multisensory integration have been demonstrated repeatedly. Regarding duration perception, the corresponding literature reports clear immediate effects of multisensory integration, but evidence on sustained effects of multisensory duration integration is scarce. In fact, a single study [Heron, J. et al. (2013). A neural hierarchy for illusions of time: Duration adaptation precedes multisensory integration, J. Vis. 13, 1-12.] investigated adaptation to multisensory conflicting intervals, and found no sustained effects of the audiovisual conflict on perceived duration of subsequently presented unimodal visual intervals. In two experiments, we provide independent evidence in support of this finding. In Experiment 1, we demonstrate that adaptation to audiovisual conflict does not alter perceived duration of subsequently presented visual test intervals. Thus, replicating the results of Heron et al. (2013), we observed no sustained effect of multisensory duration integration. However, one might argue that the prolonged exposure to consistent multisensory conflict might have prevented or hampered multisensory integration per se. In Experiment 2, we rule out this alternative explanation by showing that multisensory integration of audiovisual conflicting intervals is still effective after exposure to audiovisual conflict. This further strengthens the conclusion that multisensory integration of interval duration affects perception in an immediate, but not in a sustained manner.