Phasic and sustained interactions of multisensory interplay and temporal expectation

Enhancing rhythmic temporal expectations: The dominance of auditory modality under spatial uncertainty

To effectively process the most relevant information, the brain anticipates the optimal timing for allocating attentional resources. Behavior can be optimized by automatically aligning attention with external rhythmic structures, whether visual or auditory. Although the auditory modality is known for its efficacy in representing temporal information, the current body of research has not conclusively determined whether visual or auditory rhythmic presentations have a definitive advantage in entraining temporal attention. The present study directly examined the effects of auditory and visual rhythmic cues on the discrimination of visual targets in Experiment 1 and on auditory targets in Experiment 2. Additionally, the role of endogenous spatial attention was also considered. When and where the target was the most likely to occur were cued by unimodal (visual or auditory) and bimodal (audiovisual) signals. A sequence of salient events was employed to elicit rhythm-based temporal expectations and a symbolic predictive cue served to orient spatial attention. The results suggest a superiority of auditory over visual rhythms, irrespective of spatial attention, whether the spatial cue and rhythm converge or not (unimodal or bimodal), and regardless of the target modality (visual or auditory). These findings are discussed in terms of a modality-specific rhythmic orienting, while considering a single, supramodal system operating in a top-down manner for endogenous spatial attention.

Cross-modal implicit learning of random time patterns

Perception is sensitive to statistical regularities in the environment, including temporal characteristics of sensory inputs. Interestingly, implicit learning of temporal patterns in one modality can also improve their processing in another modality. However, it is unclear how cross-modal learning transfer affects neural responses to sensory stimuli. Here, we recorded neural activity of human volunteers using electroencephalography (EEG), while participants were exposed to brief sequences of randomly timed auditory or visual pulses. Some trials consisted of a repetition of the temporal pattern within the sequence, and subjects were tasked with detecting these trials. Unknown to the participants, some trials reappeared throughout the experiment across both modalities (Transfer) or only within a modality (Control), enabling implicit learning in one modality and its transfer. Using a novel method of analysis of single-trial EEG responses, we showed that learning temporal structures within and across modalities is reflected in neural learning curves. These putative neural correlates of learning transfer were similar both when temporal information learned in audition was transferred to visual stimuli and vice versa. The modality-specific mechanisms for learning of temporal information and general mechanisms which mediate learning transfer across modalities had distinct physiological signatures: temporal learning within modalities relied on modality-specific brain regions while learning transfer affected beta-band activity in frontal regions.

Cross-modal perceptual enhancement of unisensory targets is uni-directional and does not affect temporal expectations

Temporal structures in the environment can shape temporal expectations (TE); and previous studies demonstrated that TEs interact with multisensory interplay (MSI) when multisensory stimuli are presented synchronously. Here, we tested whether other types of MSI - evoked by asynchronous yet temporally flanking irrelevant stimuli - result in similar performance patterns. To this end, we presented sequences of 12 stimuli (10 Hz) which consisted of auditory (A), visual (V) or alternating auditory-visual stimuli (e.g. A-V-A-V-…) with either auditory or visual targets (Exp. 1). Participants discriminated target frequencies (auditory pitch or visual spatial frequency) embedded in these sequences. To test effects of TE, the proportion of early and late temporal target positions was manipulated run-wise. Performance for unisensory targets was affected by temporally flanking distractors, with auditory temporal flankers selectively improving visual target perception (Exp. 1). However, no effect of temporal expectation was observed. Control experiments (Exp. 2-3) tested whether this lack of TE effect was due to the higher presentation frequency in Exp. 1 relative to previous experiments. Importantly, even at higher stimulation frequencies redundant multisensory targets (Exp. 2-3) reliably modulated TEs. Together, our results indicate that visual target detection was enhanced by MSI. However, this cross-modal enhancement - in contrast to the redundant target effect - was still insufficient to generate TEs. We posit that unisensory target representations were either instable or insufficient for the generation of TEs while less demanding MSI still occurred; highlighting the need for robust stimulus representations when generating temporal expectations.

Minimal interplay between explicit knowledge, dynamics of learning and temporal expectations in different, complex uni- and multisensory contexts

While temporal expectations (TE) generally improve reactions to temporally predictable events, it remains unknown how the learning of temporal regularities (one time point more likely than another time point) and explicit knowledge about temporal regularities contribute to performance improvements; and whether any contributions generalise across modalities. Here, participants discriminated the frequency of diverging auditory, visual or audio-visual targets embedded in auditory, visual or audio-visual distractor sequences. Temporal regularities were manipulated run-wise (early vs. late target within sequence). Behavioural performance (accuracy, RT) plus measures from a computational learning model all suggest that learning of temporal regularities occurred but did not generalise across modalities, and that dynamics of learning (size of TE effect across runs) and explicit knowledge have little to no effect on the strength of TE. Remarkably, explicit knowledge affects performance-if at all-in a context-dependent manner: Only under complex task regimes (here, unknown target modality) might it partially help to resolve response conflict while it is lowering performance in less complex environments.

Context dependency of time-based event-related expectations for different modalities

Expectations about the temporal occurrence of events (when) are often tied with the expectations about certain event-related properties (what and where) happening at these time points. For instance, slowly waking up in the morning we expect our alarm clock to go off; however, the longer we do not hear it the more likely we already missed it. However, most current evidence for complex time-based event-related expectations (TBEEs) is based on the visual modality. Here we tested whether implicit TBEEs can act cross-modally. To this end, visual and auditory stimulus streams were presented which contained early and late targets embedded among distractors (to maximise temporal target uncertainty). Foreperiod-modality-contingencies were manipulated run-wise: visual targets either occurred early in 80% of trials and auditory targets occurred late in 80% of trials or vice versa. Participants showed increased sensitivity for expected auditory early/visual late targets which increased over time while the opposite pattern was observed for visual early/auditory late targets. A benefit in reaction times was only found for auditory early trials. Together, this pattern of results suggests that implicit context-dependent TBEEs for auditory targets after short foreperiods (be they correct or not) dominated and determined which modality became more expected at the late position irrespective of the veridical statistical regularity. Hence, TBEEs in cross-modal and uncertain environments are context-dependent, shaped by the dominant modality in temporal tasks (i.e., auditory) and only boost performance cross-modally when expectations about the event after the short foreperiod match with the run-wise context (i.e., auditory early/visual late).

Explicitly versus implicitly driven temporal expectations: No evidence for altered perceptual processing due to top-down modulations

Learning the statistical regularities of environmental events is a powerful tool for enhancing performance. However, it remains unclear whether this often implicit type of behavioral facilitation can be proactively modulated by explicit knowledge about temporal regularities. Only recently, Menceloglu and colleagues (Attention, Perception & Psychophysics, 79(1), 169-179, 2017) tested for differences between implicit versus explicit statistical learning of temporal regularities by using a within-paradigm manipulation of metacognitive temporal knowledge. The authors reported that temporal expectations were enhanced if participants had explicit knowledge about temporal regularities. Here, we attempted to replicate and extend their results, and to provide a mechanistic framework for any effects by means of computational modelling. Participants performed a letter-discrimination task, with target letters embedded in congruent or incongruent flankers. Temporal predictability was manipulated block-wise, with targets occurring more often after either a short or a long delay period. During the delay a sound was presented in half of the trials. Explicit knowledge about temporal regularities was manipulated by changing instructions: Participants received no information (implicit), information about the most likely cue-target delay (explicit), or received 100% valid cues on each trial (highly explicit). We replicated previous effects of target-flanker congruence and sound presence. However, no evidence was found for an effect of explicit knowledge on temporal expectations using Bayesian statistics. Concordantly, computational modelling suggested that explicit knowledge may only influence non-perceptual processing such as response criteria. Together, our results indicate that explicit metacognitive knowledge does not necessarily alter sensory representations or temporal expectations but rather affects response strategies.

The evolving sense of agency: Context recency and quality modulate the interaction between prospective and retrospective processes

Humans acquire a sense of agency through their interactions with the world and their sensory consequences. Previous studies have highlighted stable agency-related phenomena like intentional binding, which depend on both prospective, context-dependent and retrospective, outcome-dependent processes. In the current study, we investigated the interaction between prospective and retrospective processes underlying the adaptation of an ongoing sense of agency. The results showed that prospective intentional binding developed during a temporal window of up to 20 prior events was independent of the nature of the ongoing event. By contrast, the characteristics of the ongoing event retrospectively influenced prospective intentional binding developed during a temporal window narrower than 6 prior events. These findings characterize the interaction between prospective and retrospective mechanisms as a fundamental process to continuously update the sense of agency through sensorimotor learning. High psychosis-like experience traits weakened this interaction, suggesting that reduced adaption to the context contributes to altered self-experience.

Phasic and sustained interactions of multisensory interplay and temporal expectation

Every moment organisms are confronted with complex streams of information which they use to generate a reliable mental model of the world. There is converging evidence for several optimization mechanisms instrumental in integrating (or segregating) incoming information; among them are multisensory interplay (MSI) and temporal expectation (TE). Both mechanisms can account for enhanced perceptual sensitivity and are well studied in isolation; how these two mechanisms interact is currently less well-known. Here, we tested in a series of four psychophysical experiments for TE effects in uni- and multisensory contexts with different levels of modality-related and spatial uncertainty. We found that TE enhanced perceptual sensitivity for the multisensory relative to the best unisensory condition (i.e. multisensory facilitation according to the max-criterion). In the latter TE effects even vanished if stimulus-related spatial uncertainty was increased. Accordingly, computational modelling indicated that TE, modality-related and spatial uncertainty predict multisensory facilitation. Finally, the analysis of stimulus history revealed that matching expectation at trial n-1 selectively improves multisensory performance irrespective of stimulus-related uncertainty. Together, our results indicate that benefits of multisensory stimulation are enhanced by TE especially in noisy environments, which allows for more robust information extraction to boost performance on both short and sustained time ranges.