Perceptual Oscillation of Audiovisual Time Simultaneity

Theoretical and Technical Issues Concerning the Measurement of Alpha Frequency and the Application of Signal Detection Theory: Comment on Buergers and Noppeney (2022)

Classical and recent evidence has suggested that alpha oscillations play a critical role in temporally discriminating or binding successively presented items. Challenging this view, Buergers and Noppeney [Buergers, S., & Noppeney, U. The role of alpha oscillations in temporal binding within and across the senses. Nature Human Behaviour, 6, 732-742, 2022] found that by combining EEG, psychophysics, and signal detection theory, neither prestimulus nor resting-state alpha frequency influences perceptual sensitivity and bias in the temporal binding task. We propose the following four points that should be considered when interpreting the role of alpha oscillations, and especially their frequency, on perceptual temporal binding: (1) Multiple alpha components can be contaminated in conventional EEG analysis; (2) the effect of alpha frequency on perception will interact with alpha power; (3) prestimulus and resting-state alpha frequency can be different from poststimulus alpha frequency, which is the frequency during temporal binding and should be more directly related to temporal binding; and (4) when applying signal detection theory under the assumption of equal variance, the assumption is often incomplete and can be problematic (e.g., the magnitude relationships between individuals in parametric sensitivity may change when converted into nonparametric sensitivity). Future directions, including solutions to each of the issues, are discussed.

Audiovisual simultaneity windows reflect temporal sensory uncertainty

The ability to judge the temporal alignment of visual and auditory information is a prerequisite for multisensory integration and segregation. However, each temporal measurement is subject to error. Thus, when judging whether a visual and auditory stimulus were presented simultaneously, observers must rely on a subjective decision boundary to distinguish between measurement error and truly misaligned audiovisual signals. Here, we tested whether these decision boundaries are relaxed with increasing temporal sensory uncertainty, i.e., whether participants make the same type of adjustment an ideal observer would make. Participants judged the simultaneity of audiovisual stimulus pairs with varying temporal offset, while being immersed in different virtual environments. To obtain estimates of participants' temporal sensory uncertainty and simultaneity criteria in each environment, an independent-channels model was fitted to their simultaneity judgments. In two experiments, participants' simultaneity decision boundaries were predicted by their temporal uncertainty, which varied unsystematically with the environment. Hence, observers used a flexibly updated estimate of their own audiovisual temporal uncertainty to establish subjective criteria of simultaneity. This finding implies that, under typical circumstances, audiovisual simultaneity windows reflect an observer's cross-modal temporal uncertainty.

To remember or not to remember: Neural oscillations and ERPs as predictors of intentional associative fear learning

It is widely accepted that impaired safety learning to a safe stimulus is a pathological feature of anxiety disorders. Safety learning refers to learning that a stimulus is associated with the absence of threat. Cognitive mechanisms that underlie successful threat and safety learning are, however, poorly understood. This study aimed to identify various physiological markers, including neural oscillations and event-related potentials (ERPs) that predict successful threat and safety learning. Therefore, to detect potential differences in these markers, we measured EEG in a fear learning framework combined with a subsequent memory paradigm. Thirty-seven participants were asked to memorize a series of associations between faces and an aversive unconditioned stimulus (US) or its omission. We found a decrease of power in the alpha band in occipital brain regions during learning for both threatening (conditioned stimuli, CS+) and safe faces (control stimuli, CS-) that were subsequently remembered to be associated with a US or not. No effects in theta band were found. In regard to ERPs, a late positive potential (LPP) and a P300 component were larger for remembered than for forgotten CS-US associations. The P300 was also enhanced to remembered US and US omissions, thus replicating previous findings. These results point to the importance of cognitive resource allocation as an underlying mechanism of fear learning and electrophysiological measurements as potential biomarkers for successful threat and safety learning.

No evidence for rhythmic sampling in inhibition of return

When exogenously cued, attention reflexively reorients towards the cued position. After a brief dwelling time, attention is released and then persistently inhibited from returning to this position for up to three seconds, a phenomenon coined 'inhibition of return' (IOR). This inhibitory interpretation has shaped our understanding of the spatio-temporal dynamics of the attentional spotlight after an exogenous visual cue for more than three decades. However, a recent theory refines this traditional view and predicts that attention rhythmically alternates between possible target locations at a theta frequency, implying occasional returns of attention to the cued position. Unfortunately, previous IOR studies have only probed performance at a few, temporally wide-spread cue-target onset asynchronies (CTOAs) rendering a comparison of these contradictory predictions impossible. We therefore used a temporally fine-grained adaptation of the Posner paradigm with 25 equally and densely spaced CTOAs, which yielded a robust IOR effect in the reaction time difference between valid and invalidly cued trials. We modelled the time course of this effect across CTOAs as a linear or exponential decay (traditional IOR model), sinusoidal rhythm (rhythmic model) and a combination of both (hybrid model). Model comparison by means of goodness-of-fit indices provided strong evidence in favor of traditional IOR models, and against theta-rhythmic attentional sampling contributing to IOR. This finding was supported by an FFT analysis, which also revealed no significant theta rhythm. We therefore conclude that the spatio-temporal dynamics of attention following an exogenous cue cannot be explained by rhythmic attentional sampling.

The effect of perceptual history on the interpretation of causality

The ability to interpret spatiotemporal contingencies in terms of causal relationships plays a key role in human understanding of the external world. Indeed, the detection of such simple properties enables us to attribute causal attributes to interactions between objects. Here, we investigated the degree to which this perception of causality depends on recent experience, as has been found for other low-level properties of visual stimuli. Participants were shown launching sequences of colliding circles with varying collision lags and were asked to report their impression of causality. We found short-term attractive and long-term repulsive and attractive effects of perceptual history on the interpretation of causality. Stimuli directly following a causal impression were more likely to be judged as causal and vice versa. However, prior judgments on less recent (>5) trials biased current perception with both positive/attractive and negative/repulsive influences. We interpret these results in terms of two potential mechanisms: adaptive temporal binding windows and updating of internal representations of causality. Overall, these results demonstrate the important role of prior experience even for causality, a fundamental building block of how we understand our world.

Visual priming and serial dependence are mediated by separate mechanisms

Perceptual history influences current perception, readily revealed by visual priming (the facilitation of responses on repeated presentations of similar stimuli) and by serial dependence (systematic biases toward the previous stimuli). We asked whether the two phenomena shared perceptual mechanisms. We modified the standard “priming of pop-out” paradigm to measure both priming and serial dependence concurrently. The stimulus comprised three grating patches, one or two red, and the other green. Participants identified the color singleton (either red or green), and reproduced its orientation. Trial sequences were designed to maximize serial dependence, and long runs of priming color and position. The results showed strong effects of priming, both on reaction times and accuracy, which accumulated steadily over time, as generally reported in the literature. The serial dependence effects were also strong, but did not depend on previous color, nor on the run length. Reaction times measured under various conditions of repetition or change of priming color or position were reliably correlated with imprecision in orientation reproduction, but reliably uncorrelated with magnitude of serial dependence. The results suggest that visual priming and serial dependence are mediated by different neural mechanisms. We propose that priming affects sensitivity, possibly via attention-like mechanisms, whereas serial dependence affects criteria, two orthogonal dimensions in the signal detection theory.

Audiovisual Interactions Among Near-Threshold Oscillating Stimuli in the Far Periphery Are Phase-Dependent

Recent studies have highlighted the possible contributions of direct connectivity between early sensory cortices to audiovisual integration. Anatomical connections between the early auditory and visual cortices are concentrated in visual sites representing the peripheral field of view. Here, we aimed to engage early sensory interactive pathways with simple, far-peripheral audiovisual stimuli (auditory noise and visual gratings). Using a modulation detection task in one modality performed at an 84% correct threshold level, we investigated multisensory interactions by simultaneously presenting weak stimuli from the other modality in which the temporal modulation was barely-detectable (at 55 and 65% correct detection performance). Furthermore, we manipulated the temporal congruence between the cross-sensory streams. We found evidence for an influence of barely-detectable visual stimuli on the response times for auditory stimuli, but not for the reverse effect. These visual-to-auditory influences only occurred for specific phase-differences (at onset) between the modulated audiovisual stimuli. We discuss our findings in the light of a possible role of direct interactions between early visual and auditory areas, along with contributions from the higher-order association cortex. In sum, our results extend the behavioral evidence of audio-visual processing to the far periphery, and suggest - within this specific experimental setting - an asymmetry between the auditory influence on visual processing and the visual influence on auditory processing.

Predictive visuo-motor communication through neural oscillations

The mechanisms coordinating action and perception over time are poorly understood. The sensory cortex needs to prepare for upcoming changes contingent on action, and this requires temporally precise communication that takes into account the variable delays between sensory and motor processing. Several theorists^1,2 have proposed synchronization of the endogenous oscillatory activity observed in most regions of the brain³ as the basis for an efficient and flexible communication protocol between distal brain areas,^2,4 a concept known as "communication through coherence." Synchronization of endogenous oscillations^5,6 occurs after a salient sensory stimulus, such as a flash or a sound,^7-11 and after a voluntary action,^12-18 and this directly impacts perception, causing performance to oscillate rhythmically over time. Here we introduce a novel fMRI paradigm to probe the neural sources of oscillations, based on the concept of perturbative signals, which overcomes the low temporal resolution of BOLD signals. The assumption is that a synchronized endogenous rhythm will modulate cortical excitability rhythmically, which should be reflected in the BOLD responses to brief stimuli presented at different phases of the oscillation cycle. We record rhythmic oscillations of V1 BOLD synchronized by a simple voluntary action, in phase with behaviorally measured oscillations in visual sensitivity in the theta range. The functional connectivity between V1 and M1 also oscillates at the same rhythm. By demonstrating oscillatory temporal coupling between primary motor and sensory cortices, our results strongly implicate communication through coherence to achieve precise coordination and to encode sensory-motor timing.

Tapping Force Encodes Metrical Aspects of Rhythm

Humans possess the ability to extract highly organized perceptual structures from sequences of temporal stimuli. For instance, we can organize specific rhythmical patterns into hierarchical, or metrical, systems. Despite the evidence of a fundamental influence of the motor system in achieving this skill, few studies have attempted to investigate the organization of our motor representation of rhythm. To this aim, we studied-in musicians and non-musicians-the ability to perceive and reproduce different rhythms. In a first experiment participants performed a temporal order-judgment task, for rhythmical sequences presented via auditory or tactile modality. In a second experiment, they were asked to reproduce the same rhythmic sequences, while their tapping force and timing were recorded. We demonstrate that tapping force encodes the metrical aspect of the rhythm, and the strength of the coding correlates with the individual's perceptual accuracy. We suggest that the similarity between perception and tapping-force organization indicates a common representation of rhythm, shared between the perceptual and motor systems.

Propagation and update of auditory perceptual priors through alpha and theta rhythms

To maintain a continuous and coherent percept over time, the brain makes use of past sensory information to anticipate forthcoming stimuli. We recently showed that auditory experience of the immediate past is propagated through ear-specific reverberations, manifested as rhythmic fluctuations of decision bias at alpha frequencies. Here, we apply the same time-resolved behavioural method to investigate how perceptual performance changes over time under conditions of stimulus expectation and to examine the effect of unexpected events on behaviour. As in our previous study, participants were required to discriminate the ear-of-origin of a brief monaural pure tone embedded in uncorrelated dichotic white noise. We manipulated stimulus expectation by increasing the target probability in one ear to 80%. Consistent with our earlier findings, performance did not remain constant across trials, but varied rhythmically with delay from noise onset. Specifically, decision bias showed a similar oscillation at ~9 Hz, which depended on ear congruency between successive targets. This suggests rhythmic communication of auditory perceptual history occurs early and is not readily influenced by top-down expectations. In addition, we report a novel observation specific to infrequent, unexpected stimuli that gave rise to oscillations in accuracy at ~7.6 Hz one trial after the target occurred in the non-anticipated ear. This new behavioural oscillation may reflect a mechanism for updating the sensory representation once a prediction error has been detected.

Perceptual Oscillations in Gender Classification of Faces, Contingent on Stimulus History

Perception is a proactive ‘‘predictive’’ process, in which the brain takes advantage of past experience to make informed guesses about the world to test against sensory data. Here we demonstrate that in the judgment of the gender of faces, beta rhythms play an important role in communicating perceptual experience. Observers classified in forced choice as male or female, a sequence of face stimuli, which were physically constructed to be male or female or androgynous (equal morph). Classification of the androgynous stimuli oscillated rhythmically between male and female, following a complex waveform comprising 13.5 and 17 Hz. Parsing the trials based on the preceding stimulus showed that responses to androgynous stimuli preceded by male stimuli oscillated reliably at 17 Hz, whereas those preceded by female stimuli oscillated at 13.5 Hz. These results suggest that perceptual priors for face perception from recent perceptual memory are communicated through frequency-coded beta rhythms.

•

Perception is strongly influenced by prior expectations and perceptual experience

•

We show perceptual priors for face perception are communicated through beta rhythms

•

Priors for male and female are communicated via distinct beta frequencies

Neural correlates of subjective timing precision and confidence

Humans perceptual judgments are imprecise, as repeated exposures to the same physical stimulation (e.g. audio-visual inputs separated by a constant temporal offset) can result in different decisions. Moreover, there can be marked individual differences – precise judges will repeatedly make the same decision about a given input, whereas imprecise judges will make different decisions. The causes are unclear. We examined this using audio-visual (AV) timing and confidence judgments, in conjunction with electroencephalography (EEG) and multivariate pattern classification analyses. One plausible cause of differences in timing precision is that it scales with variance in the dynamics of evoked brain activity. Another possibility is that equally reliable patterns of brain activity are evoked, but there are systematic differences that scale with precision. Trial-by-trial decoding of input timings from brain activity suggested precision differences may not result from variable dynamics. Instead, precision was associated with evoked responses that were exaggerated (more different from baseline) ~300 ms after initial physical stimulations. We suggest excitatory and inhibitory interactions within a winner-take-all neural code for AV timing might exaggerate responses, such that evoked response magnitudes post-stimulation scale with encoding success.

Auditory Perceptual History Is Propagated through Alpha Oscillations

Perception is a proactive, “predictive” process, in which the brain relies, at least in part, on accumulated experience to make best guesses about the world to test against sensory data, updating the guesses as new experience is acquired. Using novel behavioral methods, the present study demonstrates the role of alpha rhythms in communicating past perceptual experience. Participants were required to discriminate the ear of origin of brief sinusoidal tones that were presented monaurally at random times within a burst of uncorrelated dichotic white noise masks. Performance was not constant but varied with delay after noise onset in an oscillatory manner at about 9 Hz (alpha rhythm). Importantly, oscillations occurred only for trials preceded by a target tone to the same ear, either on the previous trial or two trials back. These results suggest that communication of perceptual history generates neural oscillations within specific perceptual circuits, strongly implicating behavioral oscillations in predictive perception and with formation of working memory.

•

We demonstrate the role of alpha rhythms in the propagation of perceptual history

•

Auditory decisions were rhythmically biased by stimuli presented 1 or 2 trials back

•

Bias oscillated at ∼9 Hz only when successive stimuli occurred in the same ear

•

Alpha is strongly implicated in predictive perception and working memory formation

The Common Rhythm of Action and Perception

Research in the last decade has undermined the idea of perception as a continuous process, providing strong empirical support for its rhythmic modulation. More recently, it has been revealed that the ongoing motor processes influence the rhythmic sampling of sensory information. In this review, we will focus on a growing body of evidence suggesting that oscillation-based mechanisms may structure the dynamic interplay between the motor and sensory system and provide a unified temporal frame for their effective coordination. We will describe neurophysiological data, primarily collected in animals, showing phase-locking of neuronal oscillations to the onset of (eye) movements. These data are complemented by novel evidence in humans, which demonstrate the behavioral relevance of these oscillatory modulations and their domain-general nature. Finally, we will discuss the possible implications of these modulations for action-perception coupling mechanisms.

Superior Visual Timing Sensitivity in Auditory But Not Visual World Class Drum Corps Experts

World class drum corps require cooperation among performance artists to render precisely synchronized and asynchronized events. For example, drum corps visual aesthetics often feature salient radial and rotational motion displays from the color guard. Accordingly, extensive color guard training might predict superior visual timing sensitivity to asynchronies in radial and rotational motion displays. Less intuitively, one might instead predict superior visual timing sensitivity among world class drum corps musicians, who regularly subdivide musical tempos into brief time units. This prediction arises from the possibility that auditory training transfers cross-modally. Here, we investigated whether precise visual temporal order judgments (TOJs) more strongly align with color guard’s visual training or musicians’ auditory training. To mimic color guard visual displays, stimuli comprised bilateral plaid patterns that radiated or rotated before changing direction asynchronously. Human participants indicated whether the direction changed first on the left or right, called a TOJ. Twenty-five percussionists, 67 brass players, and 29 color guard members from a world class drum corps collectively completed 67,760 visual TOJ trials. Percussionists exhibited significantly lower TOJ thresholds than did brass players, who exhibited significantly lower TOJ thresholds than did the color guard. Group median thresholds spanned an order of magnitude, ranging between 29 ms (percussionists judging rotational asynchronies) and 290 ms (color guard judging radial asynchronies). The results suggest that visual timing can improve more by training cross-modally than intramodally, even when intramodal training and testing stimuli closely match. More broadly, pre-existing training histories can provide a unique window into the timing sensitivity of the nervous system.

Neural entrainment and network resonance in support of top-down guided attention

Which neural mechanisms provide the functional basis of top-down guided cognitive control? Here, we review recent evidence that suggest that the neural basis of attention is inherently rhythmic. In particular, we discuss two physical properties of self-sustained networks, namely entrainment and resonance, and how these shape the timescale of attentional control. Several recent findings revealed theta-band (3-8 Hz) dynamics in top-down guided behavior. These reports were paralleled by intracranial recordings, which implicated theta oscillations in the organization of functional attention networks. We discuss how the intrinsic network architecture shapes covert attentional sampling as well as overt behavior. Taken together, we posit that theta rhythmicity is an inherent feature of the attention network in support of top-down guided goal-directed behavior.

Rhythmic motor behaviour influences perception of visual time

Temporal processing is fundamental for an accurate synchronization between motor behaviour and sensory processing. Here, we investigate how motor timing during rhythmic tapping influences perception of visual time. Participants listen to a sequence of four auditory tones played at 1 Hz and continue the sequence (without auditory stimulation) by tapping four times with their finger. During finger tapping, they are presented with an empty visual interval and are asked to judge its length compared to a previously internalized interval of 150 ms. The visual temporal estimates show non-monotonic changes locked to the finger tapping: perceived time is maximally expanded at halftime between the two consecutive finger taps, and maximally compressed near tap onsets. Importantly, the temporal dynamics of the perceptual time distortion scales linearly with the timing of the motor tapping, with maximal expansion always being anchored to the centre of the inter-tap interval. These results reveal an intrinsic coupling between distortion of perceptual time and production of self-timed motor rhythms, suggesting the existence of a timing mechanism that keeps perception and action accurately synchronized.