Multiple concurrent temporal recalibrations driven by audiovisual stimuli with apparent physical differences

No effect of spatial congruence on rapid temporal recalibration to audiovisual asynchrony

The brain integrates multisensory information to construct coherent perceptual representations based on spatial and temporal congruence. Intriguingly, multisensory timing perception can be flexibly calibrated. Repeated exposure to audiovisual asynchrony induces shifts in subjective simultaneity (temporal recalibration). Spatial congruence is known to serve as a grouping cue for recalibration when the audiovisual temporal relationship is ambiguous during exposure. A single exposure to audiovisual asynchrony can also trigger temporal recalibration (rapid recalibration). However, it has been suggested that the underlying mechanisms of these temporal recalibrations differ. Here, we examined whether spatial congruence can be a grouping cue for rapid recalibration when audiovisual pairs are not defined by temporal relationships. Participants made a simultaneity judgment for a pair of audiovisual stimuli after adapting three consecutive stimuli once in a "light-sound-light" or "sound-light-sound" order with an equal temporal interval. The spatial positions of the adapting stimuli were manipulated as an audiovisual pair from the same position (e.g., left) and the remaining stimulus from another position (e.g., right). In three experiments, the spatial congruence of the audiovisual adapting stimuli did not show a modulatory effect, while we replicated the rapid recalibration effects. Rather, rapid recalibration occurred according to the temporal order of the first light and sound. Our findings suggest that, in contrast to temporal recalibration with repeated exposure, the perceptual systems underlying rapid recalibration simply combine individual visual and auditory inputs based on the order in which they arrive.

Cross-modal correspondence between auditory pitch and visual elevation modulates audiovisual temporal recalibration

Cross-modal correspondences refer to associations between feature dimensions of stimuli across sensory modalities. Research has indicated that correspondence between audiovisual stimuli influences whether these stimuli are integrated or segregated. On the other hand, the audiovisual integration process plastically changes to compensate for continuously observed spatiotemporal conflicts between sensory modalities. If and how cross-modal correspondence modulates the "recalibration" of integration is unclear. We investigated whether cross-modal correspondence between auditory pitch and visual elevation affected audiovisual temporal recalibration. Participants judged the simultaneity of a pair of audiovisual stimuli after an adaptation phase in which alternating auditory and visual stimuli equally spaced in time were presented. In the adaptation phase, auditory pitch and visual elevation were manipulated to fix the order within each pairing of audiovisual stimuli congruent with pitch-elevation correspondence (visual leading or auditory leading). We found a shift in the point of subjective simultaneity (PSS) between congruent audiovisual stimuli as a function of the adaptation conditions (Experiment 1, 2), but this shift in the PSS was not observed within incongruent pairs (Experiment 2). These results indicate that asynchronies between audiovisual signals congruent with cross-modal correspondence are selectively recalibrated.

Audio-motor but not visuo-motor temporal recalibration speeds up sensory processing

Perception of synchrony between one's own action (a finger tap) and the sensory feedback thereof (a visual flash or an auditory pip) can be recalibrated after exposure to an artificially inserted delay between them (temporal recalibration effect: TRE). TRE might be mediated by a compensatory shift of motor timing (when did I tap?) and/or the sensory timing of the feedback (when did I hear/see the feedback?). To examine this, we asked participants to voluntarily tap their index finger at a constant pace while receiving visual or auditory feedback (a flash or pip) that was either synced or somewhat delayed relative to the tap. Following this exposure phase, they then performed a simple reaction time (RT) task to measure the sensory timing of the exposure stimulus, and a sensorimotor synchronization (SMS) task (tapping in synchrony with a flash or pip as pacing stimulus) to measure the point of subjective synchrony between the tap and pacing stimulus. The results showed that after exposure to delayed auditory feedback, participants tapped earlier (~21.5 ms) relative to auditory pacing stimuli (= temporal recalibration) and reacted faster (~5.6 ms) to auditory stimuli. For visual exposure and test stimuli, there were no such compensatory effects. These results indicate that adjustments of audio-motor synchrony can to some extent be explained by a change in the speed of auditory sensory processing. We discuss this in terms of an attentional modulation of sensory processing.

Differences in the Association between Segment and Language: Early Bilinguals Pattern with Monolinguals and Are Less Accurate than Late Bilinguals

Early bilinguals often show as much sensitivity to L2-specific contrasts as monolingual speakers of the L2, but most work on cross-language speech perception has focused on isolated segments, and typically only on neighboring vowels or stop contrasts. In tasks that include sounds in context, listeners’ success is more variable, so segment discrimination in isolation may not adequately represent the phonetic detail in stored representations. The current study explores the relationship between language experience and sensitivity to segmental cues in context by comparing the categorization patterns of monolingual English listeners and early and late Spanish–English bilinguals. Participants categorized nonce words containing different classes of English- and Spanish-specific sounds as being more English-like or more Spanish-like; target segments included phonemic cues, cues for which there is no analogous sound in the other language, or phonetic cues, cues for which English and Spanish share the category but for which each language varies in its phonetic implementation. Listeners’ language categorization accuracy and reaction times were analyzed. Our results reveal a largely uniform categorization pattern across listener groups: Spanish cues were categorized more accurately than English cues, and phonemic cues were easier for listeners to categorize than phonetic cues. There were no differences in the sensitivity of monolinguals and early bilinguals to language-specific cues, suggesting that the early bilinguals’ exposure to Spanish did not fundamentally change their representations of English phonology. However, neither did the early bilinguals show more sensitivity than the monolinguals to Spanish sounds. The late bilinguals however, were significantly more accurate than either of the other groups. These findings indicate that listeners with varying exposure to English and Spanish are able to use language-specific cues in a nonce-word language categorization task. Differences in how, and not only when, a language was acquired may influence listener sensitivity to more difficult cues, and the advantage for phonemic cues may reflect the greater salience of categories unique to each language. Implications for foreign-accent categorization and cross-language speech perception are discussed, and future directions are outlined to better understand how salience varies across language-specific phonemic and phonetic cues.