Audiovisual temporal integration: Cognitive processing, neural mechanisms, developmental trajectory and potential interventions

When 2 become 1: Autistic simultaneity judgements about asynchronous audiovisual speech

It has been proposed that autistic people experience a temporal distortion whereby the temporal binding window of multisensory integration is extended. Research to date has focused on autistic children so whether these differences persist into adulthood remains unknown. In addition, the possibility that the previous observations have arisen from between-group differences in response bias, rather than perceptual differences, has not been addressed. Participants completed simultaneity judgements of audiovisual speech stimuli across a range of stimulus-onset asynchronies. Response times and accuracy data were fitted to a drift-diffusion model so that the drift rate (a measure of processing efficiency) and starting point (response bias) could be estimated. In Experiment 1, we tested a sample of non-autistic adults who completed the Autism Quotient questionnaire. Autism Quotient score was not correlated with either drift rate or response bias, nor were there between-group differences when splitting based on the first and third quantiles of scores. In Experiment 2, we compared the performance of autistic with a group of non-autistic adults. There were no between-group differences in either drift rate or starting point. The results of this study do not support the previous suggestion that autistic people have an extended temporal binding window for audiovisual speech. In addition, exploratory analysis revealed that operationalising the temporal binding window in different ways influenced whether a group difference was observed, which is an important consideration for future work.

Memory, attention and creativity as cognitive processes in musical performance: A case study of students and professionals among non-musicians and musicians

This research discusses that cognitive processes such as memory, attention and creativity differ in students and professionals, among musicians and non-musicians, dealing with musical performance. The purpose of the study was to evaluate and compare the role of memory, attention and creativity as cognitive processes in musical performance, focusing on the differences between non-musicians and musicians. The sample involved 400 individuals, students and professionals, specialising in music and economics. The research instruments used by the scholars were the Wechsler Memory Scale, the Conners Performance Test, and the Torrance Tests of Creative Thinking. Musical students possessed better-developed auditory and short-term memory, while professional musicians had better auditory, visual working and short-term memory. Analysis of attention reveals that music students score better than non-musicians on all four aspects: inattention, impulsivity, sustained attention, and vigilance. For professionals, the key aspects are impulsivity and sustained attention with better results revealed in musicians. Creative thinking was the only factor where the differences were statistically significant in all five scales and the findings proved that creativity was better developed among musicians. This study provides an in-depth analysis and adds new knowledge to existing literature and empirical data on the cognitive processes associated with musical performance, focusing on memory, attention and creativity. By examining the differences between non-musicians and musicians, as well as students and professionals, the study provides insight into how musical performance can be used as a way to develop these cognitive processes.

Psychological dissociation and temporal integration/segregation across the senses: An experimental study

There are no studies that have experimentally tested how temporal integration/segregation of sensory inputs might be linked to the emergence of dissociative experiences and alterations of emotional functioning. Thirty-six participants completed 3 sensory integration tasks. Psychometric thresholds were estimated as indexes of temporal integration/segregation processes. We collected self-report measures of pre-task trait levels of dissociation, as well as pre- post-task changes in both dissociation and emotionality. An independent sample of 21 subjects completed a control experiment administering the Attention Network Test. Results showed: (i) a significant increase of dissociative experiences after the completion of sensory integration tasks, but not after the ANT task; (ii) that subjective thresholds predicted the emergence of dissociative states; (iii) temporal integration efforts affected positive emotionality, which was explained by the extent of task-dependent dissociative states. The present findings reveal that dissociation could be understood in terms of an imbalance between "hyper-segregation" and "hyper-integration" processes.

Direct piriform-to-auditory cortical projections shape auditory-olfactory integration

In a real-world environment, the brain must integrate information from multiple sensory modalities, including the auditory and olfactory systems. However, little is known about the neuronal circuits governing how odors influence and modulate sound processing. Here, we investigated the mechanisms underlying auditory-olfactory integration using anatomical, electrophysiological, and optogenetic approaches, focusing on the auditory cortex as a key locus for cross-modal integration. First, retrograde and anterograde viral tracing strategies revealed a direct projection from the piriform cortex to the auditory cortex. Next, using in vivo electrophysiological recordings of neuronal activity in the auditory cortex of awake mice, we found that odor stimuli modulate auditory cortical responses to sound. Finally, we used in vivo optogenetic manipulations during electrophysiology to demonstrate that olfactory modulation in auditory cortex, specifically, odor-driven enhancement of sound responses, depends on direct input from the piriform cortex. Together, our results identify a novel cortical circuit shaping olfactory modulation in the auditory cortex, shedding new light on the neuronal mechanisms underlying auditory-olfactory integration.

Synchrony perception across senses: A systematic review of temporal binding window changes from infancy to adolescence in typical and atypical development

Sensory integration is increasingly acknowledged as being crucial for the development of cognitive and social abilities. However, its developmental trajectory is still little understood. This systematic review delves into the topic by investigating the literature about the developmental changes from infancy through adolescence of the Temporal Binding Window (TBW) - the epoch of time within which sensory inputs are perceived as simultaneous and therefore integrated. Following comprehensive searches across PubMed, Elsevier, and PsycInfo databases, only experimental, behavioral, English-language, peer-reviewed studies on multisensory temporal processing in 0-17-year-olds have been included. Non-behavioral, non-multisensory, and non-human studies have been excluded as those that did not directly focus on the TBW. The selection process was independently performed by two Authors. The 39 selected studies involved 2859 participants in total. Findings indicate a predisposition towards cross-modal asynchrony sensitivity and a composite, still unclear, developmental trajectory, with atypical development associated to increased asynchrony tolerance. These results highlight the need for consistent and thorough research into TBW development to inform potential interventions.

The anterior cingulate cortex is involved in intero-exteroceptive integration for spatial image transformation of the self-body

Spatial image transformation of the self-body is a fundamental function of visual perspective-taking. Recent research underscores the significance of intero-exteroceptive information integration to construct representations of our embodied self. This raises the intriguing hypothesis that interoceptive processing might be involved in the spatial image transformation of the self-body. To test this hypothesis, the present study used functional magnetic resonance imaging to measure brain activity during an arm laterality judgment (ALJ) task. In this task, participants were tasked with discerning whether the outstretched arm of a human figure, viewed from the front or back, was the right or left hand. The reaction times for the ALJ task proved longer when the stimulus presented orientations of 0°, 90°, and 270° relative to the upright orientation, and when the front view was presented rather than the back view. Reflecting the increased reaction time, increased brain activity was manifested in a cluster centered on the dorsal anterior cingulate cortex (ACC), suggesting that the activation reflects the involvement of an embodied simulation in ALJ. Furthermore, this cluster of brain activity exhibited overlap with regions where the difference in activation between the front and back views positively correlated with the participants' interoceptive sensitivity, as assessed through the heartbeat discrimination task, within the pregenual ACC. These results suggest that the ACC plays an important role in integrating intero-exteroceptive cues to spatially transform the image of our self-body.

The Role of Letter-Speech Sound Integration in Native and Second Language Reading: A Study in Native Japanese Readers Learning English

The automatic activation of letter-speech sound (L-SS) associations is a vital step in typical reading acquisition. However, the contribution of L-SS integration during nonalphabetic native and alphabetic second language (L2) reading remains unclear. This study explored whether L-SS integration plays a similar role in a nonalphabetic language as in alphabetic languages and its contribution to L2 reading among native Japanese-speaking adults with varying English proficiency. A priming paradigm in Japanese and English was performed by presenting visual letters or symbols, followed by auditory sounds. We compared behavioral and event-related responses elicited by congruent letter-sound pairs, incongruent pairs, and baseline condition (symbol-sound pairs). The behavioral experiment revealed shorter RTs in the congruent condition for Japanese and English tasks, suggesting a facilitation effect of congruency. The ERP experiment results showed an increased early N1 response to Japanese congruent pairs compared to corresponding incongruent stimuli at the left frontotemporal electrodes. Interestingly, advanced English learners exhibited greater activities in bilateral but predominantly right-lateralized frontotemporal regions for the congruent condition within the N1 time window. Moreover, the enhancement of P2 response to congruent pairs was observed in intermediate English learners. These findings indicate that, despite deviations from native language processing, advanced speakers may successfully integrate letters and sounds during English reading, whereas intermediate learners may encounter difficulty in achieving L-SS integration when reading L2. Furthermore, our results suggest that L2 proficiency may affect the level of automaticity in L-SS integration, with the right P2 congruency effect playing a compensatory role for intermediate learners.

Greater utilization of executive functions networks when listening to stories with visual stimulation is related to lower reading abilities in children

Narrative comprehension relies on basic sensory processing abilities, such as visual and auditory processing, with recent evidence for utilizing executive functions (EF), which are also engaged during reading. EF was previously related to the "supporter" of engaging the auditory and visual modalities in different cognitive tasks, with evidence of lower efficiency in this process among those with reading difficulties in the absence of a visual stimulus (i.e. while listening to stories). The current study aims to fill out the gap related to the level of reliance on these neural circuits while visual aids (pictures) are involved during story listening in relation to reading skills. Functional MRI data were collected from 44 Hebrew-speaking children aged 8-12 years while listening to stories with vs without visual stimuli (i.e., pictures). Functional connectivity of networks supporting reading was defined in each condition and compared between the conditions against behavioral reading measures. Lower reading skills were related to greater functional connectivity values between EF networks (default mode and memory networks), and between the auditory and memory networks for the stories with vs without the visual stimulation. A greater difference in functional connectivity between the conditions was related to lower reading scores. We conclude that lower reading skills in children may be related to a need for greater scaffolding, i.e., visual stimulation such as pictures describing the narratives when listening to stories, which may guide future intervention approaches.

Audio-Visual Fission Illusion and Individual Alpha Frequency: Perspective on Buergers and Noppeney (2022)

Integrating visual and auditory information is an important ability in various cognitive processes, although its neural mechanisms remain unclear. Several studies indicated a close relationship between one's temporal binding window (TBW) for audio-visual interaction and their alpha rhythm in the brain (individual alpha frequency or IAF). A recent study by Buergers and Noppeney [Buergers, S., & Noppeney, U. The role of alpha oscillations in temporal binding within and across the senses. Nature Human Behaviour, 6, 732-742, 2022], however, challenged this view using a new approach to analyze behavioral data. Conforming to the same procedures by Buergers and Noppeney, here, I analyzed the data of my previous study and examined a relationship between TBW and IAF. In contrast to Buergers and Noppeney, a significant correlation was found between occipital IAF and a new behavioral measure of TBW. Some possibilities that caused these opposing results, such as a variability of "alpha band" across studies and a large inter-individual difference in magnitude of the fission illusion, are discussed.

Pitch Improvement in Attentional Blink: A Study across Audiovisual Asymmetries

Attentional blink (AB) is a phenomenon in which the perception of a second target is impaired when it appears within 200-500 ms after the first target. Sound affects an AB and is accompanied by the appearance of an asymmetry during audiovisual integration, but it is not known whether this is related to the tonal representation of sound. The aim of the present study was to investigate the effect of audiovisual asymmetry on attentional blink and whether the presentation of pitch improves the ability to detect a target during an AB that is accompanied by audiovisual asymmetry. The results showed that as the lag increased, the subject's target recognition improved and the pitch produced further improvements. These improvements exhibited a significant asymmetry across the audiovisual channel. Our findings could contribute to better utilizations of audiovisual integration resources to improve attentional transients and auditory recognition decline, which could be useful in areas such as driving and education.

The characteristics of audiovisual temporal integration in streaming-bouncing bistable motion perception: considering both implicit and explicit processing perspectives

This study explored the behavioral and neural activity characteristics of audiovisual temporal integration in motion perception from both implicit and explicit perspectives. The streaming-bouncing bistable paradigm (SB task) was employed to investigate implicit temporal integration, while the corresponding simultaneity judgment task (SJ task) was used to examine explicit temporal integration. The behavioral results revealed a negative correlation between implicit and explicit temporal processing. In the ERP results of both tasks, three neural phases (PD100, ND180, and PD290) in the fronto-central region were identified as reflecting integration effects and the auditory-evoked multisensory N1 component may serve as a primary component responsible for cross-modal temporal processing. However, there were significant differences between the VA ERPs in the SB and SJ tasks and the influence of speed on implicit and explicit integration effects also varied. The aforementioned results, building upon the validation of previous temporal renormalization theory, suggest that implicit and explicit temporal integration operate under distinct processing modes within a shared neural network. This underscores the brain's flexibility and adaptability in cross-modal temporal processing.

Audiovisual spatial ventriloquism is reduced in musicians

There is great scientific and public interest in claims that musical training improves general cognitive and perceptual abilities. While this is controversial, recent and rather convincing evidence suggests that musical training refines the temporal integration of auditory and visual stimuli at a general level. We investigated whether musical training also affects integration in the spatial domain, via an auditory localisation experiment that measured ventriloquism (where localisation is biased towards visual stimuli on audiovisual trials) and recalibration (a unimodal localisation aftereffect). While musicians (n = 22) and non-musicians (n = 22) did not have significantly different unimodal precision or accuracy, musicians were significantly less susceptible than non-musicians to ventriloquism, with large effect sizes. We replicated these results in another experiment with an independent sample of 24 musicians and 21 non-musicians. Across both experiments, spatial recalibration did not significantly differ between the groups even though musicians resisted ventriloquism. Our results suggest that the multisensory expertise afforded by musical training refines spatial integration, a process that underpins multisensory perception.

A Randomized Controlled Trial for Audiovisual Multisensory Perception in Autistic Youth

Differences in audiovisual integration are commonly observed in autism. Temporal binding windows (TBWs) of audiovisual speech can be trained (i.e., narrowed) in non-autistic adults; this study evaluated a computer-based perceptual training in autistic youth and assessed whether treatment outcomes varied according to individual characteristics. Thirty autistic youth aged 8-21 were randomly assigned to a brief perceptual training (n = 15) or a control condition (n = 15). At post-test, the perceptual training group did not differ, on average, on TBWs for trained and untrained stimuli and perception of the McGurk illusion compared to the control group. The training benefited youth with higher language and nonverbal IQ scores; the training caused widened TBWs in youth with co-occurring cognitive and language impairments.

Past and present experience shifts audiovisual temporal perception in rats

Our brains have a propensity to integrate closely-timed auditory and visual stimuli into a unified percept; a phenomenon that is highly malleable based on prior sensory experiences, and is known to be altered in clinical populations. While the neural correlates of audiovisual temporal perception have been investigated using neuroimaging and electroencephalography techniques in humans, animal research will be required to uncover the underlying cellular and molecular mechanisms. Prior to conducting such mechanistic studies, it is important to first confirm the translational potential of any prospective animal model. Thus, in the present study, we conducted a series of experiments to determine if rats show the hallmarks of audiovisual temporal perception observed in neurotypical humans, and whether the rat behavioral paradigms could reveal when they experienced perceptual disruptions akin to those observed in neurodevelopmental disorders. After training rats to perform a temporal order judgment (TOJ) or synchrony judgment (SJ) task, we found that the rats' perception was malleable based on their past and present sensory experiences. More specifically, passive exposure to asynchronous audiovisual stimulation in the minutes prior to behavioral testing caused the rats' perception to predictably shift in the direction of the leading stimulus; findings which represent the first time that this form of audiovisual perceptual malleability has been reported in non-human subjects. Furthermore, rats performing the TOJ task also showed evidence of rapid recalibration, in which their audiovisual temporal perception on the current trial was predictably influenced by the timing lag between the auditory and visual stimuli in the preceding trial. Finally, by manipulating either experimental testing parameters or altering the rats' neurochemistry with a systemic injection of MK-801, we showed that the TOJ and SJ tasks could identify when the rats had difficulty judging the timing of audiovisual stimuli. These findings confirm that the behavioral paradigms are indeed suitable for future testing of rats with perceptual disruptions in audiovisual processing. Overall, our collective results highlight that rats represent an excellent animal model to study the cellular and molecular mechanisms underlying the acuity and malleability of audiovisual temporal perception, as they showcase the perceptual hallmarks commonly observed in humans.

Neural oscillations reflect the individual differences in the temporal perception of audiovisual speech

Multisensory integration occurs within a limited time interval between multimodal stimuli. Multisensory temporal perception varies widely among individuals and involves perceptual synchrony and temporal sensitivity processes. Previous studies explored the neural mechanisms of individual differences for beep-flash stimuli, whereas there was no study for speech. In this study, 28 subjects (16 male) performed an audiovisual speech/ba/simultaneity judgment task while recording their electroencephalography. We examined the relationship between prestimulus neural oscillations (i.e. the pre-pronunciation movement-related oscillations) and temporal perception. The perceptual synchrony was quantified using the Point of Subjective Simultaneity and temporal sensitivity using the Temporal Binding Window. Our results revealed dissociated neural mechanisms for individual differences in Temporal Binding Window and Point of Subjective Simultaneity. The frontocentral delta power, reflecting top-down attention control, is positively related to the magnitude of individual auditory leading Temporal Binding Windows (auditory Temporal Binding Windows; LTBWs), whereas the parieto-occipital theta power, indexing bottom-up visual temporal attention specific to speech, is negatively associated with the magnitude of individual visual leading Temporal Binding Windows (visual Temporal Binding Windows; RTBWs). In addition, increased left frontal and bilateral temporoparietal occipital alpha power, reflecting general attentional states, is associated with increased Points of Subjective Simultaneity. Strengthening attention abilities might improve the audiovisual temporal perception of speech and further impact speech integration.

Multisensory perception constrains the formation of object categories: a review of evidence from sensory-driven and predictive processes on categorical decisions

Although object categorization is a fundamental cognitive ability, it is also a complex process going beyond the perception and organization of sensory stimulation. Here we review existing evidence about how the human brain acquires and organizes multisensory inputs into object representations that may lead to conceptual knowledge in memory. We first focus on evidence for two processes on object perception, multisensory integration of redundant information (e.g. seeing and feeling a shape) and crossmodal, statistical learning of complementary information (e.g. the 'moo' sound of a cow and its visual shape). For both processes, the importance attributed to each sensory input in constructing a multisensory representation of an object depends on the working range of the specific sensory modality, the relative reliability or distinctiveness of the encoded information and top-down predictions. Moreover, apart from sensory-driven influences on perception, the acquisition of featural information across modalities can affect semantic memory and, in turn, influence category decisions. In sum, we argue that both multisensory processes independently constrain the formation of object categories across the lifespan, possibly through early and late integration mechanisms, respectively, to allow us to efficiently achieve the everyday, but remarkable, ability of recognizing objects. This article is part of the theme issue 'Decision and control processes in multisensory perception'.

Anterior cingulate glutamate levels associate with functional activation and connectivity during sensory integration in schizophrenia: a multimodal 1H-MRS and fMRI study

BACKGROUND

Glutamatergic dysfunction has been implicated in sensory integration deficits in schizophrenia, yet how glutamatergic function contributes to behavioural impairments and neural activities of sensory integration remains unknown.

METHODS

Fifty schizophrenia patients and 43 healthy controls completed behavioural assessments for sensory integration and underwent magnetic resonance spectroscopy (MRS) for measuring the anterior cingulate cortex (ACC) glutamate levels. The correlation between glutamate levels and behavioural sensory integration deficits was examined in each group. A subsample of 20 pairs of patients and controls further completed an audiovisual sensory integration functional magnetic resonance imaging (fMRI) task. Blood Oxygenation Level Dependent (BOLD) activation and task-dependent functional connectivity (FC) were assessed based on fMRI data. Full factorial analyses were performed to examine the Group-by-Glutamate Level interaction effects on fMRI measurements (group differences in correlation between glutamate levels and fMRI measurements) and the correlation between glutamate levels and fMRI measurements within each group.

RESULTS

We found that schizophrenia patients exhibited impaired sensory integration which was positively correlated with ACC glutamate levels. Multimodal analyses showed significantly Group-by-Glutamate Level interaction effects on BOLD activation as well as task-dependent FC in a 'cortico-subcortical-cortical' network (including medial frontal gyrus, precuneus, ACC, middle cingulate gyrus, thalamus and caudate) with positive correlations in patients and negative in controls.

CONCLUSIONS

Our findings indicate that ACC glutamate influences neural activities in a large-scale network during sensory integration, but the effects have opposite directionality between schizophrenia patients and healthy people. This implicates the crucial role of glutamatergic system in sensory integration processing in schizophrenia.

Neural correlates of audiovisual speech synchrony perception and its relationship with autistic traits

The anterior insula (AI) has the central role in coordinating attention and integrating information from multiple sensory modalities. AI dysfunction may contribute to both sensory and social impairments in autism spectrum disorder (ASD). Little is known regarding the brain mechanisms that guide multisensory integration, and how such neural activity might be affected by autistic-like symptoms in the general population. In this study, 72 healthy young adults performed an audiovisual speech synchrony judgment (SJ) task during fMRI scanning. We aimed to investigate the SJ-related brain activations and connectivity, with a focus on the AI. Compared with synchronous speech, asynchrony perception triggered stronger activations in the bilateral AI, and other frontal-cingulate-parietal regions. In contrast, synchronous perception resulted in greater involvement of the primary auditory and visual areas, indicating multisensory validation and fusion. Moreover, the AI demonstrated a stronger connection with the anterior cingulate gyrus (ACC) in the audiovisual asynchronous (vs. synchronous) condition. To facilitate asynchrony detection, the AI may integrate auditory and visual speech stimuli, and generate a control signal to the ACC that further supports conflict-resolving and response selection. Correlation analysis, however, suggested that audiovisual synchrony perception and its related AI activation and connectivity did not significantly vary with different levels of autistic traits. These findings provide novel evidence for the neural mechanisms underlying multisensory temporal processing in healthy people. Future research should examine whether such findings would be extended to ASD patients.

Features of Cognitive Ability and Central Auditory Processing of Preschool Children With Minimal and Mild Hearing Loss

OBJECTIVE

This study aimed to investigate the current status of cognitive development and central auditory processing development of preschool children with minimal and mild hearing loss (MMHL) in Nanjing, China.

METHOD

We recruited 34 children with MMHL and 45 children with normal hearing (NH). They completed a series of tests, including cognitive tests (i.e., Wechsler Preschool and Primary Scale of Intelligence and Continuous Performance Test), behavioral auditory tests (speech-in-noise [SIN] test and frequency pattern test), and objective electrophysiological audiometry (speech-evoked auditory brainstem response and cortical auditory evoked potential). In addition, teacher evaluations and demographic information and questionnaires completed by parents were collected.

RESULTS

Regarding cognitive ability, statistical differences in the verbal comprehensive index, full-scale intelligence quotient, and abnormal rate of attention test score were found between the MMHL group and the NH group. The children with MMHL performed poorer on the SIN test than the children with NH. As for the auditory electrophysiology of the two groups, the latency and amplitude of some waves of the speech-evoked auditory brainstem response and cortical auditory evoked potential were statistically different between the two groups. We attempted to explore the relationship between some key indicators of auditory processing and some key indicators of cognitive development.

CONCLUSIONS

Children with MMHL are already at increased developmental risk as early as preschool. They are more likely to have problems with attention and verbal comprehension than children with NH. This condition is not compensated with increasing age during the preschool years. The results suggest a possible relationship between the risk of cognitive deficit and divergence of auditory processing.

SUPPLEMENTAL MATERIAL

https://doi.org/10.23641/asha.22670473.

Long-term Tai Chi training reduces the fusion illusion in older adults

Sound-induced flash illusion (SiFI) is an auditory-dominated audiovisual integration phenomenon that can be used as a reliable indicator of audiovisual integration. Although previous studies have found that Tai Chi exercise has a promoting effect on cognitive processing, such as executive functions, the effect of Tai Chi exercise on early perceptual processing has yet to be investigated. This study used the classic SiFI paradigm to investigate the effects of long-term Tai Chi exercise on multisensory integration in older adults. We compared older adults with long-term Tai Chi exercise experience with those with long-term walking exercise. The results showed that the accuracy of the Tai Chi group was higher than that of the control group under the fusion illusion condition, mainly due to the increased perceptual sensitivity to flashes. However, there was no significant difference between the two groups in the fission illusion. These results indicated that the fission and fusion illusions were affected differently by Tai Chi exercise, and this was attributable to the association of the participants' flash discriminability with them. The present study provides preliminary evidence that long-term Tai Chi exercise improves older adults' multisensory integration, which occurs in early perceptual processing.

Temporal processing deficit in children and adolescents with autism spectrum disorder: An online assessment

Objective

The sensory deficit has been considered as one of the core features in children and adolescents with autism spectrum disorder (ASD). The present study aimed to examine the temporal processing of simple and more complex auditory inputs in ASD children and adolescents with an online assessment that can be conducted remotely.

Methods

One hundred fifty-eight children and adolescents aged 5-17 years participated in this study, including 79 ASD participants and 79 typically developing (TD) participants. The online assessment consisted of two temporal-order judgment tasks that required repeating the sequence of two pure tones or consonant-vowel (CV) syllabic pairs at varying interstimulus intervals (ISIs).

Results

Significantly lower accuracy rates were found in ASD than TD participants in the pure tone and the CV conditions with both short and long ISI. In addition, ASD participants (M  =  245.97 ms) showed a significantly higher passing threshold than TD participants (M  =  178.84 ms) in the CV task. Receiver operating characteristic analysis found that the age × ISI passing threshold composite yielded a sensitivity of 74.7% and a specificity of 50.6% at the cutoff point of -0.307 in differentiating ASD participants from TD participants.

Conclusion

In sum, children and adolescents with ASD showed impaired temporal processing of both simple and more complex auditory stimuli, and the online assessment seems to be sensitive in differentiating individuals with ASD from those with TD.

Auditory attentional load modulates the temporal dynamics of audiovisual integration in older adults: An ERPs study

As older adults experience degenerations in perceptual ability, it is important to gain perception from audiovisual integration. Due to attending to one or more auditory stimuli, performing other tasks is a common challenge for older adults in everyday life. Therefore, it is necessary to probe the effects of auditory attentional load on audiovisual integration in older adults. The present study used event-related potentials (ERPs) and a dual-task paradigm [Go / No-go task + rapid serial auditory presentation (RSAP) task] to investigate the temporal dynamics of audiovisual integration. Behavioral results showed that both older and younger adults responded faster and with higher accuracy to audiovisual stimuli than to either visual or auditory stimuli alone. ERPs revealed weaker audiovisual integration under the no-attentional auditory load condition at the earlier processing stages and, conversely, stronger integration in the late stages. Moreover, audiovisual integration was greater in older adults than in younger adults at the following time intervals: 60–90, 140–210, and 430–530 ms. Notably, only under the low load condition in the time interval of 140–210 ms, we did find that the audiovisual integration of older adults was significantly greater than that of younger adults. These results delineate the temporal dynamics of the interactions with auditory attentional load and audiovisual integration in aging, suggesting that modulation of auditory attentional load affects audiovisual integration, enhancing it in older adults.

The influence of temporal asynchrony on character-speech integration in Chinese children with and without dyslexia: An ERP study

Dyslexic readers have been reported to show abnormal temporal acuity and multisensory integration deficiency. Here, we investigated the influence of temporal intervals on Chinese character-speech integration in children with and without dyslexia. Visual characters were presented synchronously to the onset of speech sounds (AV0) or before speech sound by 300 ms (AV300). Event-related potentials (ERP) evoked by congruent condition (speech sounds presented with congruent Chinese characters) and by baseline condition (speech sounds presented with Korean characters) were compared. Typically developing (TD) children exhibited congruency effect in AV0 condition, whereas dyslexic children exhibited congruency effect in AV300 condition. Moreover, congruency effect in TD children was due to enhanced neural activation to congruent trials, congruency effect in dyslexic children was contributed by neural suppression for baseline trials. These results suggested that different underlying mechanisms were involved in character-speech integration for typical and dyslexic children.

Audiovisual temporal processing in adult patients with first-episode schizophrenia and high-functioning autism

Schizophrenia and autism spectrum disorder (ASD) are both neurodevelopmental disorders with altered sensory processing. Widened temporal binding window (TBW) signifies reduced sensitivity to detect stimulus asynchrony, and may be a shared feature in schizophrenia and ASD. Few studies directly compared audiovisual temporal processing ability in the two disorders. We recruited 43 adult patients with first-episode schizophrenia (FES), 35 average intelligent and verbally-fluent adult patients with high-functioning ASD and 48 controls. We employed two unisensory Temporal Order Judgement (TOJ) tasks within visual or auditory modalities, and two audiovisual Simultaneity Judgement (SJ) tasks with flash-beeps and videos of syllable utterance as stimuli. Participants with FES exhibited widened TBW affecting both speech and non-speech processing, which were not attributable to altered unisensory sensory acuity because they had normal visual and auditory TOJ thresholds. However, adults with ASD exhibited intact unisensory and audiovisual temporal processing. Lower non-verbal IQ was correlated with larger TBW width across the three groups. Taking our findings with earlier evidence in chronic samples, widened TBW is associated with schizophrenia regardless illness stage. The altered audiovisual temporal processing in ASD may ameliorate after reaching adulthood.

Musical training refines audiovisual integration but does not influence temporal recalibration

When the brain is exposed to a temporal asynchrony between the senses, it will shift its perception of simultaneity towards the previously experienced asynchrony (temporal recalibration). It is unknown whether recalibration depends on how accurately an individual integrates multisensory cues or on experiences they have had over their lifespan. Hence, we assessed whether musical training modulated audiovisual temporal recalibration. Musicians (n = 20) and non-musicians (n = 18) made simultaneity judgements to flash-tone stimuli before and after adaptation to asynchronous (± 200 ms) flash-tone stimuli. We analysed these judgements via an observer model that described the left and right boundaries of the temporal integration window (decisional criteria) and the amount of sensory noise that affected these judgements. Musicians’ boundaries were narrower (closer to true simultaneity) than non-musicians’, indicating stricter criteria for temporal integration, and they also exhibited enhanced sensory precision. However, while both musicians and non-musicians experienced cumulative and rapid recalibration, these recalibration effects did not differ between the groups. Unexpectedly, cumulative recalibration was caused by auditory-leading but not visual-leading adaptation. Overall, these findings suggest that the precision with which observers perceptually integrate audiovisual temporal cues does not predict their susceptibility to recalibration.

Time in schizophrenia: a link between psychopathology, psychophysics and technology

It has been widely demonstrated that time processing is altered in patients with schizophrenia. This perspective review delves into such temporal deficit and highlights its link to low-level sensory alterations, which are often overlooked in rehabilitation protocols for psychosis. However, if temporal impairment at the sensory level is inherent to the disease, new interventions should focus on this dimension. Beyond more traditional types of intervention, here we review the most recent digital technologies for rehabilitation and the most promising ones for sensory training. The overall aim is to synthesise existing literature on time in schizophrenia linking psychopathology, psychophysics, and technology to help future developments.

With No Attention Specifically Directed to It, Rhythmic Sound Does Not Automatically Facilitate Visual Task Performance

In a century where humans and machines—powered by artificial intelligence or not—increasingly work together, it is of interest to understand human processing of multi-sensory stimuli in relation to attention and working memory. This paper explores whether and when supporting visual information with rhythmic auditory stimuli can optimize multi-sensory information processing. In turn, this can make the interaction between humans or between machines and humans more engaging, rewarding and activating. For this purpose a novel working memory paradigm was developed where participants are presented with a series of five target digits randomly interchanged with five distractor digits. Their goal is to remember the target digits and recall them orally. Depending on the condition support is provided by audio and/or rhythm. It is expected that the sound will lead to a better performance. It is also expected that this effect of sound is different in case of rhythmic and non-rhythmic sound. Last but not least, some variability is expected across participants. To make correct conclusions, the data of the experiment was statistically analyzed in a classic way, but also predictive models were developed in order to predict outcomes based on a range of input variables related to the experiment and the participant. The effect of auditory support could be confirmed, but no difference was observed between rhythmic and non-rhythmic sounds. Overall performance was indeed affected by individual differences, such as visual dominance or perceived task difficulty. Surprisingly a music education did not significantly affect the performance and even tended toward a negative effect. To better understand the underlying processes of attention, also brain activation data, e.g., by means of electroencephalography (EEG), should be recorded. This approach can be subject to a future work.

Audiovisual synchrony detection for fluent speech in early childhood: An eye-tracking study

During childhood, the ability to detect audiovisual synchrony gradually sharpens for simple stimuli such as flashbeeps and single syllables. However, little is known about how children perceive synchrony for natural and continuous speech. This study investigated young children's gaze patterns while they were watching movies of two identical speakers telling stories side by side. Only one speaker's lip movements matched the voices and the other one either led or lagged behind the soundtrack by 600 ms. Children aged 3-6 years (n = 94, 52.13% males) showed an overall preference for the synchronous speaker, with no age-related changes in synchrony-detection sensitivity as indicated by similar gaze patterns across ages. However, viewing time to the synchronous speech was significantly longer in the auditory-leading (AL) condition compared with that in the visual-leading (VL) condition, suggesting asymmetric sensitivities for AL versus VL asynchrony have already been established in early childhood. When further examining gaze patterns on dynamic faces, we found that more attention focused on the mouth region was an adaptive strategy to read visual speech signals and thus associated with increased viewing time of the synchronous videos. Attention to detail, one dimension of autistic traits featured by local processing, has been found to be correlated with worse performances in speech synchrony processing. These findings extended previous research by showing the development of speech synchrony perception in young children, and may have implications for clinical populations (e.g., autism) with impaired multisensory integration.

Individual differences in beta frequency correlate with the audio-visual fusion illusion

Presenting one flash with two beeps induces a perception of two flashes (audio-visual [AV] fission illusion), while presenting two flashes with one beep induces a perception of one flash (fusion illusion). Although previous studies showed a relationship between the frequency of the alpha rhythm (alpha cycle) and one's susceptibility to the fission illusion, the relationship between neural oscillations and the fusion illusion is unknown. Using electroencephalography, here I investigated the frequency of oscillatory signals in the pre-stimulus period and found a significant correlation between the beta rhythm and the fusion illusion; specifically, participants with a lower beta frequency showed a larger fusion illusion. These data indicate two separate time windows of AV integration in the human brain, one defined by the alpha cycle (fission) and another defined by the beta cycle (fusion).

Lower multisensory temporal acuity in individuals with high schizotypal traits: a web-based study

Natural events are often multisensory, requiring the brain to combine information from the same spatial location and timing, across different senses. The importance of temporal coincidence has led to the introduction of the temporal binding window (TBW) construct, defined as the time range within which multisensory inputs are highly likely to be perceptually bound into a single entity. Anomalies in TBWs have been linked to confused perceptual experiences and inaccurate filtering of sensory inputs coming from different environmental sources. Indeed, larger TBWs have been associated with disorders such as schizophrenia and autism and are also correlated to a higher level of subclinical traits of these conditions in the general population. Here, we tested the feasibility of using a web-based version of a classic audio-visual simultaneity judgment (SJ) task with simple flash-beep stimuli in order to measure multisensory temporal acuity and its relationship with schizotypal traits as measured in the general population. Results show that: (i) the response distribution obtained in the web-based SJ task was strongly similar to those reported by studies carried out in controlled laboratory settings, and (ii) lower multisensory temporal acuity was associated with higher schizotypal traits in the “cognitive-perceptual” domains. Our findings reveal the possibility of adequately using a web-based audio-visual SJ task outside a controlled laboratory setting, available to a more diverse and representative pool of participants. These results provide additional evidence for a close relationship between lower multisensory acuity and the expression of schizotypal traits in the general population.

The effect of movement speed on audiovisual temporal integration in streaming-bouncing illusion

Motion perception in real situations is often stimulated by multisensory information. Speed is an essential characteristic of moving objects; however, at present, it is not clear whether speed affects the process of audiovisual temporal integration in motion perception. Therefore, this study used a streaming-bouncing task (a bistable motion perception; SB task) combined with a simultaneous judgment task (SJ task) to explore the effect of speed on audiovisual temporal integration from implicit and explicit perspectives. The experiment had a within-subjects design, two speed conditions (fast/slow), eleven audiovisual conditions [stimulus onset asynchrony (SOA): 0 ms/ ± 60 ms/ ± 120 ms/ ± 180 ms/ ± 240 ms/ ± 300 ms], and a visual-only condition. A total of 30 subjects were recruited for the study. These participants completed the SB task and the SJ task successively. The results showed the following outcomes: (1) the optimal times needed to induce the "bouncing" illusion and maximum audiovisual bounce-inducing effect (ABE) magnitude were much earlier than that for the optimal time of audiovisual synchrony, (2) speed as a bottom-up factor could affect the proportion of "bouncing" perception in SB illusions but did not affect the ABE magnitude, (3) speed could also affect the ability of audiovisual temporal integration in motion perception, and the main manifestation was that the point of subjective simultaneity (PSS) in fast speed conditions was earlier than that of slow speed conditions in the SJ task and (4) the SB task and SJ task were not related. In conclusion, the time to complete the maximum audiovisual integration was different from the optimal time for synchrony perception; moreover, speed could affect audiovisual temporal integration in motion perception but only in explicit temporal tasks.

A Platform for Virtual Reality Task Design with Intracranial Electrodes

Research with human intracranial electrodes has traditionally been constrained by the limitations of the inpatient clinical setting. Immersive virtual reality (VR), however, can transcend setting and enable novel task design with precise control over visual and auditory stimuli. This control over visual and auditory feedback makes VR an exciting platform for new in-patient, human electrocorticography (ECOG) and stereo-electroencephalography (sEEG) research. The integration of intracranial electrode recording and stimulation with VR task dynamics required foundational systems engineering. In this work, we present a custom API that bridges Unity, the leading VR game development engine, and Synapse, the proprietary software of the Tucker Davis Technologies (TDT) neural recording and stimulation platform. To demonstrate the functionality and efficiency of our API, we developed a closed-loop brain-computer interface (BCI) task in which filtered neural signals controlled the movement of a virtual object and virtual object dynamics triggered neural stimulation. This closed-loop VR-BCI task confirmed the utility, safety, and efficacy of our API and its readiness for human task deployment.

Cross-modal perceptual enhancement of unisensory targets is uni-directional and does not affect temporal expectations

Temporal structures in the environment can shape temporal expectations (TE); and previous studies demonstrated that TEs interact with multisensory interplay (MSI) when multisensory stimuli are presented synchronously. Here, we tested whether other types of MSI - evoked by asynchronous yet temporally flanking irrelevant stimuli - result in similar performance patterns. To this end, we presented sequences of 12 stimuli (10 Hz) which consisted of auditory (A), visual (V) or alternating auditory-visual stimuli (e.g. A-V-A-V-…) with either auditory or visual targets (Exp. 1). Participants discriminated target frequencies (auditory pitch or visual spatial frequency) embedded in these sequences. To test effects of TE, the proportion of early and late temporal target positions was manipulated run-wise. Performance for unisensory targets was affected by temporally flanking distractors, with auditory temporal flankers selectively improving visual target perception (Exp. 1). However, no effect of temporal expectation was observed. Control experiments (Exp. 2-3) tested whether this lack of TE effect was due to the higher presentation frequency in Exp. 1 relative to previous experiments. Importantly, even at higher stimulation frequencies redundant multisensory targets (Exp. 2-3) reliably modulated TEs. Together, our results indicate that visual target detection was enhanced by MSI. However, this cross-modal enhancement - in contrast to the redundant target effect - was still insufficient to generate TEs. We posit that unisensory target representations were either instable or insufficient for the generation of TEs while less demanding MSI still occurred; highlighting the need for robust stimulus representations when generating temporal expectations.

Surmising synchrony of sound and sight: Factors explaining variance of audiovisual integration in hurdling, tap dancing and drumming

Auditory and visual percepts are integrated even when they are not perfectly temporally aligned with each other, especially when the visual signal precedes the auditory signal. This window of temporal integration for asynchronous audiovisual stimuli is relatively well examined in the case of speech, while other natural action-induced sounds have been widely neglected. Here, we studied the detection of audiovisual asynchrony in three different whole-body actions with natural action-induced sounds–hurdling, tap dancing and drumming. In Study 1, we examined whether audiovisual asynchrony detection, assessed by a simultaneity judgment task, differs as a function of sound production intentionality. Based on previous findings, we expected that auditory and visual signals should be integrated over a wider temporal window for actions creating sounds intentionally (tap dancing), compared to actions creating sounds incidentally (hurdling). While percentages of perceived synchrony differed in the expected way, we identified two further factors, namely high event density and low rhythmicity, to induce higher synchrony ratings as well. Therefore, we systematically varied event density and rhythmicity in Study 2, this time using drumming stimuli to exert full control over these variables, and the same simultaneity judgment tasks. Results suggest that high event density leads to a bias to integrate rather than segregate auditory and visual signals, even at relatively large asynchronies. Rhythmicity had a similar, albeit weaker effect, when event density was low. Our findings demonstrate that shorter asynchronies and visual-first asynchronies lead to higher synchrony ratings of whole-body action, pointing to clear parallels with audiovisual integration in speech perception. Overconfidence in the naturally expected, that is, synchrony of sound and sight, was stronger for intentional (vs. incidental) sound production and for movements with high (vs. low) rhythmicity, presumably because both encourage predictive processes. In contrast, high event density appears to increase synchronicity judgments simply because it makes the detection of audiovisual asynchrony more difficult. More studies using real-life audiovisual stimuli with varying event densities and rhythmicities are needed to fully uncover the general mechanisms of audiovisual integration.

Wiring of higher-order cortical areas: Spatiotemporal development of cortical hierarchy

A hierarchical development of cortical areas was suggested over a century ago, but the diversity and complexity of cortical hierarchy properties have so far prevented a formal demonstration. The aim of this review is to clarify the similarities and differences in the developmental processes underlying cortical development of primary and higher-order areas. We start by recapitulating the historical and recent advances underlying the biological principle of cortical hierarchy in adults. We then revisit the arguments for a hierarchical maturation of cortical areas, and further integrate the principles of cortical areas specification during embryonic and postnatal development. We highlight how the dramatic expansion in cortical size might have contributed to the increased number of association areas sustaining cognitive complexification in evolution. Finally, we summarize the recent observations of an alteration of cortical hierarchy in neuropsychiatric disorders and discuss their potential developmental origins.

Audiovisual interactions in speeded discrimination of a visual event

The integration of information from different senses is central to our perception of the external world. Audiovisual interactions have been particularly well studied in this context and various illusions have been developed to demonstrate strong influences of these interactions on the final percept. Using audiovisual paradigms, previous studies have shown that even task-irrelevant information provided by a secondary modality can change the detection and discrimination of a primary target. These modulations have been found to be significantly dependent on the relative timing between auditory and visual stimuli. Although these interactions in time have been commonly reported, we have still limited understanding of the relationship between the modulations of event-related potentials (ERPs) and final behavioral performance. Here, we aimed to shed light on this important issue by using a speeded discrimination paradigm combined with electroencephalogram (EEG). During the experimental sessions, the timing between an auditory click and a visual flash was varied over a wide range of stimulus onset asynchronies and observers were engaged in speeded discrimination of flash location. Behavioral reaction times were significantly changed by click timing. Furthermore, the modulations of evoked activities over medial parietal/parieto-occipital electrodes were associated with this effect. These modulations were within the 126-176 ms time range and more importantly, they were also correlated with the changes in reaction times. These results provide an important functional link between audiovisual interactions at early stages of sensory processing and reaction times. Together with previous research, they further suggest that early crossmodal interactions play a critical role in perceptual performance.

Development of the Mechanisms Underlying Audiovisual Speech Perception Benefit

The natural environments in which infants and children learn speech and language are noisy and multimodal. Adults rely on the multimodal nature of speech to compensate for noisy environments during speech communication. Multiple mechanisms underlie mature audiovisual benefit to speech perception, including reduced uncertainty as to when auditory speech will occur, use of correlations between the amplitude envelope of auditory and visual signals in fluent speech, and use of visual phonetic knowledge for lexical access. This paper reviews evidence regarding infants' and children's use of temporal and phonetic mechanisms in audiovisual speech perception benefit. The ability to use temporal cues for audiovisual speech perception benefit emerges in infancy. Although infants are sensitive to the correspondence between auditory and visual phonetic cues, the ability to use this correspondence for audiovisual benefit may not emerge until age four. A more cohesive account of the development of audiovisual speech perception may follow from a more thorough understanding of the development of sensitivity to and use of various temporal and phonetic cues.

Neural Correlates of Audiovisual Temporal Binding Window in Individuals With Schizotypal and Autistic Traits: Evidence From Resting-State Functional Connectivity

Temporal proximity is an important clue for multisensory integration. Previous evidence indicates that individuals with autism and schizophrenia are more likely to integrate multisensory inputs over a longer temporal binding window (TBW). However, whether such deficits in audiovisual temporal integration extend to subclinical populations with high schizotypal and autistic traits are unclear. Using audiovisual simultaneity judgment (SJ) tasks for nonspeech and speech stimuli, our results suggested that the width of the audiovisual TBW was not significantly correlated with self-reported schizotypal and autistic traits in a group of young adults. Functional magnetic resonance imaging (fMRI) resting-state activity was also acquired to explore the neural correlates underlying inter-individual variability of TBW width. Across the entire sample, stronger resting-state functional connectivity (rsFC) between the left superior temporal cortex and the left precuneus, and weaker rsFC between the left cerebellum and the right dorsal lateral prefrontal cortex were correlated with a narrower TBW for speech stimuli. Meanwhile, stronger rsFC between the left anterior superior temporal gyrus and the right inferior temporal gyrus was correlated with a wider audiovisual TBW for non-speech stimuli. The TBW-related rsFC was not affected by levels of subclinical traits. In conclusion, this study indicates that audiovisual temporal processing may not be affected by autistic and schizotypal traits and rsFC between brain regions responding to multisensory information and timing may account for the inter-individual difference in TBW width. LAY SUMMARY: Individuals with ASD and schizophrenia are more likely to perceive asynchronous auditory and visual events as occurring simultaneously even if they are well separated in time. We investigated whether similar difficulties in audiovisual temporal processing were present in subclinical populations with high autistic and schizotypal traits. We found that the ability to detect audiovisual asynchrony was not affected by different levels of autistic and schizotypal traits. We also found that connectivity of some brain regions engaging in multisensory and timing tasks might explain an individual's tendency to bind multisensory information within a wide or narrow time window. Autism Res 2021, 14: 668-680. © 2020 International Society for Autism Research and Wiley Periodicals LLC.