Multisensory Processes: A Balancing Act across the Lifespan

The impact of premature birth on auditory-visual processes in very preterm schoolchildren

Interactions between stimuli from different sensory modalities and their integration are central to daily life, contributing to improved perception. Being born prematurely and the subsequent hospitalization can have an impact not only on sensory processes, but also on the manner in which information from different senses is combined-i.e., multisensory processes. Very preterm (VPT) children (<32 weeks gestational age) present impaired multisensory processes in early childhood persisting at least through the age of five. However, it remains largely unknown whether and how these consequences persist into later childhood. Here, we evaluated the integrity of auditory-visual multisensory processes in VPT schoolchildren. VPT children (N = 28; aged 8-10 years) received a standardized cognitive assessment and performed a simple detection task at their routine follow-up appointment. The simple detection task involved pressing a button as quickly as possible upon presentation of an auditory, visual, or simultaneous audio-visual stimulus. Compared to full-term (FT) children (N = 23; aged 6-11 years), reaction times of VPT children were generally slower and more variable, regardless of sensory modality. Nonetheless, both groups exhibited multisensory facilitation on mean reaction times and inter-quartile ranges. There was no evidence that standardized cognitive or clinical measures correlated with multisensory gains of VPT children. However, while gains in FT children exceeded predictions based on probability summation and thus forcibly invoked integrative processes, this was not the case for VPT children. Our findings provide evidence of atypical multisensory profiles in VPT children persisting into school-age. These results could help in targeting supportive interventions for this vulnerable population.

Synchrony perception across senses: A systematic review of temporal binding window changes from infancy to adolescence in typical and atypical development

Sensory integration is increasingly acknowledged as being crucial for the development of cognitive and social abilities. However, its developmental trajectory is still little understood. This systematic review delves into the topic by investigating the literature about the developmental changes from infancy through adolescence of the Temporal Binding Window (TBW) - the epoch of time within which sensory inputs are perceived as simultaneous and therefore integrated. Following comprehensive searches across PubMed, Elsevier, and PsycInfo databases, only experimental, behavioral, English-language, peer-reviewed studies on multisensory temporal processing in 0-17-year-olds have been included. Non-behavioral, non-multisensory, and non-human studies have been excluded as those that did not directly focus on the TBW. The selection process was independently performed by two Authors. The 39 selected studies involved 2859 participants in total. Findings indicate a predisposition towards cross-modal asynchrony sensitivity and a composite, still unclear, developmental trajectory, with atypical development associated to increased asynchrony tolerance. These results highlight the need for consistent and thorough research into TBW development to inform potential interventions.

[Interpretation of a concept for functional movement disorders from the perspective of older patients]

A recently published concept considers a significant proportion of the occurrence and persistence of functional movement disorders (FMD) to be explained by increased/incorrect weighting of the expected movement (feedforward signal) in the presence of decreased/altered actual feedback of the movement. In the context of aging and age-associated diseases, there is an increased likelihood that these prerequisites will occur, also in combination. For example, the feedforward signal can be enhanced by accumulation of a wealth of experience but can for example become prone to error due to changes in attention and (fear of) falling. Conversely, the actual feedback is subject to age-related changes, such as reduction of sensory functions. This could explain why FMDs also occur in old age and offer treatment approaches for this so far poorly studied disorder. It follows that a specific focus on (the correction of) feedforward signals and strengthening as well as training of the actual feedback are potentially promising therapeutic approaches for older people with FMD.

Keeping up with ourselves: Multimodal processes underlying body ownership across the lifespan

The sense of a bodily self is thought to depend on adaptive weighting and integration of bodily afferents and prior beliefs. While the physical body changes in shape, size, and functionality across the lifespan, the sense of body ownership remains relatively stable. Yet, little is known about how multimodal integration underlying such sense of ownership is altered in ontogenetic periods of substantial physical changes. We aimed to study this link for the motor and the tactile domain in a mixed-realty paradigm where participants ranging from 7 to 80 years old saw their own body with temporally mismatching multimodal signals. Participants were either stroked on their hand or moved it, while they saw it in multiple trials with different visual delays. For each trial, they judged the visuo-motor/tactile synchrony and rated the sense of ownership for the seen hand. Visual dependence and proprioceptive acuity were additionally assessed. The results show that across the lifespan body ownership decreases with increasing temporal multisensory mismatch, both in the tactile and the motor domain. We found an increased sense of ownership with increasing age independent of delay and modality. Delay sensitivity during multisensory conflicts was not consistently related to age. No effects of age were found on visual dependence or proprioceptive accuracy. The results are at least partly in line with an enhanced weighting of top-down and a reduced weighting of bottom-up signals for the momentary sense of bodily self with increasing age.

Auditory change detection and visual selective attention: association between MMN and N2pc

While the auditory and visual systems each provide distinct information to our brain, they also work together to process and prioritize input to address ever-changing conditions. Previous studies highlighted the trade-off between auditory change detection and visual selective attention; however, the relationship between them is still unclear. Here, we recorded electroencephalography signals from 106 healthy adults in three experiments. Our findings revealed a positive correlation at the population level between the amplitudes of event-related potential indices associated with auditory change detection (mismatch negativity) and visual selective attention (posterior contralateral N2) when elicited in separate tasks. This correlation persisted even when participants performed a visual task while disregarding simultaneous auditory stimuli. Interestingly, as visual attention demand increased, participants whose posterior contralateral N2 amplitude increased the most exhibited the largest reduction in mismatch negativity, suggesting a within-subject trade-off between the two processes. Taken together, our results suggest an intimate relationship and potential shared mechanism between auditory change detection and visual selective attention. We liken this to a total capacity limit that varies between individuals, which could drive correlated individual differences in auditory change detection and visual selective attention, and also within-subject competition between the two, with task-based modulation of visual attention causing within-participant decrease in auditory change detection sensitivity.

Alpha-band sensory entrainment improves audiovisual temporal acuity

Visual and auditory stimuli are transmitted from the environment to sensory cortices with different timing, requiring the brain to encode when sensory inputs must be segregated or integrated into a single percept. The probability that different audiovisual (AV) stimuli are integrated into a single percept even when presented asynchronously is reflected in the construct of temporal binding window (TBW). There is a strong interest in testing whether it is possible to broaden or shrink TBW by using different neuromodulatory approaches that can speed up or slow down ongoing alpha oscillations, which have been repeatedly hypothesized to be an important determinant of the TBWs size. Here, we employed a web-based sensory entrainment protocol combined with a simultaneity judgment task using simple flash-beep stimuli. The aim was to test whether AV temporal acuity could be modulated trial by trial by synchronizing ongoing neural oscillations in the prestimulus period to a rhythmic sensory stream presented in the upper (∼12 Hz) or lower (∼8.5 Hz) alpha range. As a control, we implemented a nonrhythmic condition where only the first and the last entrainers were employed. Results show that upper alpha entrainment shrinks AV TBW and improves AV temporal acuity when compared with lower alpha and control conditions. Our findings represent a proof of concept of the efficacy of sensory entrainment to improve AV temporal acuity in a trial-by-trial manner, and they strengthen the idea that alpha oscillations may reflect the temporal unit of AV temporal binding.

Touch and look: The role of affective touch in promoting infants' attention towards complex visual scenes

In a complex social environment, stimuli from different sensory modalities need to be integrated to decode communicative meanings. From very early in life, infants have to combine a multitude of sensory features with social and affective attributes. Of all senses, touch constitutes a privileged channel to carry affective-motivational meanings and foster social connection. In the present study, we investigate whether sharing sensory stimulation that varies for its affective value differentially affects infants' attention towards visual stimuli. 6 to 11-month-old infants (N = 42) were familiarized with two characters respectively matched with tactile (affective or non-affective) and auditory stimulation; then repeatedly exposed to scenes where the two characters moved towards target objects. Our results showed a main effect of stimulation (sound vs. touch) on looking times during familiarization, with longer looking times when sound is provided. During scenes presentation, a main effect of the type of touch (affective vs. non affective) emerged, with longer looking times in infants that previously experienced affective touch, suggesting that this sensory experience may critically engage the self and modulate infant attention. Overall, these findings suggest that while sound acts as attention getter, affective touch supports sustained attention towards complex visual scenes beyond the stimulation period itself.

The emergence of the multisensory brain: From the womb to the first steps

The becoming of the human being is a multisensory process that starts in the womb. By integrating spontaneous neuronal activity with inputs from the external world, the developing brain learns to make sense of itself through multiple sensory experiences. Over the past ten years, advances in neuroimaging and electrophysiological techniques have allowed the exploration of the neural correlates of multisensory processing in the newborn and infant brain, thus adding an important piece of information to behavioral evidence of early sensitivity to multisensory events. Here, we review recent behavioral and neuroimaging findings to document the origins and early development of multisensory processing, particularly showing that the human brain appears naturally tuned to multisensory events at birth, which requires multisensory experience to fully mature. We conclude the review by highlighting the potential uses and benefits of multisensory interventions in promoting healthy development by discussing emerging studies in preterm infants.

The Development of Multisensory Integration at the Neuronal Level

Multisensory integration is a fundamental function of the brain. In the typical adult, multisensory neurons' response to paired multisensory (e.g., audiovisual) cues is significantly more robust than the corresponding best unisensory response in many brain regions. Synthesizing sensory signals from multiple modalities can speed up sensory processing and improve the salience of outside events or objects. Despite its significance, multisensory integration is testified to be not a neonatal feature of the brain. Neurons' ability to effectively combine multisensory information does not occur rapidly but develops gradually during early postnatal life (for cats, 4-12 weeks required). Multisensory experience is critical for this developing process. If animals were restricted from sensing normal visual scenes or sounds (deprived of the relevant multisensory experience), the development of the corresponding integrative ability could be blocked until the appropriate multisensory experience is obtained. This section summarizes the extant literature on the development of multisensory integration (mainly using cat superior colliculus as a model), sensory-deprivation-induced cross-modal plasticity, and how sensory experience (sensory exposure and perceptual learning) leads to the plastic change and modification of neural circuits in cortical and subcortical areas.

The characteristics of audiovisual temporal integration in streaming-bouncing bistable motion perception: considering both implicit and explicit processing perspectives

This study explored the behavioral and neural activity characteristics of audiovisual temporal integration in motion perception from both implicit and explicit perspectives. The streaming-bouncing bistable paradigm (SB task) was employed to investigate implicit temporal integration, while the corresponding simultaneity judgment task (SJ task) was used to examine explicit temporal integration. The behavioral results revealed a negative correlation between implicit and explicit temporal processing. In the ERP results of both tasks, three neural phases (PD100, ND180, and PD290) in the fronto-central region were identified as reflecting integration effects and the auditory-evoked multisensory N1 component may serve as a primary component responsible for cross-modal temporal processing. However, there were significant differences between the VA ERPs in the SB and SJ tasks and the influence of speed on implicit and explicit integration effects also varied. The aforementioned results, building upon the validation of previous temporal renormalization theory, suggest that implicit and explicit temporal integration operate under distinct processing modes within a shared neural network. This underscores the brain's flexibility and adaptability in cross-modal temporal processing.

Task-dependent spatial processing in the visual cortex

To solve spatial tasks, the human brain asks for support from the visual cortices. Nonetheless, representing spatial information is not fixed but depends on the reference frames in which the spatial inputs are involved. The present study investigates how the kind of spatial representations influences the recruitment of visual areas during multisensory spatial tasks. Our study tested participants in an electroencephalography experiment involving two audio-visual (AV) spatial tasks: a spatial bisection, in which participants estimated the relative position in space of an AV stimulus in relation to the position of two other stimuli, and a spatial localization, in which participants localized one AV stimulus in relation to themselves. Results revealed that spatial tasks specifically modulated the occipital event-related potentials (ERPs) after the onset of the stimuli. We observed a greater contralateral early occipital component (50-90 ms) when participants solved the spatial bisection, and a more robust later occipital response (110-160 ms) when they processed the spatial localization. This observation suggests that different spatial representations elicited by multisensory stimuli are sustained by separate neurophysiological mechanisms.

The Visual Advantage Effect in Comparing Uni-Modal and Cross-Modal Probabilistic Category Learning

People rely on multiple learning systems to complete weather prediction (WP) tasks with visual cues. However, how people perform in audio and audiovisual modalities remains elusive. The present research investigated how the cue modality influences performance in probabilistic category learning and conscious awareness about the category knowledge acquired. A modified weather prediction task was adopted, in which the cues included two dimensions from visual, auditory, or audiovisual modalities. The results of all three experiments revealed better performances in the visual modality relative to the audio and audiovisual modalities. Moreover, participants primarily acquired unconscious knowledge in the audio and audiovisual modalities, while conscious knowledge was acquired in the visual modality. Interestingly, factors such as the amount of training, the complexity of visual stimuli, and the number of objects to which the two cues belonged influenced the amount of conscious knowledge acquired but did not change the visual advantage effect. These findings suggest that individuals can learn probabilistic cues and category associations across different modalities, but a robust visual advantage persists. Specifically, visual associations can be learned more effectively, and are more likely to become conscious. The possible causes and implications of these effects are discussed.

Behavioral and neural measures of infant responsivity increase with maternal multisensory input in non-irritable infants

INTRODUCTION

Parents often use sensory stimulation during early-life interactions with infants. These interactions, including gazing, rocking, or singing, scaffold child development. Previous studies have examined infant neural processing during highly controlled sensory stimulus presentation paradigms.

OBJECTIVE

In this study, we investigated infant behavioral and neural responsiveness during a mother-child social interaction during which the mother provided infant stimulation with a progressive increase in the number of sensory modalities.

METHODS

We prospectively collected and analyzed video-coded behavioral interactions and electroencephalogram (EEG) frontal asymmetry (FAS) from infants (n = 60) at 2-4 months born at ≥ 34 weeks gestation. As the number of sensory modalities progressively increased during the interaction, infant behaviors of emotional connection in facial expressiveness, sensitivity to mother, and vocal communication increased significantly. Conversely, infant FAS for the entire cohort did not change significantly. However, when we accounted for infant irritability, both video-coded behaviors and EEG FAS markers of infant responsiveness increased across the interaction in the non-irritable infants. The non-irritable infants (49%) demonstrated positive FAS, indicating readiness to engage with, rather than to withdraw from, multisensory but not unisensory interactions with their mothers.

RESULTS

These results suggest that multisensory input from mothers is associated with greater infant neural approach state and highlight the importance of infant behavioral state during neural measures of infant responsiveness.

Development and experience-dependence of multisensory spatial processing

Multisensory spatial processes are fundamental for efficient interaction with the world. They include not only the integration of spatial cues across sensory modalities, but also the adjustment or recalibration of spatial representations to changing cue reliabilities, crossmodal correspondences, and causal structures. Yet how multisensory spatial functions emerge during ontogeny is poorly understood. New results suggest that temporal synchrony and enhanced multisensory associative learning capabilities first guide causal inference and initiate early coarse multisensory integration capabilities. These multisensory percepts are crucial for the alignment of spatial maps across sensory systems, and are used to derive more stable biases for adult crossmodal recalibration. The refinement of multisensory spatial integration with increasing age is further promoted by the inclusion of higher-order knowledge.

Multisensory integration in the mammalian brain: diversity and flexibility in health and disease

Multisensory integration (MSI) occurs in a variety of brain areas, spanning cortical and subcortical regions. In traditional studies on sensory processing, the sensory cortices have been considered for processing sensory information in a modality-specific manner. The sensory cortices, however, send the information to other cortical and subcortical areas, including the higher association cortices and the other sensory cortices, where the multiple modality inputs converge and integrate to generate a meaningful percept. This integration process is neither simple nor fixed because these brain areas interact with each other via complicated circuits, which can be modulated by numerous internal and external conditions. As a result, dynamic MSI makes multisensory decisions flexible and adaptive in behaving animals. Impairments in MSI occur in many psychiatric disorders, which may result in an altered perception of the multisensory stimuli and an abnormal reaction to them. This review discusses the diversity and flexibility of MSI in mammals, including humans, primates and rodents, as well as the brain areas involved. It further explains how such flexibility influences perceptual experiences in behaving animals in both health and disease. This article is part of the theme issue 'Decision and control processes in multisensory perception'.

Multisensory perception constrains the formation of object categories: a review of evidence from sensory-driven and predictive processes on categorical decisions

Although object categorization is a fundamental cognitive ability, it is also a complex process going beyond the perception and organization of sensory stimulation. Here we review existing evidence about how the human brain acquires and organizes multisensory inputs into object representations that may lead to conceptual knowledge in memory. We first focus on evidence for two processes on object perception, multisensory integration of redundant information (e.g. seeing and feeling a shape) and crossmodal, statistical learning of complementary information (e.g. the 'moo' sound of a cow and its visual shape). For both processes, the importance attributed to each sensory input in constructing a multisensory representation of an object depends on the working range of the specific sensory modality, the relative reliability or distinctiveness of the encoded information and top-down predictions. Moreover, apart from sensory-driven influences on perception, the acquisition of featural information across modalities can affect semantic memory and, in turn, influence category decisions. In sum, we argue that both multisensory processes independently constrain the formation of object categories across the lifespan, possibly through early and late integration mechanisms, respectively, to allow us to efficiently achieve the everyday, but remarkable, ability of recognizing objects. This article is part of the theme issue 'Decision and control processes in multisensory perception'.

Interactionally Embedded Gestalt Principles of Multimodal Human Communication

Natural human interaction requires us to produce and process many different signals, including speech, hand and head gestures, and facial expressions. These communicative signals, which occur in a variety of temporal relations with each other (e.g., parallel or temporally misaligned), must be rapidly processed as a coherent message by the receiver. In this contribution, we introduce the notion of interactionally embedded, affordance-driven gestalt perception as a framework that can explain how this rapid processing of multimodal signals is achieved as efficiently as it is. We discuss empirical evidence showing how basic principles of gestalt perception can explain some aspects of unimodal phenomena such as verbal language processing and visual scene perception but require additional features to explain multimodal human communication. We propose a framework in which high-level gestalt predictions are continuously updated by incoming sensory input, such as unfolding speech and visual signals. We outline the constituent processes that shape high-level gestalt perception and their role in perceiving relevance and prägnanz. Finally, we provide testable predictions that arise from this multimodal interactionally embedded gestalt-perception framework. This review and framework therefore provide a theoretically motivated account of how we may understand the highly complex, multimodal behaviors inherent in natural social interaction.

Examining the latent structure and correlates of sensory reactivity in autism: a multi-site integrative data analysis by the autism sensory research consortium

BACKGROUND

Differences in responding to sensory stimuli, including sensory hyperreactivity (HYPER), hyporeactivity (HYPO), and sensory seeking (SEEK) have been observed in autistic individuals across sensory modalities, but few studies have examined the structure of these "supra-modal" traits in the autistic population.

METHODS

Leveraging a combined sample of 3868 autistic youth drawn from 12 distinct data sources (ages 3-18 years and representing the full range of cognitive ability), the current study used modern psychometric and meta-analytic techniques to interrogate the latent structure and correlates of caregiver-reported HYPER, HYPO, and SEEK within and across sensory modalities. Bifactor statistical indices were used to both evaluate the strength of a "general response pattern" factor for each supra-modal construct and determine the added value of "modality-specific response pattern" scores (e.g., Visual HYPER). Bayesian random-effects integrative data analysis models were used to examine the clinical and demographic correlates of all interpretable HYPER, HYPO, and SEEK (sub)constructs.

RESULTS

All modality-specific HYPER subconstructs could be reliably and validly measured, whereas certain modality-specific HYPO and SEEK subconstructs were psychometrically inadequate when measured using existing items. Bifactor analyses supported the validity of a supra-modal HYPER construct (ωH = .800) but not a supra-modal HYPO construct (ωH = .653), and supra-modal SEEK models suggested a more limited version of the construct that excluded some sensory modalities (ωH = .800; 4/7 modalities). Modality-specific subscales demonstrated significant added value for all response patterns. Meta-analytic correlations varied by construct, although sensory features tended to correlate most with other domains of core autism features and co-occurring psychiatric symptoms (with general HYPER and speech HYPO demonstrating the largest numbers of practically significant correlations).

LIMITATIONS

Conclusions may not be generalizable beyond the specific pool of items used in the current study, which was limited to caregiver report of observable behaviors and excluded multisensory items that reflect many "real-world" sensory experiences.

CONCLUSION

Of the three sensory response patterns, only HYPER demonstrated sufficient evidence for valid interpretation at the supra-modal level, whereas supra-modal HYPO/SEEK constructs demonstrated substantial psychometric limitations. For clinicians and researchers seeking to characterize sensory reactivity in autism, modality-specific response pattern scores may represent viable alternatives that overcome many of these limitations.

Neuroscience: Merging multisensory memories

How animals form and retain memories across multiple sensory modalities and how multisensory learning can enhance memory is largely unknown. A recent study sheds light on the neural mechanism underlying multisensory memory convergence in the Drosophila melanogaster brain.

The parental umwelt: Effects of parenthood on sensory processing in rodents

An animal's umwelt, comprising its perception of the sensory environment, which is inherently subjective, can change across the lifespan in accordance with major life events. In mammals, the onset of motherhood, in particular, is associated with a neural and sensory plasticity that alters a mother's detection and use of sensory information such as infant-related sensory stimuli. Although the literature surrounding mammalian mothers is well established, very few studies have addressed the effects of parenthood on sensory plasticity in mammalian fathers. In this review, we summarize the major findings on the effects of parenthood on behavioural and neural responses to sensory stimuli from pups in rodent mothers, with a focus on the olfactory, auditory, and somatosensory systems, as well as multisensory integration. We also review the available literature on sensory plasticity in rodent fathers. Finally, we discuss the importance of sensory plasticity for effective parental care, hormonal modulation of plasticity, and an exploration of temporal, ecological, and life-history considerations of sensory plasticity associated with parenthood. The changes in processing and/or perception of sensory stimuli associated with the onset of parental care may have both transient and long-lasting effects on parental behaviour and cognition in both mothers and fathers; as such, several promising areas of study, such as on the molecular/genetic, neurochemical, and experiential underpinnings of parenthood-related sensory plasticity, as well as determinants of interspecific variation, remain potential avenues for further exploration.

Audiovisual multisensory integration in individuals with reading and language impairments: A systematic review and meta-analysis

Differences in sensory function have been documented for a number of neurodevelopmental conditions, including reading and language impairments. Prior studies have measured audiovisual multisensory integration (i.e., the ability to combine inputs from the auditory and visual modalities) in these populations. The present study sought to systematically review and quantitatively synthesize the extant literature on audiovisual multisensory integration in individuals with reading and language impairments. A comprehensive search strategy yielded 56 reports, of which 38 were used to extract 109 group difference and 68 correlational effect sizes. There was an overall difference between individuals with reading and language impairments and comparisons on audiovisual integration. There was a nonsignificant trend towards moderation according to sample type (i.e., reading versus language) and publication/small study bias for this model. Overall, there was a small but non-significant correlation between metrics of audiovisual integration and reading or language ability; this model was not moderated by sample or study characteristics, nor was there evidence of publication/small study bias. Limitations and future directions for primary and meta-analytic research are discussed.

Crossmodal interactions in human learning and memory

Most studies of memory and perceptual learning in humans have employed unisensory settings to simplify the study paradigm. However, in daily life we are often surrounded by complex and cluttered scenes made up of many objects and sources of sensory stimulation. Our experiences are, therefore, highly multisensory both when passively observing the world and when acting and navigating. We argue that human learning and memory systems are evolved to operate under these multisensory and dynamic conditions. The nervous system exploits the rich array of sensory inputs in this process, is sensitive to the relationship between the sensory inputs, and continuously updates sensory representations, and encodes memory traces based on the relationship between the senses. We review some recent findings that demonstrate a range of human learning and memory phenomena in which the interactions between visual and auditory modalities play an important role, and suggest possible neural mechanisms that can underlie some surprising recent findings. We outline open questions as well as directions of future research to unravel human perceptual learning and memory.

Susceptibility to the sound-induced flash illusion is associated with gait speed in a large sample of middle-aged and older adults

BACKGROUND

Multisensory integration is the ability to appropriately merge information from different senses for the purpose of perceiving and acting in the environment. During walking, information from multiple senses must be integrated appropriately to coordinate effective movements. We tested the association between a well characterised multisensory task, the Sound-Induced Flash Illusion (SIFI), and gait speed in 3255 participants from The Irish Longitudinal Study on Ageing. High susceptibility to this illusion at longer stimulus onset asynchronies characterises older adults, and has been associated with cognitive and functional impairments, therefore it should be associated with slower gait speed.

METHOD

Gait was measured under three conditions; usual pace, cognitive dual tasking, and maximal walking speed. A separate logistic mixed effects regression model was run for 1) gait at usual pace, 2) change in gait speed for the cognitive dual tasking relative to usual pace and 3) change in maximal walking speed relative to usual pace. In all cases a binary response indicating a correct/incorrect response to each SIFI trial was the dependent variable. The model controlled for covariates including age, sex, education, vision and hearing abilities, Body Mass Index, and cognitive function.

RESULTS

Slower gait was associated with more illusions, particularly at longer temporal intervals between the flash-beep pair and the second beep, indicating that those who integrated incongruent sensory inputs over longer intervals, also walked slower. The relative changes in gait speed for cognitive dual tasking and maximal walking speed were also significantly associated with SIFI at longer SOAs.

CONCLUSIONS

These findings support growing evidence that mobility, susceptibility to falling and balance control are associated with multisensory processing in ageing.

Transforming musculoskeletal anatomy learning with haptic surface painting

Anatomical body painting has traditionally been utilized to support learner engagement and understanding of surface anatomy. Learners apply two-dimensional representations of surface markings directly on to the skin, based on the identification of key landmarks. Esthetically satisfying representations of musculature and viscera can also be created. However, established body painting approaches do not typically address three-dimensional spatial anatomical concepts. Haptic Surface Painting (HSP) is a novel activity, distinct from traditional body painting, and aims to develop learner spatial awareness. The HSP process is underpinned by previous work describing how a Haptico-visual observation and drawing method can support spatial, holistic, and collaborative anatomy learning. In HSP, superficial and underlying musculoskeletal and vascular structures are located haptically by palpation. Transparent colors are then immediately applied to the skin using purposive and cross-contour drawing techniques to produce corresponding visual representations of learner observation and cognition. Undergraduate students at a United Kingdom medical school (n = 7) participated in remote HSP workshops and focus groups. A phenomenological study of learner perspectives identified four themes from semantic qualitative analysis of transcripts: Three-dimensional haptico-visual exploration relating to learner spatial awareness of their own anatomy; cognitive freedom and accessibility provided by a flexible and empowering learning process; altered perspectives of anatomical detail, relationships, and clinical relevance; and delivery and context, relating to curricular integration, session format, and educator guidance. This work expands the pedagogic repertoire of anatomical body painting and has implications for anatomy educators seeking to integrate innovative, engaging, and effective learning approaches for transforming student learning.

Brain Macro-Structural Alterations in Aging Rats: A Longitudinal Lifetime Approach

Aging is accompanied by macro-structural alterations in the brain that may relate to age-associated cognitive decline. Animal studies could allow us to study this relationship, but so far it remains unclear whether their structural aging patterns correspond to those in humans. Therefore, by applying magnetic resonance imaging (MRI) and deformation-based morphometry (DBM), we longitudinally screened the brains of male RccHan:WIST rats for structural changes across their average lifespan. By combining dedicated region of interest (ROI) and voxel-wise approaches, we observed an increase in their global brain volume that was superimposed by divergent local morphologic alterations, with the largest aging effects in early and middle life. We detected a modality-dependent vulnerability to shrinkage across the visual, auditory, and somato-sensory cortical areas, whereas the piriform cortex showed partial resistance. Furthermore, shrinkage emerged in the amygdala, subiculum, and flocculus as well as in frontal, parietal, and motor cortical areas. Strikingly, we noticed the preservation of ectorhinal, entorhinal, retrosplenial, and cingulate cortical regions, which all represent higher-order brain areas and extraordinarily grew with increasing age. We think that the findings of this study will further advance aging research and may contribute to the establishment of interventional approaches to preserve cognitive health in advanced age.

Neural dynamics driving audio-visual integration in autism

Audio-visual (AV) integration plays a crucial role in supporting social functions and communication in autism spectrum disorder (ASD). However, behavioral findings remain mixed and, importantly, little is known about the underlying neurophysiological bases. Studies in neurotypical adults indicate that oscillatory brain activity in different frequencies subserves AV integration, pointing to a central role of (i) individual alpha frequency (IAF), which would determine the width of the cross-modal binding window; (ii) pre-/peri-stimulus theta oscillations, which would reflect the expectation of AV co-occurrence; (iii) post-stimulus oscillatory phase reset, which would temporally align the different unisensory signals. Here, we investigate the neural correlates of AV integration in children with ASD and typically developing (TD) peers, measuring electroencephalography during resting state and in an AV integration paradigm. As for neurotypical adults, AV integration dynamics in TD children could be predicted by the IAF measured at rest and by a modulation of anticipatory theta oscillations at single-trial level. Conversely, in ASD participants, AV integration/segregation was driven exclusively by the neural processing of the auditory stimulus and the consequent auditory-induced phase reset in visual regions, suggesting that a disproportionate elaboration of the auditory input could be the main factor characterizing atypical AV integration in autism.

Computational modeling of human multisensory spatial representation by a neural architecture

Our brain constantly combines sensory information in unitary percept to build coherent representations of the environment. Even though this process could appear smooth, integrating sensory inputs from various sensory modalities must overcome several computational issues, such as recoding and statistical inferences problems. Following these assumptions, we developed a neural architecture replicating humans' ability to use audiovisual spatial representations. We considered the well-known ventriloquist illusion as a benchmark to evaluate its phenomenological plausibility. Our model closely replicated human perceptual behavior, proving a truthful approximation of the brain's ability to develop audiovisual spatial representations. Considering its ability to model audiovisual performance in a spatial localization task, we release our model in conjunction with the dataset we recorded for its validation. We believe it will be a powerful tool to model and better understand multisensory integration processes in experimental and rehabilitation environments.

Cognition Assessment Technologies on Deaf People

In recent years there has been a growing interest in research about the different ways of processing and consolidating cognition in deaf people. It is known that hearing loss can lead to differences in some executive functions like control inhibitory or working memory. This literature review describes executive functions in deaf people and how they could be evaluated through technological devices complementing traditional assessments, like neuropsychological batteries. We identified biometric devices, digital and physical interfaces, and software from the literature, whose goal is to design or adapt technology to assess some cognition domains in several ways. The results of the review suggest the need to understand the cognitive phenomenon that significantly impacts the context of deaf people; moreover, it becomes relevant as a line of research in the Cognitive Science of Hearing. Using technologies to measure them and gain a better understanding of cognition in deaf people may provide possibilities for designing or adapting targeted educational or therapeutic strategies.

The Impact of Singing on Visual and Multisensory Speech Perception in Children on the Autism Spectrum

Autistic children show reduced multisensory integration of audiovisual speech stimuli in response to the McGurk illusion. Previously, it has been shown that adults can integrate sung McGurk tokens. These sung speech tokens offer more salient visual and auditory cues, in comparison to the spoken tokens, which may increase the identification and integration of visual speech cues in autistic children. Forty participants (20 autism, 20 non-autistic peers) aged 7-14 completed the study. Participants were presented with speech tokens in four modalities: auditory-only, visual-only, congruent audiovisual, and incongruent audiovisual (i.e., McGurk; auditory 'ba' and visual 'ga'). Tokens were also presented in two formats: spoken and sung. Participants indicated what they perceived via a four-button response box (i.e., 'ba', 'ga', 'da', or 'tha'). Accuracies and perception of the McGurk illusion were calculated for each modality and format. Analysis of visual-only identification indicated a significant main effect of format, whereby participants were more accurate in sung versus spoken trials, but no significant main effect of group or interaction effect. Analysis of the McGurk trials indicated no significant main effect of format or group and no significant interaction effect. Sung speech tokens improved identification of visual speech cues, but did not boost the integration of visual cues with heard speech across groups. Additional work is needed to determine what properties of spoken speech contributed to the observed improvement in visual accuracy and to evaluate whether more prolonged exposure to sung speech may yield effects on multisensory integration.

The development of audio-visual temporal precision precedes its rapid recalibration

Through development, multisensory systems reach a balance between stability and flexibility: the systems integrate optimally cross-modal signals from the same events, while remaining adaptive to environmental changes. Is continuous intersensory recalibration required to shape optimal integration mechanisms, or does multisensory integration develop prior to recalibration? Here, we examined the development of multisensory integration and rapid recalibration in the temporal domain by re-analyzing published datasets for audio-visual, audio-tactile, and visual-tactile combinations. Results showed that children reach an adult level of precision in audio-visual simultaneity perception and show the first sign of rapid recalibration at 9 years of age. In contrast, there was very weak rapid recalibration for other cross-modal combinations at all ages, even when adult levels of temporal precision had developed. Thus, the development of audio-visual rapid recalibration appears to require the maturation of temporal precision. It may serve to accommodate distance-dependent travel time differences between light and sound.

On the Application of Developmental Cognitive Neuroscience in Educational Environments

The paper overviews components of neurologic processing efficiencies to develop innovative methodologies and thinking to school-based applications and changes in educational leadership based on sound findings in the cognitive neurosciences applied to schools and learners. Systems science can allow us to better manage classroom-based learning and instruction on the basis of relatively easily evaluated efficiencies or inefficiencies and optimization instead of simply examining achievement. "Medicalizing" the learning process with concepts such as "learning disability" or employing grading methods such as pass-fail does little to aid in understanding the processes that learners employ to acquire, integrate, remember, and apply information learned. The paper endeavors to overview and provided reference to tools that can be employed that allow a better focus on nervous system-based strategic approaches to classroom learning.

Somatosensory processing in deaf and deafblind individuals: How does the brain adapt as a function of sensory and linguistic experience? A critical review

How do deaf and deafblind individuals process touch? This question offers a unique model to understand the prospects and constraints of neural plasticity. Our brain constantly receives and processes signals from the environment and combines them into the most reliable information content. The nervous system adapts its functional and structural organization according to the input, and perceptual processing develops as a function of individual experience. However, there are still many unresolved questions regarding the deciding factors for these changes in deaf and deafblind individuals, and so far, findings are not consistent. To date, most studies have not taken the sensory and linguistic experiences of the included participants into account. As a result, the impact of sensory deprivation vs. language experience on somatosensory processing remains inconclusive. Even less is known about the impact of deafblindness on brain development. The resulting neural adaptations could be even more substantial, but no clear patterns have yet been identified. How do deafblind individuals process sensory input? Studies on deafblindness have mostly focused on single cases or groups of late-blind individuals. Importantly, the language backgrounds of deafblind communities are highly variable and include the usage of tactile languages. So far, this kind of linguistic experience and its consequences have not been considered in studies on basic perceptual functions. Here, we will provide a critical review of the literature, aiming at identifying determinants for neuroplasticity and gaps in our current knowledge of somatosensory processing in deaf and deafblind individuals.

Thalamocortical contribution to flexible learning in neural systems

Animal brains evolved to optimize behavior in dynamic environments, flexibly selecting actions that maximize future rewards in different contexts. A large body of experimental work indicates that such optimization changes the wiring of neural circuits, appropriately mapping environmental input onto behavioral outputs. A major unsolved scientific question is how optimal wiring adjustments, which must target the connections responsible for rewards, can be accomplished when the relation between sensory inputs, action taken, and environmental context with rewards is ambiguous. The credit assignment problem can be categorized into context-independent structural credit assignment and context-dependent continual learning. In this perspective, we survey prior approaches to these two problems and advance the notion that the brain's specialized neural architectures provide efficient solutions. Within this framework, the thalamus with its cortical and basal ganglia interactions serves as a systems-level solution to credit assignment. Specifically, we propose that thalamocortical interaction is the locus of meta-learning where the thalamus provides cortical control functions that parametrize the cortical activity association space. By selecting among these control functions, the basal ganglia hierarchically guide thalamocortical plasticity across two timescales to enable meta-learning. The faster timescale establishes contextual associations to enable behavioral flexibility, while the slower one enables generalization to new contexts.

The different brain areas occupied for integrating information of hierarchical linguistic units: a study based on EEG and TMS

Human language units are hierarchical, and reading acquisition involves integrating multisensory information (typically from auditory and visual modalities) to access meaning. However, it is unclear how the brain processes and integrates language information at different linguistic units (words, phrases, and sentences) provided simultaneously in auditory and visual modalities. To address the issue, we presented participants with sequences of short Chinese sentences through auditory, visual, or combined audio-visual modalities while electroencephalographic responses were recorded. With a frequency tagging approach, we analyzed the neural representations of basic linguistic units (i.e. characters/monosyllabic words) and higher-level linguistic structures (i.e. phrases and sentences) across the 3 modalities separately. We found that audio-visual integration occurs in all linguistic units, and the brain areas involved in the integration varied across different linguistic levels. In particular, the integration of sentences activated the local left prefrontal area. Therefore, we used continuous theta-burst stimulation to verify that the left prefrontal cortex plays a vital role in the audio-visual integration of sentence information. Our findings suggest the advantage of bimodal language comprehension at hierarchical stages in language-related information processing and provide evidence for the causal role of the left prefrontal regions in processing information of audio-visual sentences.

Multisensory representations of space and time in sensory cortices

Clear evidence demonstrated a supramodal organization of sensory cortices with multisensory processing occurring even at early stages of information encoding. Within this context, early recruitment of sensory areas is necessary for the development of fine domain-specific (i.e., spatial or temporal) skills regardless of the sensory modality involved, with auditory areas playing a crucial role in temporal processing and visual areas in spatial processing. Given the domain-specificity and the multisensory nature of sensory areas, in this study, we hypothesized that preferential domains of representation (i.e., space and time) of visual and auditory cortices are also evident in the early processing of multisensory information. Thus, we measured the event-related potential (ERP) responses of 16 participants while performing multisensory spatial and temporal bisection tasks. Audiovisual stimuli occurred at three different spatial positions and time lags and participants had to evaluate whether the second stimulus was spatially (spatial bisection task) or temporally (temporal bisection task) farther from the first or third audiovisual stimulus. As predicted, the second audiovisual stimulus of both spatial and temporal bisection tasks elicited an early ERP response (time window 50-90 ms) in visual and auditory regions. However, this early ERP component was more substantial in the occipital areas during the spatial bisection task, and in the temporal regions during the temporal bisection task. Overall, these results confirmed the domain specificity of visual and auditory cortices and revealed that this aspect selectively modulates also the cortical activity in response to multisensory stimuli.

Infant selective attention to native and non-native audiovisual speech

The current study utilized eye-tracking to investigate the effects of intersensory redundancy and language on infant visual attention and detection of a change in prosody in audiovisual speech. Twelve-month-old monolingual English-learning infants viewed either synchronous (redundant) or asynchronous (non-redundant) presentations of a woman speaking in native or non-native speech. Halfway through each trial, the speaker changed prosody from infant-directed speech (IDS) to adult-directed speech (ADS) or vice versa. Infants focused more on the mouth of the speaker on IDS trials compared to ADS trials regardless of language or intersensory redundancy. Additionally, infants demonstrated greater detection of prosody changes from IDS speech to ADS speech in native speech. Planned comparisons indicated that infants detected prosody changes across a broader range of conditions during redundant stimulus presentations. These findings shed light on the influence of language and prosody on infant attention and highlight the complexity of audiovisual speech processing in infancy.

The multisensory cocktail party problem in children: Synchrony-based segregation of multiple talking faces improves in early childhood

Extraction of meaningful information from multiple talkers relies on perceptual segregation. The temporal synchrony statistics inherent in everyday audiovisual (AV) speech offer a powerful basis for perceptual segregation. We investigated the developmental emergence of synchrony-based perceptual segregation of multiple talkers in 3-7-year-old children. Children either saw four identical or four different faces articulating temporally jittered versions of the same utterance and heard the audible version of the same utterance either synchronized with one of the talkers or desynchronized with all of them. Eye tracking revealed that selective attention to the temporally synchronized talking face increased while attention to the desynchronized faces decreased with age and that attention to the talkers' mouth primarily drove responsiveness. These findings demonstrate that the temporal synchrony statistics inherent in fluent AV speech assume an increasingly greater role in perceptual segregation of the multisensory clutter created by multiple talking faces in early childhood.

The relationship between multisensory associative learning and multisensory integration

Integrating sensory information from multiple modalities leads to more precise and efficient perception and behaviour. The process of determining which sensory information should be perceptually bound is reliant on both low-level stimulus features, as well as multisensory associations learned throughout development based on the statistics of our environment. Here, we explored the relationship between multisensory associative learning and multisensory integration using encephalography (EEG) and behavioural measures. Sixty-one participants completed a three-phase study. First, participants were exposed to novel audiovisual shape-tone pairings with frequent and infrequent stimulus pairings and complete a target detection task. EEG recordings of the mismatch negativity (MMN) and P3 were calculated as neural indices of multisensory associative learning. Next, the same learned stimulus pairs were presented in audiovisual as well as unisensory auditory and visual modalities while both early (<120 ms) and late neural indices of multisensory integration were recorded. Finally, participants completed an analogous behavioural speeded-response task, with behavioural indices of multisensory gain calculated using the Race Model. Significant relationships were found in fronto-central and occipital areas between neural measures of associative learning and both early and late indices of multisensory integration in frontal and centro-parietal areas, respectively. Participants who showed stronger indices of associative learning also exhibited stronger indices of multisensory integration of the stimuli they learned to associate. Furthermore, a significant relationship was found between neural index of early multisensory integration and behavioural indices of multisensory gain. These results provide insight into the neural underpinnings of how higher-order processes such as associative learning guide multisensory integration.

Joint attention in infants at high familial risk for autism spectrum disorder and the association with thalamic and hippocampal macrostructure

Autism spectrum disorder (ASD) is a heritable neurodevelopmental disorder. Infants diagnosed with ASD can show impairments in spontaneous gaze-following and will seldom engage in joint attention (JA). The ability to initiate JA (IJA) can be more significantly impaired than the ability to respond to JA (RJA). In a longitudinal study, 101 infants who had a familial risk for ASD were enrolled (62% males). Participants completed magnetic resonance imaging scans at 4 or 6 months of age. Subcortical volumes (thalamus, hippocampus, amygdala, basal ganglia, ventral diencephalon, and cerebellum) were automatically extracted. Early gaze and JA behaviors were assessed with standardized measures. The majority of infants were IJA nonresponders (n = 93, 92%), and over half were RJA nonresponders (n = 50, 52%). In the nonresponder groups, models testing the association of subcortical volumes with later ASD diagnosis accounted for age, sex, and cerebral volumes. In the nonresponder IJA group, using regression method, the left hippocampus (B = −0.009, aOR = 0.991, P = 0.025), the right thalamus (B = −0.016, aOR = 0.984, P = 0.026), as well as the left thalamus (B = 0.015, aOR = 1.015, P = 0.019), predicted later ASD diagnosis. Alterations in thalamic and hippocampal macrostructure in at-risk infants who do not engage in IJA may reflect an enhanced vulnerability and may be the key predictors of later ASD development.

Digital haptics improve speed of visual search performance in a dual-task setting

Dashboard-mounted touchscreen tablets are now common in vehicles. Screen/phone use in cars likely shifts drivers’ attention away from the road and contributes to risk of accidents. Nevertheless, vision is subject to multisensory influences from other senses. Haptics may help maintain or even increase visual attention to the road, while still allowing for reliable dashboard control. Here, we provide a proof-of-concept for the effectiveness of digital haptic technologies (hereafter digital haptics), which use ultrasonic vibrations on a tablet screen to render haptic perceptions. Healthy human participants (N = 25) completed a divided-attention paradigm. The primary task was a centrally-presented visual conjunction search task, and the secondary task entailed control of laterally-presented sliders on the tablet. Sliders were presented visually, haptically, or visuo-haptically and were vertical, horizontal or circular. We reasoned that the primary task would be performed best when the secondary task was haptic-only. Reaction times (RTs) on the visual search task were fastest when the tablet task was haptic-only. This was not due to a speed-accuracy trade-off; there was no evidence for modulation of VST accuracy according to modality of the tablet task. These results provide the first quantitative support for introducing digital haptics into vehicle and similar contexts.

The Components of Interpersonal Synchrony in the Typical Population and in Autism: A Conceptual Analysis

Interpersonal synchrony – the tendency for social partners to temporally co-ordinate their behaviour when interacting – is a ubiquitous feature of social interactions. Synchronous interactions play a key role in development, and promote social bonding and a range of pro-social behavioural outcomes across the lifespan. The process of achieving and maintaining interpersonal synchrony is highly complex, with inputs required from across perceptual, temporal, motor, and socio-cognitive domains. In this conceptual analysis, we synthesise evidence from across these domains to establish the key components underpinning successful non-verbal interpersonal synchrony, how such processes interact, and factors that may moderate their operation. We also consider emerging evidence that interpersonal synchrony is reduced in autistic populations. We use our account of the components contributing to interpersonal synchrony in the typical population to identify potential points of divergence in interpersonal synchrony in autism. The relationship between interpersonal synchrony and broader aspects of social communication in autism are also considered, together with implications for future research.

Audiovisual synchrony detection for fluent speech in early childhood: An eye-tracking study

During childhood, the ability to detect audiovisual synchrony gradually sharpens for simple stimuli such as flashbeeps and single syllables. However, little is known about how children perceive synchrony for natural and continuous speech. This study investigated young children's gaze patterns while they were watching movies of two identical speakers telling stories side by side. Only one speaker's lip movements matched the voices and the other one either led or lagged behind the soundtrack by 600 ms. Children aged 3-6 years (n = 94, 52.13% males) showed an overall preference for the synchronous speaker, with no age-related changes in synchrony-detection sensitivity as indicated by similar gaze patterns across ages. However, viewing time to the synchronous speech was significantly longer in the auditory-leading (AL) condition compared with that in the visual-leading (VL) condition, suggesting asymmetric sensitivities for AL versus VL asynchrony have already been established in early childhood. When further examining gaze patterns on dynamic faces, we found that more attention focused on the mouth region was an adaptive strategy to read visual speech signals and thus associated with increased viewing time of the synchronous videos. Attention to detail, one dimension of autistic traits featured by local processing, has been found to be correlated with worse performances in speech synchrony processing. These findings extended previous research by showing the development of speech synchrony perception in young children, and may have implications for clinical populations (e.g., autism) with impaired multisensory integration.

Lower multisensory temporal acuity in individuals with high schizotypal traits: a web-based study

Natural events are often multisensory, requiring the brain to combine information from the same spatial location and timing, across different senses. The importance of temporal coincidence has led to the introduction of the temporal binding window (TBW) construct, defined as the time range within which multisensory inputs are highly likely to be perceptually bound into a single entity. Anomalies in TBWs have been linked to confused perceptual experiences and inaccurate filtering of sensory inputs coming from different environmental sources. Indeed, larger TBWs have been associated with disorders such as schizophrenia and autism and are also correlated to a higher level of subclinical traits of these conditions in the general population. Here, we tested the feasibility of using a web-based version of a classic audio-visual simultaneity judgment (SJ) task with simple flash-beep stimuli in order to measure multisensory temporal acuity and its relationship with schizotypal traits as measured in the general population. Results show that: (i) the response distribution obtained in the web-based SJ task was strongly similar to those reported by studies carried out in controlled laboratory settings, and (ii) lower multisensory temporal acuity was associated with higher schizotypal traits in the “cognitive-perceptual” domains. Our findings reveal the possibility of adequately using a web-based audio-visual SJ task outside a controlled laboratory setting, available to a more diverse and representative pool of participants. These results provide additional evidence for a close relationship between lower multisensory acuity and the expression of schizotypal traits in the general population.

Interpersonal Affective Touch in a Virtual World: Feeling the Social Presence of Others to Overcome Loneliness

Humans are by nature social beings tuned to communicate and interact from the very beginning of their lives. The sense of touch represents the most direct and intimate channel of communication and a powerful means of connection between the self and the others. In our digital age, the development and diffusion of internet-based technologies and virtual environments offer new opportunities of communication overcoming physical distance. It however, happens that social interactions are often mediated, and the tactile aspects of communication are overlooked, thus diminishing the feeling of social presence, which may contribute to an increased sense of social disconnection and loneliness. The current manuscript aims to review the extant literature about the socio-affective dimension of touch and current advancements in interactive virtual environments in order to provide a new perspective on multisensory virtual communication. Specifically, we suggest that interpersonal affective touch might critically impact virtual social exchanges, promoting a sense of co-presence and social connection between individuals, possibly overcoming feelings of sensory loneliness. This topic of investigation will be of crucial relevance from a theoretical perspective aiming to understand how we integrate multisensory signals in processing and making sense of interpersonal exchanges, this is important in both typical and atypical populations. Moreover, it will pave the way to promising applications by exploring the possibility to use technical innovations to communicate more interactively in the case of people who suffer from social isolation and disconnection from others.

Multisensory integration approach, cognitive domains, meaningful learning: reflections for undergraduate nursing education

ABSTRACT Teaching with a multisensory approach helps students link new information to prior knowledge and understand relationships between concepts. This study aimed to reflect on convergences between the Multisensory Integration Approach Model with the Learning Assimilation Theory and Meaningful Retention with Bloom’s Cognitive Process Domain, and to propose a taxonomic table of lesson planning for teaching Acute Coronary Syndrome, considering the confluence of these references. The three frameworks consider the importance of students’ prior knowledge, the process of abstraction and generalization of knowledge, and the relationship between working and long-term memory. By observing such convergences and the taxonomic table produced, it is observed that teaching topics of interest to nursing undergraduate students, adopting the Multisensory Integration Approach Model as a taxonomic table component (pre-organizing or recall activities to arouse different sensory perceptions aligned with instructional objectives and forms of assessment), in the light of the Learning Assimilation Theory and Meaningful Retention, has the potential to favor the reception and processing of instructional content.

Towards understanding how we pay attention in naturalistic visual search settings

Research on attentional control has largely focused on single senses and the importance of behavioural goals in controlling attention. However, everyday situations are multisensory and contain regularities, both likely influencing attention. We investigated how visual attentional capture is simultaneously impacted by top-down goals, the multisensory nature of stimuli, and the contextual factors of stimuli's semantic relationship and temporal predictability. Participants performed a multisensory version of the Folk et al. (1992) spatial cueing paradigm, searching for a target of a predefined colour (e.g. a red bar) within an array preceded by a distractor. We manipulated: 1) stimuli's goal-relevance via distractor's colour (matching vs. mismatching the target), 2) stimuli's multisensory nature (colour distractors appearing alone vs. with tones), 3) the relationship between the distractor sound and colour (arbitrary vs. semantically congruent) and 4) the temporal predictability of distractor onset. Reaction-time spatial cueing served as a behavioural measure of attentional selection. We also recorded 129-channel event-related potentials (ERPs), analysing the distractor-elicited N2pc component both canonically and using a multivariate electrical neuroimaging framework. Behaviourally, arbitrary target-matching distractors captured attention more strongly than semantically congruent ones, with no evidence for context modulating multisensory enhancements of capture. Notably, electrical neuroimaging of surface-level EEG analyses revealed context-based influences on attention to both visual and multisensory distractors, in how strongly they activated the brain and type of activated brain networks. For both processes, the context-driven brain response modulations occurred long before the N2pc time-window, with topographic (network-based) modulations at ∼30 ms, followed by strength-based modulations at ∼100 ms post-distractor onset. Our results reveal that both stimulus meaning and predictability modulate attentional selection, and they interact while doing so. Meaning, in addition to temporal predictability, is thus a second source of contextual information facilitating goal-directed behaviour. More broadly, in everyday situations, attention is controlled by an interplay between one's goals, stimuli's perceptual salience, meaning and predictability. Our study calls for a revision of attentional control theories to account for the role of contextual and multisensory control.