Time-dependent changes in learning audiovisual associations: a single-trial fMRI study

A novel task and methods to evaluate inter-individual variation in audio-visual associative learning

Learning audio-visual associations is foundational to a number of real-world skills, such as reading acquisition or social communication. Characterizing individual differences in such learning has therefore been of interest to researchers in the field. Here, we present a novel audio-visual associative learning task designed to efficiently capture inter-individual differences in learning, with the added feature of using non-linguistic stimuli, so as to unconfound language and reading proficiency of the learner from their more domain-general learning capability. By fitting trial-by-trial performance in our novel learning task using simple-to-use statistical tools, we demonstrate the expected inter-individual variability in learning rate as well as high precision in its estimation. We further demonstrate that such measured learning rate is linked to working memory performance in Italian-speaking (N = 58) and French-speaking (N = 51) adults. Finally, we investigate the extent to which learning rate in our task, which measures cross-modal audio-visual associations while mitigating familiarity confounds, predicts reading ability across participants with different linguistic backgrounds. The present work thus introduces a novel non-linguistic audio-visual associative learning task that can be used across languages. In doing so, it brings a new tool to researchers in the various domains that rely on multi-sensory integration from reading to social cognition or socio-emotional learning.

Morphometry Difference of the Hippocampal Formation Between Blind and Sighted Individuals

The detailed morphometry alterations of the human hippocampal formation (HF) for blind individuals are still understudied. 50 subjects were recruited from Yantai Affiliated Hospital of Binzhou Medical University, including 16 congenital blindness, 14 late blindness, and 20 sighted controls. Volume and shape analysis were conducted between the blind (congenital or late) and sighted groups to observe the (sub)regional alterations of the HF. No significant difference of the hippocampal volume was observed between the blind and sighted subjects. Rightward asymmetry of the hippocampal volume was found for both congenital and late blind individuals, while no significant hemispheric difference was observed for the sighted controls. Shape analysis showed that the superior and inferior parts of both the hippocampal head and tail expanded, while the medial and lateral parts constrained for the blind individuals as compared to the sighted controls. The morphometry alterations for the congenital blind and late blind individuals are nearly the same. Significant expansion of the superior part of the hippocampal tail for both congenital and late blind groups were observed for the left hippocampi after FDR correction. Current results suggest that the cross-model plastic may occur in both hemispheres of the HF to improve the navigation ability without the stimuli of visual cues, and the alteration is more prominent for the left hemisphere.

Meta-Analyses Support a Taxonomic Model for Representations of Different Categories of Audio-Visual Interaction Events in the Human Brain

Our ability to perceive meaningful action events involving objects, people, and other animate agents is characterized in part by an interplay of visual and auditory sensory processing and their cross-modal interactions. However, this multisensory ability can be altered or dysfunctional in some hearing and sighted individuals, and in some clinical populations. The present meta-analysis sought to test current hypotheses regarding neurobiological architectures that may mediate audio-visual multisensory processing. Reported coordinates from 82 neuroimaging studies (137 experiments) that revealed some form of audio-visual interaction in discrete brain regions were compiled, converted to a common coordinate space, and then organized along specific categorical dimensions to generate activation likelihood estimate (ALE) brain maps and various contrasts of those derived maps. The results revealed brain regions (cortical "hubs") preferentially involved in multisensory processing along different stimulus category dimensions, including 1) living versus nonliving audio-visual events, 2) audio-visual events involving vocalizations versus actions by living sources, 3) emotionally valent events, and 4) dynamic-visual versus static-visual audio-visual stimuli. These meta-analysis results are discussed in the context of neurocomputational theories of semantic knowledge representations and perception, and the brain volumes of interest are available for download to facilitate data interpretation for future neuroimaging studies.

Neural Correlates of Audiovisual Temporal Binding Window in Individuals With Schizotypal and Autistic Traits: Evidence From Resting-State Functional Connectivity

Temporal proximity is an important clue for multisensory integration. Previous evidence indicates that individuals with autism and schizophrenia are more likely to integrate multisensory inputs over a longer temporal binding window (TBW). However, whether such deficits in audiovisual temporal integration extend to subclinical populations with high schizotypal and autistic traits are unclear. Using audiovisual simultaneity judgment (SJ) tasks for nonspeech and speech stimuli, our results suggested that the width of the audiovisual TBW was not significantly correlated with self-reported schizotypal and autistic traits in a group of young adults. Functional magnetic resonance imaging (fMRI) resting-state activity was also acquired to explore the neural correlates underlying inter-individual variability of TBW width. Across the entire sample, stronger resting-state functional connectivity (rsFC) between the left superior temporal cortex and the left precuneus, and weaker rsFC between the left cerebellum and the right dorsal lateral prefrontal cortex were correlated with a narrower TBW for speech stimuli. Meanwhile, stronger rsFC between the left anterior superior temporal gyrus and the right inferior temporal gyrus was correlated with a wider audiovisual TBW for non-speech stimuli. The TBW-related rsFC was not affected by levels of subclinical traits. In conclusion, this study indicates that audiovisual temporal processing may not be affected by autistic and schizotypal traits and rsFC between brain regions responding to multisensory information and timing may account for the inter-individual difference in TBW width. LAY SUMMARY: Individuals with ASD and schizophrenia are more likely to perceive asynchronous auditory and visual events as occurring simultaneously even if they are well separated in time. We investigated whether similar difficulties in audiovisual temporal processing were present in subclinical populations with high autistic and schizotypal traits. We found that the ability to detect audiovisual asynchrony was not affected by different levels of autistic and schizotypal traits. We also found that connectivity of some brain regions engaging in multisensory and timing tasks might explain an individual's tendency to bind multisensory information within a wide or narrow time window. Autism Res 2021, 14: 668-680. © 2020 International Society for Autism Research and Wiley Periodicals LLC.

Children's neural activity during number line estimations assessed by functional near-infrared spectroscopy (fNIRS)

Number line estimation (NLE) is an educational task in which children estimate the location of a value (e.g., 25) on a blank line that represents a numerical range (e.g., 0-100). NLE performance is a strong predictor of success in mathematics, and error patterns on this task help provide a glimpse into how children may represent number internally. However, a missing and fundamental element of this puzzle is the identification of neural correlates of NLE in children. That is, understanding possible neural signatures related to NLE performance will provide valuable insight into the cognitive processes that underlie children's development of NLE ability. Using functional near-infrared spectroscopy (fNIRS), we provide the first investigation of concurrent behavioral and cortical signatures of NLE performance in children. Specifically, our results highlight significant fronto-parietal changes in cortical activation in response to increases in NLE scale (e.g., 0-100 vs. 0-100,000). Furthermore, our results demonstrate that NLE performance feedback (auditory, visual, or audiovisual), as well as children's grade (2nd vs. 3rd) influence cortical responding during an NLE task.

Dynamics of brain activation during learning of syllable-symbol paired associations

Initial stages of reading acquisition require the learning of letter and speech sound combinations. While the long-term effects of audio-visual learning are rather well studied, relatively little is known about the short-term learning effects at the brain level. Here we examined the cortical dynamics of short-term learning using magnetoencephalography (MEG) and electroencephalography (EEG) in two experiments that respectively addressed active and passive learning of the association between shown symbols and heard syllables. In experiment 1, learning was based on feedback provided after each trial. The learning of the audio-visual associations was contrasted with items for which the feedback was meaningless. In experiment 2, learning was based on statistical learning through passive exposure to audio-visual stimuli that were consistently presented with each other and contrasted with audio-visual stimuli that were randomly paired with each other. After 5-10 min of training and exposure, learning-related changes emerged in neural activation around 200 and 350 ms in the two experiments. The MEG results showed activity changes at 350 ms in caudal middle frontal cortex and posterior superior temporal sulcus, and at 500 ms in temporo-occipital cortex. Changes in brain activity coincided with a decrease in reaction times and an increase in accuracy scores. Changes in EEG activity were observed starting at the auditory P2 response followed by later changes after 300 ms. The results show that the short-term learning effects emerge rapidly (manifesting in later stages of audio-visual integration processes) and that these effects are modulated by selective attention processes.

Single-Trial Phase Entrainment of Theta Oscillations in Sensory Regions Predicts Human Associative Memory Performance

Episodic memories are rich in sensory information and often contain integrated information from different sensory modalities. For instance, we can store memories of a recent concert with visual and auditory impressions being integrated in one episode. Theta oscillations have recently been implicated in playing a causal role synchronizing and effectively binding the different modalities together in memory. However, an open question is whether momentary fluctuations in theta synchronization predict the likelihood of associative memory formation for multisensory events. To address this question we entrained the visual and auditory cortex at theta frequency (4 Hz) and in a synchronous or asynchronous manner by modulating the luminance and volume of movies and sounds at 4 Hz, with a phase offset at 0° or 180°. EEG activity from human subjects (both sexes) was recorded while they memorized the association between a movie and a sound. Associative memory performance was significantly enhanced in the 0° compared with the 180° condition. Source-level analysis demonstrated that the physical stimuli effectively entrained their respective cortical areas with a corresponding phase offset. The findings suggested a successful replication of a previous study (Clouter et al., 2017). Importantly, the strength of entrainment during encoding correlated with the efficacy of associative memory such that small phase differences between visual and auditory cortex predicted a high likelihood of correct retrieval in a later recall test. These findings suggest that theta oscillations serve a specific function in the episodic memory system: binding the contents of different modalities into coherent memory episodes.SIGNIFICANCE STATEMENT How multisensory experiences are bound to form a coherent episodic memory representation is one of the fundamental questions in human episodic memory research. Evidence from animal literature suggests that the relative timing between an input and theta oscillations in the hippocampus is crucial for memory formation. We precisely controlled the timing between visual and auditory stimuli and the neural oscillations at 4 Hz using a multisensory entrainment paradigm. Human associative memory formation depends on coincident timing between sensory streams processed by the corresponding brain regions. We provide evidence for a significant role of relative timing of neural theta activity in human episodic memory on a single-trial level, which reveals a crucial mechanism underlying human episodic memory.

Temporal processing of audiovisual stimuli is enhanced in musicians: evidence from magnetoencephalography (MEG)

Numerous studies have demonstrated that the structural and functional differences between professional musicians and non-musicians are not only found within a single modality, but also with regard to multisensory integration. In this study we have combined psychophysical with neurophysiological measurements investigating the processing of non-musical, synchronous or various levels of asynchronous audiovisual events. We hypothesize that long-term multisensory experience alters temporal audiovisual processing already at a non-musical stage. Behaviorally, musicians scored significantly better than non-musicians in judging whether the auditory and visual stimuli were synchronous or asynchronous. At the neural level, the statistical analysis for the audiovisual asynchronous response revealed three clusters of activations including the ACC and the SFG and two bilaterally located activations in IFG and STG in both groups. Musicians, in comparison to the non-musicians, responded to synchronous audiovisual events with enhanced neuronal activity in a broad left posterior temporal region that covers the STG, the insula and the Postcentral Gyrus. Musicians also showed significantly greater activation in the left Cerebellum, when confronted with an audiovisual asynchrony. Taken together, our MEG results form a strong indication that long-term musical training alters the basic audiovisual temporal processing already in an early stage (direct after the auditory N1 wave), while the psychophysical results indicate that musical training may also provide behavioral benefits in the accuracy of the estimates regarding the timing of audiovisual events.

Strengths and weaknesses of multimodal processing in a group of adults with gliomas

The present study aimed to analyze the multimodal skills that would be spared, altered, or impaired by gliomas that slowly infiltrate various and diversely localized areas in the cerebral hemispheres. Ten patients and 60 healthy controls were evaluated using four multimodal processing paradigms across 11 tasks. Our objectives were as follows: (a) to describe the strengths and weaknesses of the glioma patients' multimodal processing performance after accounting for task specificity and their individual performances compared to those of the control group; (b) to determine the correlation between lesion localization and impairments; and (c) to identify the tasks that were most sensitive to tumor infiltration and plasticity limits. Our results show that patients as a whole were efficient at most tasks; however, the patients exhibited difficulties in the productive picture-naming task, the receptive verbal judgment task, and the visual/graphic portion of the dual-attention task. The individual case reports show that the difficulties were distributed across the patients and did not correlate with lesion localization and tumor type.

Age of onset of blindness affects brain anatomical networks constructed using diffusion tensor tractography

Studying blindness with various onset ages may elucidate the ways that unimodal sensory deprivation at different periods of development shape the human brain. In order to determine the effect of the onset age on brain anatomical networks, we extended a previous study of 17 early blind (EB) subjects with an additional 97 subjects with various onset ages. We constructed binary anatomical networks of these subjects and sighted controls (SC) using diffusion tensor tractography and calculated the topological properties of the network. Based on onset age, the subjects were divided into congenitally blind (CB), EB, adolescent-blind (AB), and late-blind (LB) subgroups. The LB subjects demonstrated a greater connectivity density and a higher global efficiency, similar to the SC. The CB and EB subgroups showed large group differences from the other groups in their topological networks, specifically, a reduced connectivity density and a decreased global efficiency compared with the SC, especially in the frontal and occipital cortices. Additionally, significant correlations were found between age of onset and the topological properties of the anatomical network in the blind. Our results suggest that visual experience during an early period of development is critical for establishing an intact efficient anatomical network in the human brain.

Cross-modal versus within-modal recall: differences in behavioral and brain responses

Although human experience is multisensory in nature, previous research has focused predominantly on memory for unisensory as opposed to multisensory information. In this work, we sought to investigate behavioral and neural differences between the cued recall of cross-modal audiovisual associations versus within-modal visual or auditory associations. Participants were presented with cue-target associations comprised of pairs of nonsense objects, pairs of nonsense sounds, objects paired with sounds, and sounds paired with objects. Subsequently, they were required to recall the modality of the target given the cue while behavioral accuracy, reaction time, and blood oxygenation level dependent (BOLD) activation were measured. Successful within-modal recall was associated with modality-specific reactivation in primary perceptual regions, and was more accurate than cross-modal retrieval. When auditory targets were correctly or incorrectly recalled using a cross-modal visual cue, there was re-activation in auditory association cortex, and recall of information from cross-modal associations activated the hippocampus to a greater degree than within-modal associations. Findings support theories that propose an overlap between regions active during perception and memory, and show that behavioral and neural differences exist between within- and cross-modal associations. Overall the current study highlights the importance of the role of multisensory information in memory.

Physical and perceptual factors shape the neural mechanisms that integrate audiovisual signals in speech comprehension

Face-to-face communication challenges the human brain to integrate information from auditory and visual senses with linguistic representations. Yet the role of bottom-up physical (spectrotemporal structure) input and top-down linguistic constraints in shaping the neural mechanisms specialized for integrating audiovisual speech signals are currently unknown. Participants were presented with speech and sinewave speech analogs in visual, auditory, and audiovisual modalities. Before the fMRI study, they were trained to perceive physically identical sinewave speech analogs as speech (SWS-S) or nonspeech (SWS-N). Comparing audiovisual integration (interactions) of speech, SWS-S, and SWS-N revealed a posterior-anterior processing gradient within the left superior temporal sulcus/gyrus (STS/STG): Bilateral posterior STS/STG integrated audiovisual inputs regardless of spectrotemporal structure or speech percept; in left mid-STS, the integration profile was primarily determined by the spectrotemporal structure of the signals; more anterior STS regions discarded spectrotemporal structure and integrated audiovisual signals constrained by stimulus intelligibility and the availability of linguistic representations. In addition to this "ventral" processing stream, a "dorsal" circuitry encompassing posterior STS/STG and left inferior frontal gyrus differentially integrated audiovisual speech and SWS signals. Indeed, dynamic causal modeling and Bayesian model comparison provided strong evidence for a parallel processing structure encompassing a ventral and a dorsal stream with speech intelligibility training enhancing the connectivity between posterior and anterior STS/STG. In conclusion, audiovisual speech comprehension emerges in an interactive process with the integration of auditory and visual signals being progressively constrained by stimulus intelligibility along the STS and spectrotemporal structure in a dorsal fronto-temporal circuitry.

Cross auditory-spatial learning in early-blind individuals

Cross-modal processing enables the utilization of information received via different sensory organs to facilitate more complicated human actions. We used functional MRI on early-blind individuals to study the neural processes associated with cross auditory-spatial learning. The auditory signals, converted from echoes of ultrasonic signals emitted from a navigation device, were novel to the participants. The subjects were trained repeatedly for 4 weeks in associating the auditory signals with different distances. Subjects' blood-oxygenation-level-dependent responses were captured at baseline and after training using a sound-to-distance judgment task. Whole-brain analyses indicated that the task used in the study involved auditory discrimination as well as spatial localization. The learning process was shown to be mediated by the inferior parietal cortex and the hippocampus, suggesting the integration and binding of auditory features to distances. The right cuneus was found to possibly serve a general rather than a specific role, forming an occipital-enhanced network for cross auditory-spatial learning. This functional network is likely to be unique to those with early blindness, since the normal-vision counterparts shared activities only in the parietal cortex.

Smoking experience modulates the cortical integration of vision and haptics

Human neuroplasticity of multisensory integration has been studied mainly in the context of natural or artificial training situations in healthy subjects. However, regular smokers also offer the opportunity to assess the impact of intensive daily multisensory interactions with smoking-related objects on the neural correlates of crossmodal object processing. The present functional magnetic resonance imaging study revealed that smokers show a comparable visuo-haptic integration pattern for both smoking paraphernalia and control objects in the left lateral occipital complex, a region playing a crucial role in crossmodal object recognition. Moreover, the degree of nicotine dependence correlated positively with the magnitude of visuo-haptic integration in the left lateral occipital complex (LOC) for smoking-associated but not for control objects. In contrast, in the left LOC non-smokers displayed a visuo-haptic integration pattern for control objects, but not for smoking paraphernalia. This suggests that prolonged smoking-related multisensory experiences in smokers facilitate the merging of visual and haptic inputs in the lateral occipital complex for the respective stimuli. Studying clinical populations who engage in compulsive activities may represent an ecologically valid approach to investigating the neuroplasticity of multisensory integration.

A model for memory systems based on processing modes rather than consciousness

Prominent models of human long-term memory distinguish between memory systems on the basis of whether learning and retrieval occur consciously or unconsciously. Episodic memory formation requires the rapid encoding of associations between different aspects of an event which, according to these models, depends on the hippocampus and on consciousness. However, recent evidence indicates that the hippocampus mediates rapid associative learning with and without consciousness in humans and animals, for long-term and short-term retention. Consciousness seems to be a poor criterion for differentiating between declarative (or explicit) and non declarative (or implicit) types of memory. A new model is therefore required in which memory systems are distinguished based on the processing operations involved rather than by consciousness.

Detection and characterization of single-trial fMRI bold responses: paradigm free mapping

This work presents a novel method of mapping the brain's response to single stimuli in space and time without prior knowledge of the paradigm timing: paradigm free mapping (PFM). This method is based on deconvolution of the hemodynamic response from the voxel time series assuming a linear response and using a ridge-regression algorithm. Statistical inference is performed by defining a spatio-temporal t-statistic and by controlling for multiple comparisons using the false discovery rate procedure. The methodology was validated on five subjects who performed self-paced and visually cued finger tapping at 7 Tesla, with moderate (TR = 2 s) and high (TR = 0.4 s) temporal resolution. The results demonstrate that detection of single-trial BOLD events is feasible without a priori information on the stimulus paradigm. The proposed method opens up the possibility of designing temporally unconstrained paradigms to study the cortical response to unpredictable mental events.

Cortical plasticity of audio-visual object representations

Several regions in human temporal and frontal cortex are known to integrate visual and auditory object features. The processing of audio–visual (AV) associations in these regions has been found to be modulated by object familiarity. The aim of the present study was to explore training-induced plasticity in human cortical AV integration. We used functional magnetic resonance imaging to analyze the neural correlates of AV integration for unfamiliar artificial object sounds and images in naïve subjects (PRE training) and after a behavioral training session in which subjects acquired associations between some of these sounds and images (POST-training). In the PRE-training session, unfamiliar artificial object sounds and images were mainly integrated in right inferior frontal cortex (IFC). The POST-training results showed extended integration-related IFC activations bilaterally, and a recruitment of additional regions in bilateral superior temporal gyrus/sulcus and intraparietal sulcus. Furthermore, training-induced differential response patterns to mismatching compared with matching (i.e., associated) artificial AV stimuli were most pronounced in left IFC. These effects were accompanied by complementary training-induced congruency effects in right posterior middle temporal gyrus and fusiform gyrus. Together, these findings demonstrate that short-term cross-modal association learning was sufficient to induce plastic changes of both AV integration of object stimuli and mechanisms of AV congruency processing.

Object familiarity and semantic congruency modulate responses in cortical audiovisual integration areas

The cortical integration of auditory and visual features is crucial for efficient object recognition. Previous studies have shown that audiovisual (AV) integration is affected by where and when auditory and visual features occur. However, because relatively little is known about the impact of what is integrated, we here investigated the impact of semantic congruency and object familiarity on the neural correlates of AV integration. We used functional magnetic resonance imaging to identify regions involved in the integration of both (congruent and incongruent) familiar animal sounds and images and of arbitrary combinations of unfamiliar artificial sounds and object images. Unfamiliar object images and sounds were integrated in the inferior frontal cortex (IFC), possibly reflecting learning of novel AV associations. Integration of familiar, but semantically incongruent combinations also correlated with IFC activation and additionally involved the posterior superior temporal sulcus (pSTS). For highly familiar semantically congruent AV pairings, we again found AV integration effects in pSTS and additionally in superior temporal gyrus. These findings demonstrate that the neural correlates of object-related AV integration reflect both semantic congruency and familiarity of the integrated sounds and images.

Human brain activity associated with audiovisual perception and attention

Coherent perception of objects in our environment often requires perceptual integration of auditory and visual information. Recent behavioral data suggest that audiovisual integration depends on attention. The current study investigated the neural basis of audiovisual integration using 3-Tesla functional magnetic resonance imaging (fMRI) in 12 healthy volunteers during attention to auditory or visual features, or audiovisual feature combinations of abstract stimuli (simultaneous harmonic sounds and colored circles). Audiovisual attention was found to modulate activity in the same frontal, temporal, parietal and occipital cortical regions as auditory and visual attention. In addition, attention to audiovisual feature combinations produced stronger activity in the superior temporal cortices than attention to only auditory or visual features. These modality-specific areas might be involved in attention-dependent perceptual binding of synchronous auditory and visual events into coherent audiovisual objects. Furthermore, the modality-specific temporal auditory and occipital visual cortical areas showed attention-related modulations during both auditory and visual attention tasks. This result supports the proposal that attention to stimuli in one modality can spread to encompass synchronously presented stimuli in another modality.

Functionally segregated neural substrates for arbitrary audiovisual paired-association learning

To clarify the neural substrates and their dynamics during crossmodal association learning, we conducted functional magnetic resonance imaging (MRI) during audiovisual paired-association learning of delayed matching-to-sample tasks. Thirty subjects were involved in the study; 15 performed an audiovisual paired-association learning task, and the remainder completed a control visuo-visual task. Each trial consisted of the successive presentation of a pair of stimuli. Subjects were asked to identify predefined audiovisual or visuo-visual pairs by trial and error. Feedback for each trial was given regardless of whether the response was correct or incorrect. During the delay period, several areas showed an increase in the MRI signal as learning proceeded: crossmodal activity increased in unimodal areas corresponding to visual or auditory areas, and polymodal responses increased in the occipitotemporal junction and parahippocampal gyrus. This pattern was not observed in the visuo-visual intramodal paired-association learning task, suggesting that crossmodal associations might be formed by binding unimodal sensory areas via polymodal regions. In both the audiovisual and visuo-visual tasks, the MRI signal in the superior temporal sulcus (STS) in response to the second stimulus and feedback peaked during the early phase of learning and then decreased, indicating that the STS might be key to the creation of paired associations, regardless of stimulus type. In contrast to the activity changes in the regions discussed above, there was constant activity in the frontoparietal circuit during the delay period in both tasks, implying that the neural substrates for the formation and storage of paired associates are distinct from working memory circuits.

Prototype-distortion category learning: a two-phase learning process across a distributed network

This paper reviews a body of work conducted in our laboratory that applies functional magnetic resonance imaging (fMRI) to better understand the biological response and change that occurs during prototype-distortion learning. We review results from two experiments (Little, Klein, Shobat, McClure, & Thulborn, 2004; Little & Thulborn, 2005) that provide support for increasing neuronal efficiency by way of a two-stage model that includes an initial period of recruitment of tissue across a distributed network that is followed by a period of increasing specialization with decreasing volume across the same network. Across the two studies, participants learned to classify patterns of random-dot distortions (Posner & Keele, 1968) into categories. At four points across this learning process subjects underwent examination by fMRI using a category-matching task. A large-scale network, altered across the protocol, was identified to include the frontal eye fields, both inferior and superior parietal lobules, and visual cortex. As behavioral performance increased, the volume of activation within these regions first increased and later in the protocol decreased. Based on our review of this work we propose that: (i) category learning is reflected as specialization of the same network initially implicated to complete the novel task, and (ii) this network encompasses regions not previously reported to be affected by prototype-distortion learning.

Neural correlates of relational memory: successful encoding and retrieval of semantic and perceptual associations

Using event-related functional magnetic resonance imaging, we identified brain regions involved in successful relational memory (RM) during encoding and retrieval for semantic and perceptual associations or in general, independent of phase and content. Participants were scanned while encoding and later retrieving associations between pairs of words (semantic RM) or associations between words and fonts (perceptual RM). Encoding success activity (ESA) was identified by comparing study-phase activity for items subsequently remembered (hits) versus forgotten (misses) and retrieval success activity (RSA) by comparing test-phase activity for hits versus misses. The study yielded three main sets of findings. First, ESA-RSA differences were found within the medial temporal lobes (MTLs) and within the prefrontal cortex (PFC). Within the left MTL, ESA was greater in the anterior hippocampus, and RSA was greater in the posterior parahippocampal cortex/hippocampus. This finding is consistent with the notion of an encoding-retrieval gradient along the longitudinal MTL axis. Within the left PFC, ESA was greater in ventrolateral PFC, and RSA was greater in dorsolateral and anterior PFC. This is the first evidence of a dissociation in successful encoding and retrieval activity within left PFC. Second, consistent with the transfer-appropriate processing principle, some ESA regions were reactivated during RSA in a content-specific manner. For semantic RM, these regions included the left ventrolateral PFC, whereas for perceptual RM, they included occipitoparietal and right parahippocampal regions. Finally, only one region in the entire brain was associated with RM in general (i.e., for both semantic and perceptual ESA and RSA): the left hippocampus. This finding highlights the fundamental role of the hippocampus in RM.

Correlations of cortical activation and behavior during the application of newly learned categories

Large individual differences are commonly observed during the early stages of category learning in both functional MRI (fMRI) activation maps and behavioral data. The current investigation characterizes this variability by correlating the volume of activation with behavioral performance. Healthy subjects were trained to classify patterns of random dots into categories. Training was carried out using a 4-choice categorization task with feedback. Functional MRI was performed prior to any training and then following each of 3 training sessions. The fMRI sessions involved the presentation of 3 separate paradigms which required the skill imparted by the training to determine whether two patterns of dots belonged to the same category. Contrasts between the 3 paradigms allowed the examination of the effects of training and of familiarity with the task. For fMRI performed with those materials used during training, increases in the volume of activation were observed initially. As behavioral performance continued to improve, reductions in activation were observed across regions involved in visuospatial processing and spatial attention. These reductions in activation were observed only for those materials used in training and only after high levels of performance were achieved. The magnitude of these reductions in activation correlated with each individual's own rate of learning. The present data support the observation that at least two stages of cortical activation underlie the use of newly learned categories. The first, recruitment of nearby tissue, is observed as initial increases in the volumes of activation. These initial stages of recruitment are followed by specialization across the same network which is observed as a reduction in activation with continued improvements in behavioral performance.

Changing patterns of brain activation during category learning revealed by functional MRI

Functional magnetic resonance imaging (fMRI) was used to investigate neural changes as a function of category learning in normals (n=8). Subjects were trained to classify patterns of dots into four categories over 4 consecutive days. fMRI monitored the changes that occurred during learning prior to training and then following each training session. During fMRI, subjects determined whether two patterns of dots were members of the same category. The behavioral changes that occurred as a result of the training were observed as increases in response accuracy within shortened response times. fMRI illustrated initial increases in volumes of activation distributed across the known visuospatial processing networks. The regions affected by learning were identified as those involved in the planning and execution of eye movements (frontal and supplementary eye fields, FEF and SEF), spatial attention (superior and inferior parietal lobules, SPL and IPL) and visual processing (primary, secondary, and tertiary visual cortices). The volumes of activation then decreased as training progressed further. Of the two proposed mechanisms for learning, that of strengthened connectivity on a given network and that of selection of different networks, our data supports the former.

Acoustic noise and functional magnetic resonance imaging: current strategies and future prospects

Functional magnetic resonance imaging (fMRI) has become the method of choice for studying the neural correlates of cognitive tasks. Nevertheless, the scanner produces acoustic noise during the image acquisition process, which is a problem in the study of auditory pathway and language generally. The scanner acoustic noise not only produces activation in brain regions involved in auditory processing, but also interferes with the stimulus presentation. Several strategies can be used to address this problem, including modifications of hardware and software. Although reduction of the source of the acoustic noise would be ideal, substantial hardware modifications to the current base of installed MRI systems would be required. Therefore, the most common strategy employed to minimize the problem involves software modifications. In this work we consider three main types of acquisitions: compressed, partially silent, and silent. For each implementation, paradigms using block and event-related designs are assessed. We also provide new data, using a silent event-related (SER) design, which demonstrate higher blood oxygen level-dependent (BOLD) response to a simple auditory cue when compared to a conventional image acquisition.

Early auditory-visual interactions in human cortex during nonredundant target identification

A common finding of behavioral studies is that objects characterized by redundant multisensory cues are identified more rapidly than the same objects presented in either unimodal condition. In a previous electrophysiological study in humans, we have described a network of crossmodal interactions that could be associated with this facilitation effect [M.H. Giard, F. Peronnet, J. Cogn. Neurosci. 11(5) (1999) 473-490]. Here, we sought to determine whether the recognition of objects characterized by nonredundant bimodal components may still induce crossmodal neural interactions. Subjects had to identify three objects defined either by auditory or visual features alone, or by the combination of nonredundant auditory and visual features. As expected, behavioral measures showed no sign of facilitation in bimodal processing. Yet, event-related potential analysis revealed the existence of early (<200 ms latency) crossmodal activities in sensory-specific and nonspecific cortical areas, that were partly dependent on the sensory dominance of the subjects to perform the task. Comparative analysis of the interaction patterns involved in redundant and nonredundant cue processing provides evidence for the robustness of the principle of crossmodal neural synergy that applies whatever the stimulus content (redundant or nonredundant information), and for the high flexibility of the neural networks of integration that are sensitive both to the nature of the perceptual task and to the sensory skill of the individual in that particular task.

Detection of audio-visual integration sites in humans by application of electrophysiological criteria to the BOLD effect

Electrophysiological studies in nonhuman primates and other mammals have shown that sensory cues from different modalities that appear at the same time and in the same location can increase the firing rate of multisensory cells in the superior colliculus to a level exceeding that predicted by summing the responses to the unimodal inputs. In contrast, spatially disparate multisensory cues can induce a profound response depression. We have previously demonstrated using functional magnetic resonance imaging (fMRI) that similar indices of crossmodal facilitation and inhibition are detectable in human cortex when subjects listen to speech while viewing visually congruent and incongruent lip and mouth movements. Here, we have used fMRI to investigate whether similar BOLD signal changes are observable during the crossmodal integration of nonspeech auditory and visual stimuli, matched or mismatched solely on the basis of their temporal synchrony, and if so, whether these crossmodal effects occur in similar brain areas as those identified during the integration of audio-visual speech. Subjects were exposed to synchronous and asynchronous auditory (white noise bursts) and visual (B/W alternating checkerboard) stimuli and to each modality in isolation. Synchronous and asynchronous bimodal inputs produced superadditive BOLD response enhancement and response depression across a large network of polysensory areas. The most highly significant of these crossmodal gains and decrements were observed in the superior colliculi. Other regions exhibiting these crossmodal interactions included cortex within the superior temporal sulcus, intraparietal sulcus, insula, and several foci in the frontal lobe, including within the superior and ventromedial frontal gyri. These data demonstrate the efficacy of using an analytic approach informed by electrophysiology to identify multisensory integration sites in humans and suggest that the particular network of brain areas implicated in these crossmodal integrative processes are dependent on the nature of the correspondence between the different sensory inputs (e.g. space, time, and/or form).

Integrated brain activity in medial temporal and prefrontal areas predicts subsequent memory performance: human declarative memory formation at the system level

After an era in which lesion studies have identified the declarative memory system and its essential anatomical structures, functional imaging and event-related potential studies have begun to delineate the neural underpinnings of declarative memory formation at the system level. By memory formation, we refer to those mnemonic processes present during encoding that transform perceptual representations into enduring memories. Recent studies have revealed that distinct regions in medial temporal and prefrontal areas exhibit more neural activity during successful than unsuccessful memory formation. We attempt to identify the nature of the processes underlying these subsequent memory effects. Reviewed data suggest specific mnemonic operations in the medial temporal lobe that may be integrated with semantic/perceptual operations and subserving operations in the prefrontal cortex. The formation of relational and non-relational memories may be supported by distinct subregions within these two brain regions. While the medial temporal lobe may have a serial organizational structure, with a processing hierarchy, interactions between medial temporal and prefrontal areas seem to occur in a parallel and bi-directional fashion. Interacting with this system, emotionally arousing events enhance neural activity in the amygdala, which in turn may modulate processing in other brain regions responsible for declarative memory formation.

Current awareness

In order to keep subscribers up-to-date with the latest developments in their field, John Wiley & Sons are providing a current awareness service in each issue of the journal. The bibliography contains newly published material in the field of NMR in biomedicine. Each bibliography is divided into 9 sections: 1 Books, Reviews ' Symposia; 2 General; 3 Technology; 4 Brain and Nerves; 5 Neuropathology; 6 Cancer; 7 Cardiac, Vascular and Respiratory Systems; 8 Liver, Kidney and Other Organs; 9 Muscle and Orthopaedic. Within each section, articles are listed in alphabetical order with respect to author. If, in the preceding period, no publications are located relevant to any one of these headings, that section will be omitted.