Skill dependent audiovisual integration in the fusiform induces repetition suppression

Letter-speech sound integration in typical reading development during the first years of formal education

This study investigated the neural basis of letter and speech sound (LS) integration in 53 typical readers (35 girls, all White) during the first 2 years of reading education (ages 7-9). Changes in both sensory (multisensory vs unisensory) and linguistic (congruent vs incongruent) aspects of LS integration were examined. The left superior temporal cortex and bilateral inferior frontal cortex showed increasing activation for multisensory over unisensory LS over time, driven by reduced activation to speech sounds. No changes were noted in the congruency effect. However, at age nine, heightened activation to incongruent over congruent LS pairs were observed, correlating with individual differences in reading development. This suggests that the incongruency effect evolves at varying rates depending on reading skills.

Neural basis of sound-symbolic pseudoword-shape correspondences

Non-arbitrary mapping between the sound of a word and its meaning, termed sound symbolism, is commonly studied through crossmodal correspondences between sounds and visual shapes, e.g., auditory pseudowords, like 'mohloh' and 'kehteh', are matched to rounded and pointed visual shapes, respectively. Here, we used functional magnetic resonance imaging (fMRI) during a crossmodal matching task to investigate the hypotheses that sound symbolism (1) involves language processing; (2) depends on multisensory integration; (3) reflects embodiment of speech in hand movements. These hypotheses lead to corresponding neuroanatomical predictions of crossmodal congruency effects in (1) the language network; (2) areas mediating multisensory processing, including visual and auditory cortex; (3) regions responsible for sensorimotor control of the hand and mouth. Right-handed participants (n = 22) encountered audiovisual stimuli comprising a simultaneously presented visual shape (rounded or pointed) and an auditory pseudoword ('mohloh' or 'kehteh') and indicated via a right-hand keypress whether the stimuli matched or not. Reaction times were faster for congruent than incongruent stimuli. Univariate analysis showed that activity was greater for the congruent compared to the incongruent condition in the left primary and association auditory cortex, and left anterior fusiform/parahippocampal gyri. Multivoxel pattern analysis revealed higher classification accuracy for the audiovisual stimuli when congruent than when incongruent, in the pars opercularis of the left inferior frontal (Broca's area), the left supramarginal, and the right mid-occipital gyri. These findings, considered in relation to the neuroanatomical predictions, support the first two hypotheses and suggest that sound symbolism involves both language processing and multisensory integration.

Similarities and differences in the neural correlates of letter and speech sound integration in blind and sighted readers

Learning letter and speech sound (LS) associations is a major step in reading acquisition common for all alphabetic scripts, including Braille used by blind readers. The left superior temporal cortex (STC) plays an important role in audiovisual LS integration in sighted people, but it is still unknown what neural mechanisms are responsible for audiotactile LS integration in blind individuals. Here, we investigated the similarities and differences between LS integration in blind Braille (N = 42, age range: 9-60 y.o.) and sighted print (N = 47, age range: 9-60 y.o.) readers who acquired reading using different sensory modalities. In both groups, the STC responded to both isolated letters and isolated speech sounds, showed enhanced activation when they were presented together, and distinguished between congruent and incongruent letter and speech sound pairs. However, the direction of the congruency effect was different between the groups. Sighted subjects showed higher activity for incongruent LS pairs in the bilateral STC, similarly to previously studied typical readers of transparent orthographies. In the blind, congruent pairs resulted in an increased response in the right STC. These differences may be related to more sequential processing of Braille as compared to print reading. At the same time, behavioral efficiency in LS discrimination decisions and the congruency effect were found to be related to age and reading skill only in sighted participants, suggesting potential differences in the developmental trajectories of LS integration between blind and sighted readers.

Beyond the Visual Word Form Area - a cognitive characterization of the left ventral occipitotemporal cortex

The left ventral occipitotemporal cortex has been traditionally viewed as a pathway for visual object recognition including written letters and words. Its crucial role in reading was strengthened by the studies on the functionally localized "Visual Word Form Area" responsible for processing word-like information. However, in the past 20 years, empirical studies have challenged the assumptions of this brain region as processing exclusively visual or even orthographic stimuli. In this review, we aimed to present the development of understanding of the left ventral occipitotemporal cortex from the visually based letter area to the modality-independent symbolic language related region. We discuss theoretical and empirical research that includes orthographic, phonological, and semantic properties of language. Existing results showed that involvement of the left ventral occipitotemporal cortex is not limited to unimodal activity but also includes multimodal processes. The idea of the integrative nature of this region is supported by the broad functional and structural connectivity with language-related and attentional brain networks. We conclude that although the function of the area is not yet fully understood in human cognition, its role goes beyond visual word form processing. The left ventral occipitotemporal cortex seems to be crucial for combining higher-level language information with abstract forms that convey meaning independently of modality.

Neural Basis Of Sound-Symbolic Pseudoword-Shape Correspondences

Non-arbitrary mapping between the sound of a word and its meaning, termed sound symbolism, is commonly studied through crossmodal correspondences between sounds and visual shapes, e.g., auditory pseudowords, like 'mohloh' and 'kehteh', are matched to rounded and pointed visual shapes, respectively. Here, we used functional magnetic resonance imaging (fMRI) during a crossmodal matching task to investigate the hypotheses that sound symbolism (1) involves language processing; (2) depends on multisensory integration; (3) reflects embodiment of speech in hand movements. These hypotheses lead to corresponding neuroanatomical predictions of crossmodal congruency effects in (1) the language network; (2) areas mediating multisensory processing, including visual and auditory cortex; (3) regions responsible for sensorimotor control of the hand and mouth. Right-handed participants ( n = 22) encountered audiovisual stimuli comprising a simultaneously presented visual shape (rounded or pointed) and an auditory pseudoword ('mohloh' or 'kehteh') and indicated via a right-hand keypress whether the stimuli matched or not. Reaction times were faster for congruent than incongruent stimuli. Univariate analysis showed that activity was greater for the congruent compared to the incongruent condition in the left primary and association auditory cortex, and left anterior fusiform/parahippocampal gyri. Multivoxel pattern analysis revealed higher classification accuracy for the audiovisual stimuli when congruent than when incongruent, in the pars opercularis of the left inferior frontal (Broca's area), the left supramarginal, and the right mid-occipital gyri. These findings, considered in relation to the neuroanatomical predictions, support the first two hypotheses and suggest that sound symbolism involves both language processing and multisensory integration.

HIGHLIGHTS

fMRI investigation of sound-symbolic correspondences between auditory pseudowords and visual shapesFaster reaction times for congruent than incongruent audiovisual stimuliGreater activation in auditory and visual cortices for congruent stimuliHigher classification accuracy for congruent stimuli in language and visual areasSound symbolism involves language processing and multisensory integration.

Neural correlates of multisensory integration in the human brain: an ALE meta-analysis

Previous fMRI research identified superior temporal sulcus as central integration area for audiovisual stimuli. However, less is known about a general multisensory integration network across senses. Therefore, we conducted activation likelihood estimation meta-analysis with multiple sensory modalities to identify a common brain network. We included 49 studies covering all Aristotelian senses i.e., auditory, visual, tactile, gustatory, and olfactory stimuli. Analysis revealed significant activation in bilateral superior temporal gyrus, middle temporal gyrus, thalamus, right insula, and left inferior frontal gyrus. We assume these regions to be part of a general multisensory integration network comprising different functional roles. Here, thalamus operate as first subcortical relay projecting sensory information to higher cortical integration centers in superior temporal gyrus/sulcus while conflict-processing brain regions as insula and inferior frontal gyrus facilitate integration of incongruent information. We additionally performed meta-analytic connectivity modelling and found each brain region showed co-activations within the identified multisensory integration network. Therefore, by including multiple sensory modalities in our meta-analysis the results may provide evidence for a common brain network that supports different functional roles for multisensory integration.

Neurofunctional basis underlying audiovisual integration of print and speech sound in Chinese children

Effortless print-sound integration is essential to reading development, and the superior temporal cortex (STC) is the most critical brain region. However, to date, the conclusion is almost restricted to alphabetic orthographies. To examine the neural basis in non-alphabetic languages and its relationship with reading abilities, we conducted a functional magnetic resonance imaging study in typically developing Chinese children. Two neuroimaging-based indicators of audiovisual processing-additive enhancement (higher activation in the congruent than the average activation of unimodal conditions) and neural integration (different activations between the congruent versus incongruent conditions)-were used to investigate character-sounds (opaque) and pinyin-sounds (transparent) processing. We found additive enhancement in bilateral STCs processing both character and pinyin stimulations. Moreover, the neural integrations in the left STC for the two scripts were strongly correlated. In terms of differentiation, first, areas beyond the STCs showed additive enhancement in processing pinyin-sounds. Second, while the bilateral STCs, left inferior/middle frontal and parietal regions manifested a striking neural integration (incongruent > congruent) for character-sounds, no significant clusters were revealed for pinyin-sounds. Finally, the neural integration in the left middle frontal gyrus for characters was specifically associated with silent reading comprehension proficiency, indicating automatic semantic processing during implicit character-sound integration. In contrast, the neural integration in the left STC for pinyin was specifically associated with oral reading fluency that relies on grapho-phonological mapping. To summarize, this study revealed both script-universal and script-specific neurofunctional substrates of print-sound integration as well as their processing- and region-dependent associations with reading abilities in typical Chinese children.

How Learning to Read Changes the Listening Brain

Reading acquisition reorganizes existing brain networks for speech and visual processing to form novel audio-visual language representations. This requires substantial cortical plasticity that is reflected in changes in brain activation and functional as well as structural connectivity between brain areas. The extent to which a child's brain can accommodate these changes may underlie the high variability in reading outcome in both typical and dyslexic readers. In this review, we focus on reading-induced functional changes of the dorsal speech network in particular and discuss how its reciprocal interactions with the ventral reading network contributes to reading outcome. We discuss how the dynamic and intertwined development of both reading networks may be best captured by approaching reading from a skill learning perspective, using audio-visual learning paradigms and longitudinal designs to follow neuro-behavioral changes while children's reading skills unfold.

Cortical responses to letters and ambiguous speech vary with reading skills in dyslexic and typically reading children

•

Text recalibrates ambiguous speech perception in children with and without dyslexia.

•

Dyslexia and poorer reading skills are linked to reduced left fusiform activation.

•

Poorer letter-speech sound matching is linked to higher superior temporal activation.

One of the proposed issues underlying reading difficulties in dyslexia is insufficiently automatized letter-speech sound associations. In the current fMRI experiment, we employ text-based recalibration to investigate letter-speech sound mappings in 8–10 year-old children with and without dyslexia. Here an ambiguous speech sound /a?a/ midway between /aba/ and /ada/ is combined with disambiguating “aba” or “ada” text causing a perceptual shift of the ambiguous /a?a/ sound towards the text (recalibration). This perceptual shift has been found to be reduced in adults but not in children with dyslexia compared to typical readers. Our fMRI results show significantly reduced activation in the left fusiform in dyslexic compared to typical readers, despite comparable behavioural performance. Furthermore, enhanced audio-visual activation within this region was linked to better reading and phonological skills. In contrast, higher activation in bilateral superior temporal cortex was associated with lower letter-speech sound identification fluency. These findings reflect individual differences during the early stages of reading development with reduced recruitment of the left fusiform in dyslexic readers together with an increased involvement of the superior temporal cortex in children with less automatized letter-speech sound associations.

The Connectivity Fingerprints of Highly-Skilled and Disordered Reading Persist Across Cognitive Domains

The capacity to produce and understand written language is a uniquely human skill that exists on a continuum, and foundational to other facets of human cognition. Multivariate classifiers based on support vector machines (SVM) have provided much insight into the networks underlying reading skill beyond what traditional univariate methods can tell us. Shallow models like SVM require large amounts of data, and this problem is compounded when functional connections, which increase exponentially with network size, are predictors of interest. Data reduction using independent component analyses (ICA) mitigates this problem, but conventionally assumes linear relationships. Multilayer feedforward networks, in contrast, readily find optimal low-dimensional encodings of complex patterns that include complex nonlinear or conditional relationships. Samples of poor and highly-skilled young readers were selected from two open access data sets using rhyming and mental multiplication tasks, respectively. Functional connectivity was computed for the rhyming task within a functionally-defined reading network and used to train multilayer feedforward classifier models to simultaneously associate functional connectivity patterns with lexicality (word vs. pseudoword) and reading skill (poor vs. highly-skilled). Classifiers identified validation set lexicality with significantly better than chance accuracy, and reading skill with near-ceiling accuracy. Critically, a series of replications used pre-trained rhyming-task models to classify reading skill from mental multiplication task participants' connectivity with near-ceiling accuracy. The novel deep learning approach presented here provides the clearest demonstration to date that reading-skill dependent functional connectivity within the reading network influences brain processing dynamics across cognitive domains.

Task load modulates tDCS effects on brain network for phonological processing

Motor participation in phonological processing can be modulated by task nature across the speech perception to speech production range. The pars opercularis of the left inferior frontal gyrus (LIFG) would be increasingly active across this range, because of changing motor demands. Here, we investigated with simultaneous tDCS and fMRI whether the task load modulation of tDCS effects translates into predictable patterns of functional connectivity. Findings were analysed under the "multi-node framework", according to which task load and the network structure underlying cognitive functions are modulators of tDCS effects. In a within-subject study, participants (N = 20) performed categorical perception, lexical decision and word naming tasks [which differentially recruit the target of stimulation (LIFG)], which were repeatedly administered in three tDCS sessions (anodal, cathodal and sham). The LIFG, left superior temporal gyrus and their right homologues formed the target network subserving phonological processing. C-tDCS inhibition and A-tDCS excitation should increase with task load. Correspondingly, the larger the task load, the larger the relevance of the target for the task and smaller the room for compensation of C-tDCS inhibition by less relevant nodes. Functional connectivity analyses were performed with partial correlations, and network compensation globally inferred by comparing the relative number of significant connections each condition induced relative to sham. Overall, simultaneous tDCS and fMRI was adequate to show that motor participation in phonological processing is modulated by task nature. Network responses induced by C-tDCS across phonological processing tasks matched predictions. A-tDCS effects were attributed to optimisation of network efficiency.

A longitudinal neuroimaging dataset on multisensory lexical processing in school-aged children

Here we describe the open access dataset entitled “Longitudinal Brain Correlates of Multisensory Lexical Processing in Children” hosted on OpenNeuro.org. This dataset examines reading development through a longitudinal multimodal neuroimaging and behavioral approach, including diffusion-weighted and T1-weighted structural magnetic resonance imaging (MRI), task based functional MRI, and a battery of psycho-educational assessments and parental questionnaires. Neuroimaging, psycho-educational testing, and functional task behavioral data were collected from 188 typically developing children when they were approximately 10.5 years old (session T1). Seventy children returned approximately 2.5 years later (session T2), of which all completed longitudinal follow-ups of psycho-educational testing, and 49 completed neuroimaging and functional tasks. At session T1 participants completed auditory, visual, and audio-visual word and pseudo-word rhyming judgment tasks in the scanner. At session T2 participants completed visual word and pseudo-word rhyming judgement tasks in the scanner.

Measurement(s)	reading and spelling ability • intelligence • brain • brain physiology trait
Technology Type(s)	psychoeducational test administration • magnetic resonance imaging • functional magnetic resonance imaging • Diffusion Weighted Imaging
Factor Type(s)	age • reading disability • type of task • parental educational level
Sample Characteristic - Organism	Homo sapiens

Machine-accessible metadata file describing the reported data: 10.6084/m9.figshare.11298188

A mesial-to-lateral dissociation for orthographic processing in the visual cortex

Efficient reading requires a fast conversion of the written word to both phonological and semantic codes. We tested the hypothesis that, within the left occipitotemporal cortical regions involved in visual word recognition, distinct subregions harbor slightly different orthographic codes adapted to those 2 functions. While the lexico-semantic pathway may operate on letter or open-bigram information, the phonological pathway requires the identification of multiletter graphemes such as "ch" or "ou" in order to map them onto phonemes. To evaluate the existence of a specific stage of graphemic encoding, 20 adults performed lexical decision and naming tasks on words and pseudowords during functional MRI. Graphemic encoding was facilitated or disrupted by coloring and spacing the letters either congruently with multiletter graphemes (ch-ai-r) or incongruently with them (c-ha-ir). This manipulation affected behavior, primarily during the naming of pseudowords, and modulated brain activity in the left midfusiform sulcus, at a site medial to the classical visual word form area (VWFA). This putative grapheme-related area (GRA) differed from the VWFA in being preferentially connected functionally to dorsal parietal areas involved in letter-by-letter reading, while the VWFA showed effects of lexicality and spelling-to-sound regularity. Our results suggest a partial dissociation within left occipitotemporal cortex: the midfusiform GRA would encode orthographic information at a sublexical graphemic level, while the lateral occipitotemporal VWFA would contribute primarily to direct lexico-semantic access.

Increased Functional Brain Network Efficiency During Audiovisual Temporal Asynchrony Integration Task in Aging

Audiovisual integration significantly changes over the lifespan, but age-related functional connectivity in audiovisual temporal asynchrony integration tasks remains underexplored. In the present study, electroencephalograms (EEGs) of 27 young adults (22–25 years) and 25 old adults (61–76 years) were recorded during an audiovisual temporal asynchrony integration task with seven conditions [auditory (A), visual (V), AV, A50V, A100V, V50A and V100A]. We calculated the phase lag index (PLI)-weighted connectivity networks modulated by the audiovisual tasks and found that the PLI connections showed obvious dynamic changes after stimulus onset. In the theta (4–7 Hz) and alpha (8–13 Hz) bands, the AV and V50A conditions induced stronger functional connections and higher global and local efficiencies, reflecting a stronger audiovisual integration effect, which was attributed to the auditory information arriving at the primary auditory cortex earlier than the visual information reaching the primary visual cortex. Importantly, the functional connectivity and network efficiencies of old adults revealed higher global and local efficiencies and higher degree in both the theta and alpha bands. These larger network efficiencies indicated that old adults might experience more difficulties in attention and cognitive control during the audiovisual integration task with temporal asynchrony than young adults. There were significant associations between network efficiencies and peak time of integration only in young adults. We propose that an audiovisual task with multiple conditions might arouse the appropriate attention in young adults but would lead to a ceiling effect in old adults. Our findings provide new insights into the network topography of old adults during audiovisual integration and highlight higher functional connectivity and network efficiencies due to greater cognitive demand.

Longitudinal Task-Related Functional Connectivity Changes Predict Reading Development

Longitudinal studies suggest developmentally dependent changes in lexical processing during reading development, implying a change in inter-regional functional connectivity over this period. The current study used functional magnetic resonance imaging (fMRI) to explore developmental changes in functional connectivity across multiple runs of a rhyming judgment task in young readers (8–14 years) over an average 2.5-year span. Changes in functional segregation are correlated with and predict changes in the skill with which typically developing children learn to apply the alphabetic principle, as measured by pseudoword decoding. This indicates a developmental shift in the proportion of specialized functional clusters is associated with changes in reading skill and suggests a dependency of reading development on changes of particular neural pathways, specifically decreases in transitivity is indicative of greater network integration. This work provides evidence that characteristics of these pathways, quantified using graph-theoretic metrics, can be used to predict individual differences in reading development.

Cross-modal comparisons of stimulus specificity and commonality in phonological processing

Phonological processing is a fundamental ability which underlies language comprehension. Functional neuroanatomy of phonology constitutes a matter of ongoing debate. In the present study, subjects performed visual (rhyme detection) and auditory (identification of spoken words starting with a given consonant) tasks that were contrasted with matched nonverbal tasks. We identified regions critical for phonological processing which were either stimulus specific or supramodal. The results revealed a high degree of modality specificity in both visual and auditory networks. Moreover, we observed a modality independent region in the left middle temporal gyrus (MTG)/superior temporal sulcus (STS), between a more anterior temporal area with auditory specificity and a more posterior temporal area with visual specificity. This dissociation in functional neuroanatomy suggests that this area may be a core region for supramodal phonological processing, acting as a gateway between spatially separate, but stimulus specific, phonological processes and more general linguistic functions.