Reading Increases the Compositionality of Visual Word Representations

Characterizing different types of developmental dyslexias in French: The Malabi screener

Reading is a complex process involving multiple stages. An impairment in any of these stages may cause distinct types of reading deficits- distinct types of dyslexia. We describe the Malabi, a screener to identify deficits in various orthographic, lexical, and sublexical components of the reading process in French. The Malabi utilizes stimuli that are sensitive to different forms of dyslexia, including "attentional dyslexia", as it is traditionally refered to, characterized by letter-to-word binding impairments leading to letter migrations between words (e.g., "bar cat" misread as "bat car"), and "letter-position dyslexia", resulting in letter transpositions within words (e.g., "destiny" misread as "density"). After collecting reading error norms from 138 French middle-school students, we analyzed error types of 16 students with developmental dyslexia. We identified three selective cases of attentional dyslexia and one case of letter-position dyslexia. Further tests confirmed our diagnosis and demonstrate, for the first time, how these dyslexias are manifested in French. These results underscore the significance of recognizing and discussing the existence of multiple dyslexias, both in research contexts when selecting participants for dyslexia studies, and in practical settings where educators and practitioners work with students to develop personalized support. The test and supporting materials are available on Open Science Framework (https://osf.io/3pgzb/).

Does the visual word form area split in bilingual readers? A millimeter-scale 7-T fMRI study

In expert readers, a brain region known as the visual word form area (VWFA) is highly sensitive to written words, exhibiting a posterior-to-anterior gradient of increasing sensitivity to orthographic stimuli whose statistics match those of real words. Using high-resolution 7-tesla functional magnetic resonance imaging (fMRI), we ask whether, in bilingual readers, distinct cortical patches specialize for different languages. In 21 English-French bilinguals, unsmoothed 1.2-millimeters fMRI revealed that the VWFA is actually composed of several small cortical patches highly selective for reading, with a posterior-to-anterior word-similarity gradient, but with near-complete overlap between the two languages. In 10 English-Chinese bilinguals, however, while most word-specific patches exhibited similar reading specificity and word-similarity gradients for reading in Chinese and English, additional patches responded specifically to Chinese writing and, unexpectedly, to faces. Our results show that the acquisition of multiple writing systems can indeed tune the visual cortex differently in bilinguals, sometimes leading to the emergence of cortical patches specialized for a single language.

General object-based features account for letter perception

After years of experience, humans become experts at perceiving letters. Is this visual capacity attained by learning specialized letter features, or by reusing general visual features previously learned in service of object categorization? To explore this question, we first measured the perceptual similarity of letters in two behavioral tasks, visual search and letter categorization. Then, we trained deep convolutional neural networks on either 26-way letter categorization or 1000-way object categorization, as a way to operationalize possible specialized letter features and general object-based features, respectively. We found that the general object-based features more robustly correlated with the perceptual similarity of letters. We then operationalized additional forms of experience-dependent letter specialization by altering object-trained networks with varied forms of letter training; however, none of these forms of letter specialization improved the match to human behavior. Thus, our findings reveal that it is not necessary to appeal to specialized letter representations to account for perceptual similarity of letters. Instead, we argue that it is more likely that the perception of letters depends on domain-general visual features.

For over a century, scientists have conducted behavioral experiments to investigate how the visual system recognizes letters, but it has proven difficult to propose a model of the feature space underlying this capacity. Here we leveraged recent advances in machine learning to model a wide variety of features ranging from specialized letter features to general object-based features. Across two large-scale behavioral experiments we find that general object-based features account well for letter perception, and that adding letter specialization did not improve the correspondence to human behavior. It is plausible that the ability to recognize letters largely relies on general visual features unaltered by letter learning.

Trailblazers in Neuroscience: Using compositionality to understand how parts combine in whole objects

A fundamental question for any visual system is whether its image representation can be understood in terms of its components. Decomposing any image into components is challenging because there are many possible decompositions with no common dictionary, and enumerating them leads to a combinatorial explosion. Even in perception, many objects are readily seen as containing parts, but there are many exceptions. These exceptions include objects that are not perceived as containing parts, properties like symmetry that cannot be localized to any single part, and also special categories like words and faces whose perception is widely believed to be holistic. Here, I describe a novel approach we have used to address these issues and evaluate compositionality at the behavioral and neural levels. The key design principle is to create a large number of objects by combining a small number of pre-defined components in all possible ways. This allows for building component-based models that explain whole objects using a combination of these components. Importantly, any systematic error in model fits can be used to detect the presence of emergent or holistic properties. Using this approach, we have found that whole object representations are surprisingly predictable from their components, that some components are preferred to others in perception, and that emergent properties can be discovered or explained using compositional models. Thus, compositionality is a powerful approach for understanding how whole objects relate to their parts.

Evolution of reading and face circuits during the first three years of reading acquisition

Although words and faces activate neighboring regions in the fusiform gyrus, we lack an understanding of how such category selectivity emerges during development. To investigate the organization of reading and face circuits at the earliest stage of reading acquisition, we measured the fMRI responses to words, faces, houses, and checkerboards in three groups of 60 French children: 6-year-old pre-readers, 6-year-old beginning readers and 9-year-old advanced readers. The results showed that specific responses to written words were absent prior to reading, but emerged in beginning readers, irrespective of age. Likewise, specific responses to faces were barely visible in pre-readers and continued to evolve in the 9-year-olds, yet primarily driven by age rather than by schooling. Crucially, the sectors of ventral visual cortex that become specialized for words and faces harbored their own functional connectivity prior to reading acquisition: the VWFA with left-hemispheric spoken language areas, and the FFA with the contralateral region and the amygdalae. The results support the view that reading acquisition occurs through the recycling of a pre-existing but plastic circuit which, in pre-readers, already connects the VWFA site to other distant language areas. We argue that reading acquisition does not compete with the face system directly, through a pruning of preexisting face responses, but indirectly, by hindering the slow growth of face responses in the left hemisphere, thus increasing a pre-existing right hemispheric bias.

Non-monotonic developmental trend of holistic processing in visual expertise: the case of Chinese character recognition

Holistic processing has been identified as an expertise marker of face and object recognition. By contrast, reduced holistic processing is purportedly an expertise marker in recognising orthographic characters in Chinese. Does holistic processing increase or decrease in expertise development? Is orthographic recognition a domain-specific exception to all other kinds of recognition (e.g. face and objects)? In two studies, we examined the developmental trend of holistic processing in Chinese character recognition in Chinese and non-Chinese children, and its relationship with literacy abilities: Chinese first graders—with emergent Chinese literacy acquired in kindergarten—showed increased holistic processing perhaps as an inchoate expertise marker when compared with kindergartners and non-Chinese first graders; however, the holistic processing effect was reduced in higher-grade Chinese children. These results suggest a non-monotonic inverted U-shape trend of holistic processing in visual expertise development: An increase in holistic processing due to initial reading experience followed by a decrease in holistic processing due to literacy enhancement. This result marks the development of holistic and analytic processing skills, both of which can be essential for mastering visual recognition. This study is the first to investigate the developmental trend of holistic processing in Chinese character recognition using the composite paradigm.

A separable neural code in monkey IT enables perfect CAPTCHA decoding

Reading distorted letters is easy for us but so challenging for the machine vision that it is used on websites as CAPTCHA (Completely Automated Public Turing Test to tell Computers and Humans Apart). How does our brain solve this problem? One solution is to have neurons selective for letter combinations but invariant to distortions. Another is for neurons to encode letter distortions and longer strings to enable separable decoding. Here, we provide evidence for the latter possibility using neural recordings in the monkey inferior temporal (IT) cortex. Neural responses to distorted strings were explained better as a product (but not sum) of shape and distortion tuning, whereas by contrast, responses to letter combinations were explained better as a sum (but not product) of letters. These two rules were sufficient for perfect CAPTCHA decoding and were also emergent in neural networks trained for word recognition. Thus, a separable neural code enables efficient letter recognition.

NEW & NOTEWORTHY Many websites ask us to recognize distorted letters to deny access to malicious computer programs. Why is this task easy for our brains but hard for the computers? Here, we show that, in the monkey inferior temporal cortex, an area critical for recognition, single neurons encode distorted letter strings according to highly systematic rules that enable perfect distorted letter decoding. Remarkably, the same rules were present in neural networks trained for text recognition.

Emergence of a compositional neural code for written words: Recycling of a convolutional neural network for reading

The visual word form area (VWFA) is a region of human inferotemporal cortex that emerges at a fixed location in the occipitotemporal cortex during reading acquisition and systematically responds to written words in literate individuals. According to the neuronal recycling hypothesis, this region arises through the repurposing, for letter recognition, of a subpart of the ventral visual pathway initially involved in face and object recognition. Furthermore, according to the biased connectivity hypothesis, its reproducible localization is due to preexisting connections from this subregion to areas involved in spoken-language processing. Here, we evaluate those hypotheses in an explicit computational model. We trained a deep convolutional neural network of the ventral visual pathway, first to categorize pictures and then to recognize written words invariantly for case, font, and size. We show that the model can account for many properties of the VWFA, particularly when a subset of units possesses a biased connectivity to word output units. The network develops a sparse, invariant representation of written words, based on a restricted set of reading-selective units. Their activation mimics several properties of the VWFA, and their lesioning causes a reading-specific deficit. The model predicts that, in literate brains, written words are encoded by a compositional neural code with neurons tuned either to individual letters and their ordinal position relative to word start or word ending or to pairs of letters (bigrams).

Spatiotemporal dynamics of orthographic and lexical processing in the ventral visual pathway

Reading is a rapid, distributed process that engages multiple components of the ventral visual stream. To understand the neural constituents and their interactions that allow us to identify written words, we performed direct intra-cranial recordings in a large cohort of humans. This allowed us to isolate the spatiotemporal dynamics of visual word recognition across the entire left ventral occipitotemporal cortex. We found that mid-fusiform cortex is the first brain region sensitive to lexicality, preceding the traditional visual word form area. The magnitude and duration of its activation are driven by the statistics of natural language. Information regarding lexicality and word frequency propagates posteriorly from this region to visual word form regions and to earlier visual cortex, which, while active earlier, show sensitivity to words later. Further, direct electrical stimulation of this region results in reading arrest, further illustrating its crucial role in reading. This unique sensitivity of mid-fusiform cortex to sub-lexical and lexical characteristics points to its central role as the orthographic lexicon-the long-term memory representations of visual word forms.

A compositional neural code in high-level visual cortex can explain jumbled word reading

We read jubmled wrods effortlessly, but the neural correlates of this remarkable ability remain poorly understood. We hypothesized that viewing a jumbled word activates a visual representation that is compared to known words. To test this hypothesis, we devised a purely visual model in which neurons tuned to letter shape respond to longer strings in a compositional manner by linearly summing letter responses. We found that dissimilarities between letter strings in this model can explain human performance on visual search, and responses to jumbled words in word reading tasks. Brain imaging revealed that viewing a string activates this letter-based code in the lateral occipital (LO) region and that subsequent comparisons to stored words are consistent with activations of the visual word form area (VWFA). Thus, a compositional neural code potentially contributes to efficient reading.